VE-Bench - Peking University's first open-source new metric for evaluating video editing quality.
VE-Bench, recently released by the MMCAL research team at Peking University, is the first metric specifically designed for evaluating video editing quality. VE-Bench is designed to closely align with human perception, providing a more accurate assessment of video editing effectiveness...
What is VE-Bench?
VE-Bench, recently released by the MMCAL research team at Peking University, is the first metric specifically designed for evaluating the quality of video editing. VE-Bench is designed to closely align with human perception, providing a more accurate assessment of video editing results. When evaluating edited videos, VE-Bench QA considers not only visual quality metrics such as aesthetics and distortion emphasized in traditional video quality assessment methods, but also focuses on text-video alignment and correlation modeling between the source and edited videos.
VE-Bench comprises two main parts: VE-Bench DB and VE-Bench QA. VE-Bench DB is a video quality assessment database containing a rich collection of source videos, editing instructions, editing results from different video editing models, and subjective rating samples from 24 participants with diverse backgrounds, totaling 28,080 rating samples. VE-Bench QA is a quantitative, human-perception-consistent metric tool specifically designed for text-driven video editing tasks. The VE-Bench code and data are available on GitHub.
Main functions of VE-Bench
- Video Quality Assessment Model (VE-Bench QA)VE-Bench includes a quality assessment model designed to provide metrics consistent with human perception for edited videos. It considers visual quality metrics such as aesthetics and distortion, which are emphasized in traditional video quality assessment methods, and also focuses on text-video alignment and correlation modeling between the source and edited videos.
- Video Quality Assessment Database (VE-Bench DB)VE-Bench DB is a video quality assessment database built for video editing. It contains a wealth of source videos, editing instructions, editing results from different video editing models, and subjective rating samples from 24 participants with different backgrounds, totaling 28,080 rating samples.
- Text-Video Consistency AssessmentVE-Bench QA is based on BLIP for video-text correlation modeling. It extends the BLIP visual branch to three dimensions by adding a Temporal Adapter and obtains the output by cross-attention with the results of the text branch.
- Source video - Edited video dynamic correlation assessmentVE-Bench QA projects the source video and the edited video into a high-dimensional space using a spatiotemporal Transformer. After stitching them together, it uses an attention mechanism to calculate the correlation between the two and finally obtains the corresponding output through regression calculation.
- Traditional visual quality assessmentVE-Bench QA references the excellent work DOVER on natural scene video quality evaluation, and outputs corresponding results through a backbone network pre-trained in terms of aesthetics and distortion.
- Multi-dimensional assessmentVE-Bench QA evaluates text-driven video editing from three dimensions: text-video consistency, source video-edited video dynamic relevance, and traditional visual quality.
VE-Bench Technical Principles
- Linear layer regressionThe final score is obtained by passing the output of each branch through linear layer regression.
- Deep learning algorithmsVE-Bench employs advanced deep learning algorithms and, through learning from a large number of real-world video samples, has established a model capable of simulating the human visual and auditory systems.
- Multimodal learning frameworkVE-Bench can process image, audio, and text information in videos simultaneously. Through training with a large number of real-world video samples, VE-Bench has learned how to analyze the technical parameters of videos and capture those artistic aesthetics and emotional expressions that are difficult to quantify.
VE-Bench project address
- Github repository:https://github.com/littlespray/VE-Bench
- arXiv technical paper:https://arxiv.org/pdf/2408.11481
Application scenarios of VE-Bench
- FilmmakingVE-Bench can be used in film post-production to help directors and editors analyze whether the color, lighting effects, and transitions of each shot meet expectations, ensuring that the emotional atmosphere of the film is accurately conveyed.
- Short video platform content optimizationShort video creators can use VE-Bench to monitor the visual effects, audio quality, and narrative structure of their videos in real time, and optimize them based on the system's suggestions to improve the overall viewing experience and engagement rate.
- Precision Marketing in the Advertising IndustryVE-Bench can analyze video content to help the advertising industry conduct precise marketing, ensuring that advertising videos match the emotional and visual expectations of the target audience.
- Video editing quality assessmentVE-Bench focuses on the relationship between the edited video and the original video. For example, in the task of "removing the girl's earrings", the character ID needs to be preserved, and the source video and the edited result should have a strong semantic relevance.