LongVU - Meta AI's open-source long video understanding model
LongVU is a long-video understanding model developed by the Meta AI team, based on a spatiotemporal adaptive compression mechanism. It addresses the challenge of processing long videos limited by the context size of large language models (LLMs). LongVU is based on cross-modal queries and...
What is LongVU?
LongVU is a long-video understanding model developed by the Meta AI team, based on a spatiotemporal adaptive compression mechanism. It addresses the challenge of processing long videos limited by the context size of large language models (LLMs). Based on cross-modal queries and inter-frame dependencies, LongVU can preserve the visual details of long videos while reducing the number of video tags. LongVU uses DINOv2 features to remove highly similar redundant frames, employs text-guided cross-modal queries for selective frame feature reduction, and performs spatial tag compression based on temporal dependencies when necessary. LongVU can effectively process a large number of frames with minimal loss of visual information within a given context length.
Main functions of LongVU
- Spatiotemporal adaptive compressionLongVU processes long videos by reducing the number of video tags, preserving key visual details in the video, and can handle very long video content within a limited context length.
- Cross-modal queryText-guided cross-modal queries selectively reduce the features of video frames, preserving detailed information of the frames most relevant to the text query while reducing other frames to a low-resolution tagged representation.
- Inter-frame dependency exploitationBased on the analysis of temporal dependencies between video frames, LongVU can perform spatial tag compression based on dependencies, reducing the context length requirements of the model.
- Long video comprehensionLongVU can effectively handle video input sampled at 1fps and can adaptively reduce the average number of tags per frame for hourly long videos to 2, adapting to multimodal large language models (MLLM) with an 8k context length.
LongVU's technical principles
- Time compression strategy: Use DINOv2 features to identify and remove highly similar redundant video frames, reducing redundancy in the temporal dimension of the video.
- Reduced selectivityText-guided cross-modal queries retain complete tokens for frames related to the text query, while spatial pooling is applied to other frames to reduce redundancy in spatial dimensions.
- Spatial tag compressionFor extremely long videos, LongVU further compresses spatial labels based on the temporal dependencies between frames. By calculating the spatial label similarity between frames, spatial labels of subsequent frames that are too similar to the first frame are removed, reducing the amount of data the model needs to process.
- Multimodal trainingLongVU combines image-language pre-training and video-language fine-tuning, using large-scale video-text pairs for training, to improve the model's performance in video understanding tasks.
LongVU's project address
- Project official website:vision-cair.github.io/LongVU
- GitHub repository:https://github.com/Vision-CAIR/LongVU
- HuggingFace model library:https://huggingface.co/collections/Vision-CAIR/longvu-67181d2debabfc1eb050c21d
- arXiv technical paper:https://arxiv.org/pdf/2410.17434
- Experience the demo online:https://huggingface.co/spaces/Vision-CAIR/LongVU
Application scenarios of LongVU
- Video content analysisLongVU is used to analyze long video content and extract key information, such as identifying important events and scenes in surveillance videos, news reports, or documentaries.
- Video search and indexingBased on its understanding of video content, LongVU helps build video search engines, allowing users to quickly locate relevant segments in videos using text queries.
- Video content generationLongVU is used to generate descriptions, summaries, or subtitles for video content, improving the accessibility and accessibility of the video content.
- Video Q&A SystemLongVU supports building video question-and-answer systems, where users can ask questions about video content, and the system can understand the questions and provide accurate answers.
- Education and trainingIn the field of education, LongVU is used to analyze teaching videos, extract key teaching points, and help students better understand and master course content.