Video-XL - An open-source, ultra-long visual understanding model jointly developed by the Academy of Artificial Intelligence and several universities.
Video-XL is an ultra-long visual understanding model designed specifically for understanding hour-long videos, jointly developed by researchers from the Beijing Academy of Artificial Intelligence, Shanghai Jiao Tong University, Renmin University of China, the Chinese Academy of Sciences, Beijing University of Posts and Telecommunications, and Peking University. Based on...
What is Video-XL?
Video-XL is an ultra-long visual understanding model designed specifically for hour-long videos, jointly developed by researchers from the Beijing Academy of Artificial Intelligence (BAAI), Shanghai Jiao Tong University, Renmin University of China, the Chinese Academy of Sciences, Beijing University of Posts and Telecommunications (BUPT), and Peking University. Based on visual contextual latent summarization technology, it compresses visual information into a compact form, improving processing efficiency and reducing information loss. It performs exceptionally well in multiple long video understanding benchmarks; for example, it achieves nearly 10% higher accuracy than existing state-of-the-art methods on VNBench, and close to 95% accuracy when processing 2048 frames of video on a single 80GB GPU. Video-XL demonstrates a balance between efficiency and performance, showing potential for long video-specific tasks such as movie summarization, surveillance anomaly detection, and ad placement recognition.
Video-XL's main functions
- Hour-level video comprehensionVideo-XL can handle very long videos, achieving video content understanding at the hour level.
- Visual compressionBased on the visual context latent summarization technique, a large amount of visual information is compressed into a more compact form to adapt to the processing capabilities of the model.
- High-efficiency computingWhile maintaining high accuracy, it reduces the consumption of computing resources and can process a large number of video frames on a single GPU.
- Multimodal data processingIt can handle a variety of data types, including single images, multiple images, and videos.
- Long video specific task processingSuitable for specific tasks related to long videos, such as movie summaries, monitoring anomaly detection, and ad placement recognition.
The technical principles of Video-XL
- Visual Context Latent Summarization: Introducing Visual Summary Tags (VSTs) to compress the activations of the original visual context (such as keys and values in self-attention modules) into highly compact representations.
- Block processingThe visual marker sequence is divided into blocks of the same size, VSTs are inserted into each block, and the visual signal is progressively compressed using LLM.
- Autoregressive methodsDuring the encoding of each block, in addition to using an extra projection matrix, Video-XL reuses all modules to transform the VST hidden state, compressing the information of the visual signal into the activation of the VST.
- Unified visual coding schemeIt encodes single images, multiple images, and videos into the same space to process different multimodal data.
- Long Video Dataset (VICO): Develop new long video datasets specifically designed for handling longer videos and dynamic visual contexts, enhancing the model's ability to understand long videos.
Video-XL's project address
- GitHub repository:https://github.com/VectorSpaceLab/Video-XL
- HuggingFace model library:https://huggingface.co/sy1998/Video_XL
- arXiv technical paper:https://arxiv.org/pdf/2409.14485
Application scenarios of Video-XL
- Summary of movie and video contentVideo-XL can understand and summarize long video content, such as the main plot and events of movies or TV series, providing users with a concise overview.
- Video surveillance analysisIn the field of security monitoring, it is used for abnormal behavior detection to identify abnormal events or unusual activities in surveillance videos.
- Advertising placement and analysisVideo-XL helps analyze the effectiveness of ad placement in video content, understand the relationship between ads and video content, and the audience's reaction to ads.
- Educational and training videosIn the field of education, Video-XL is used to understand and summarize long teaching videos, helping students quickly grasp the key points of the course.
- Video content retrievalVideo-XL is used for content retrieval in video databases, helping users quickly find key segments or related information in videos.