VideoChat-Flash - Shanghai AI Lab and other institutions launch multimodal large-scale models for long video modeling.

What is VideoChat-Flash?

VideoChat-Flash is a multimodal large language model (MLLM) for long video modeling, jointly developed by the Shanghai Artificial Intelligence Laboratory and Nanjing University, among other institutions. The model efficiently processes long videos using hierarchical compression technology (HiCo), significantly reducing computational load while retaining key information. It employs a multi-stage learning scheme progressing from short to long videos, combined with the real-world long video dataset LongVid, to further enhance its ability to understand long videos.

Main functions of VideoChat-Flash

• Long video comprehension abilityVideoChat-Flash effectively processes long videos using HiCo (Hi-Co Compression) technology, capable of handling video content lasting several hours. In the "Needle in a Haystack" (NIAH) mission, it achieved 99.1% accuracy for 10,000 frames (approximately 3 hours of video) for the first time in an open-source model.
• High-efficiency model architectureThe model significantly reduces computation by encoding each video frame into only 16 tokens, resulting in inference speeds 5-10 times faster than its predecessor. A multi-stage, short-to-long learning approach, combined with the real-world long-video dataset LongVid, further enhances the model's performance.
• Powerful video understanding capabilitiesVideoChat-Flash performed exceptionally well in multiple long and short video benchmark tests, outperforming other open-source MLLM models and even surpassing larger-scale models in some tasks.
• Multi-hop contextual understandingVideoChat-Flash supports multi-hop NIAH tasks, enabling the tracking of multiple related image sequences in long videos, further enhancing the ability to understand complex contexts.

The technical principles of VideoChat-Flash

• HiCo layered compression technologyHiCo is one of the core innovations of VideoChat-Flash, designed to efficiently process redundant visual information in long videos.
  • Fragment-level compressionThe process involves dividing a long video into shorter segments and encoding each segment independently.
  • Video-level compressionBased on segment encoding, the contextual information of the entire video is further compressed to reduce the number of tags that need to be processed.
  • Semantic association optimizationBy combining semantic information from user queries, unnecessary video tagging can be further reduced, thereby reducing computational load.
• Multi-stage learning programVideoChat-Flash employs a multi-stage learning approach, progressing from short videos to long videos, to gradually improve the model's ability to understand long contexts.
  • initial stageSupervised fine-tuning using short videos and their annotations builds the model's basic understanding capabilities.
  • Expansion phaseGradually introduce long video data to train the model to handle more complex contexts.
  • Mixed corpus trainingUltimately, training is performed on a mixed corpus containing both short and long videos to achieve a comprehensive understanding of videos of different lengths.
• LongVid Real-World Long Video DatasetTo support model training, the research team built the LongVid dataset, which contains 300,000 hours of real-world long videos and 200 million words of annotations. This dataset provides the model with rich training material, enabling it to better adapt to long video understanding tasks.
• Model ArchitectureThe architecture of VideoChat-Flash consists of three main parts: a visual encoder, a visual-language connector, and a large language model (LLM). Through this hierarchical architecture, the model can efficiently encode video content into compact token sequences and perform long context modeling through LLM.

VideoChat-Flash project address

• GitHub repository:https://github.com/OpenGVLab/VideoChat-Flash
• arXiv technical paper:https://arxiv.org/pdf/2501.00574

Application scenarios of VideoChat-Flash

• Video subtitle generation and translationThe model can generate detailed and accurate video subtitles, suitable for multilingual translation and accessibility subtitle generation, helping viewers better understand video content.
• Video Q&A and InteractionVideoChat-Flash supports natural language question answering based on video content. Users can ask questions to obtain key information from videos, such as movie plot analysis or knowledge points from documentaries.
• Embodied AI and Robot LearningIn the field of embodied AI, VideoChat-Flash can help robots learn complex tasks, such as making coffee, through long-duration videos from their own perspective, guiding the robot to complete the task by analyzing key events in the video.
• Sports video analysis and highlights generationThe model can analyze sports game videos, extract key events, and generate highlights to help viewers quickly understand the exciting moments of the game.
• Surveillance video analysisVideoChat-Flash can handle long-duration surveillance videos, identify and track key events, and improve the efficiency and accuracy of surveillance systems.