InternVideo 2.5 - A video multimodal large-scale model open-sourced by Shanghai AI Lab in collaboration with Nanjing University and the Chinese Academy of Sciences.

What is InternVideo 2.5?

InternVideo 2.5 is an open-source video multimodal large-scale model jointly developed by the Shanghai Artificial Intelligence Laboratory, Nanjing University, and the Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences. It has made significant progress in the field of video understanding, particularly excelling in long video processing and fine-grained spatiotemporal perception. The model can process videos up to tens of thousands of frames long, a six-fold improvement over its predecessor. It can accurately locate target frames in long videos, achieving needle-in-a-haystack retrieval. It supports general video question answering and can perform professional vision tasks such as object tracking and segmentation.

Main functions of InternVideo 2.5

• Ultra-long video processingInternVideo 2.5 can handle videos up to 10,000 frames per second, a 6-fold increase in video processing capability compared to its predecessor (from 3,000 frames to 10,000 frames per second). It can accurately locate target frames in long videos, enabling "needle-in-a-haystack" retrieval and significantly improving the efficiency of video analysis.
• Fine-grained spatiotemporal perceptionThe model can accurately identify and locate objects, scenes, and actions in videos, and understand subtle spatiotemporal relationships. It supports general video question answering and can complete professional vision tasks such as target tracking and segmentation. It can identify the motion trajectory and state changes of objects in complex scenes, providing accurate visual support for fields such as autonomous driving and security monitoring.
• Multimodal fusionInternVideo 2.5 deeply integrates visual perception and language understanding, enabling it to generate detailed descriptions and answers to user questions based on video content.
• Professional vision task supportThrough Task Preference Optimization (TPO), InternVideo 2.5 can handle a variety of professional vision tasks, such as object tracking, segmentation, and temporal localization. It can dynamically adjust the model's processing strategy according to task requirements, providing optimal visual analysis capabilities in different scenarios.
• High-efficiency pre-training and optimizationInternVideo 2.5 employs a progressive, multi-stage training approach, using over 300,000 hours of video corpus for pre-training. This improves model performance and reduces training costs.

Technical Principles of InternVideo 2.5

• Long-term rich context modeling (LRC)LRC enables InternVideo 2.5 to handle videos up to tens of thousands of frames long by extending the model's context length and detail awareness. The core of this is:
  • Video length adaptive token representationThe frame sampling strategy is dynamically adjusted based on the length and content characteristics of the video. For example, short sequences (such as second-level videos) use dense sampling (15 frames per second), while long sequences (such as minute-level or hour-level videos) use sparse sampling (1 frame per second) to ensure that motion information can be effectively captured at different time scales.
  • Hierarchical Context Compression (HiCo)This involves using a layered compression mechanism to reduce spatiotemporal redundancy in video signals while preserving critical information. Specifically, this includes:
    • Spacetime Token MergingThis method performs token merging based on semantic similarity, rather than simple pooling based on location. It significantly reduces redundant information while preserving details.
    • Multimodal context integrationIn the language model processing stage, the compressed visual tokens are further integrated to ensure deep fusion of visual and linguistic information.
• Task Preference Optimization (TPO)TPO guides model learning by transforming the labeled information of fine-grained visual tasks into differentiable task preferences. This enables InternVideo 2.5 to handle various specialized visual tasks, such as object tracking, segmentation, and temporal localization. Specific implementation methods include:
  • Task-specific component integrationAdd task-specific components (such as time headers, mask adapters, etc.) to the model and train it using a task-specific dataset.
  • Visual concept pre-trainingPre-training with a large number of images and video text pairs further enhances the model's visual understanding capabilities.
• Progressive multi-stage training programInternVideo 2.5 employs a progressive, multi-stage training approach to gradually improve model performance.
  • Basic learning stage: Perform task recognition instruction adjustment and video language alignment training.
  • Fine-grained perception training phaseEnhance visual understanding capabilities by integrating task-specific components and visual concept pre-training.
  • Comprehensive optimization phaseMulti-task training and instruction tuning are performed on a hybrid corpus to optimize all model components.
• High-efficiency distributed systemsTo support training and testing of large-scale video data, InternVideo 2.5 developed a distributed system based on multimodal sequence parallelism. The system combines distributed sequence and tensor processing with dynamic data packaging technology, significantly improving training efficiency.

InternVideo 2.5 project address

• Github repository:https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2.5
• HuggingFace model library:https://huggingface.co/OpenGVLab/InternVideo2_5
• arXiv technical paper:https://arxiv.org/pdf/2501.12386

Application scenarios of InternVideo 2.5

• Video content understanding and retrievalInternVideo 2.5 can quickly find relevant video content based on the user's text query and supports complex video retrieval tasks.
• Video editing and creationInternVideo 2.5 provides intelligent support for video editing, such as automatically generating highlight clips, subtitles, or narration. It can edit and adjust videos according to user instructions, improving video creation efficiency.
• Security monitoringIn the field of surveillance and security, InternVideo 2.5 can analyze surveillance video in real time, quickly locate abnormal events, and issue alarms.
• autonomous drivingInternVideo 2.5 can process camera data from autonomous vehicles in real time and accurately identify road signs, traffic signals and obstacles.