Vidi2 - A multimodal video understanding and generation model launched by ByteDance

What is Vidi2?

Vidi2 is a multimodal large language model launched by ByteDance, focusing on video understanding and creation. It has achieved industry-leading performance in multimodal temporal retrieval (TR) and made significant progress in spatiotemporal localization (STG) and video question answering (Video QA). Based on text queries, Vidi2 can not only identify the corresponding timestamps in the video but also mark the bounding boxes of target objects. Vidi2 introduces new benchmarks, VUE-STG and VUE-TR-V2, to better evaluate STG capabilities. In practical applications, Vidi2 supports features such as intelligent editing, intelligent scene composition, and intelligent subtitles, helping creators to create videos more efficiently.

Vidi2's main functions

• Video Understanding and RetrievalIt can process video footage of several hours in length, quickly locate relevant time periods in the video based on text queries, and support long video understanding in complex scenarios.
• Spatiotemporal positioning (STG)It can not only identify timestamps in videos, but also mark the bounding boxes of target objects within these time periods, achieving accurate spatiotemporal positioning.
• Video Q&AIt supports video question answering tasks, can answer user questions based on video content, and provides more comprehensive multimodal reasoning capabilities.
• Intelligent creation toolsIt supports multiple functions such as automatic editing, intelligent scene splitting, and intelligent subtitles, helping creators quickly convert long video footage into short videos suitable for platforms such as TikTok.
• Multi-view switching and intelligent reconstruction graphIt supports automatic multi-view switching and intelligent, composition-aware re-selection and cropping to improve the visual effect of videos.

Vidi2's technical principles

• Multimodal fusionVidi2 combines a visual encoder (used to process video frames and clips) with a large language model, and through instructional fine-tuning and alignment, forms an integrated long-video understanding and expression capability.
• Spatiotemporal positioning (STG)Vidi2 uses spatiotemporal localization technology to accurately locate relevant timestamps and bounding boxes of target objects in videos based on text queries. It processes the spatiotemporal information of videos using deep learning models (such as Transformer) based on multimodal feature extraction and similarity calculation.
• Video Understanding and RetrievalVidi2 utilizes multimodal text-to-video retrieval technology to convert text and video data into vector representations, and then achieves text-to-video retrieval through similarity calculations (such as cosine similarity). Through feature extraction and deep learning techniques, it performs structured analysis of video content, enabling efficient video retrieval.
• Multi-granularity temporal modelingVidi2 employs multi-granularity temporal modeling technology, which can process information at different time scales, from single frames to long video clips. Combined with memory and retrieval mechanisms (such as sliding window + memory token/retrieval enhancement), it achieves efficient understanding and retrieval of long videos.
• Cross-modal alignmentVidi2 uses cross-modal alignment technology to effectively align visual, audio, and textual information in a video, thereby achieving a more comprehensive understanding of video content.

Vidi2 project address

• Project official websitehttps://bytedance.github.io/vidi-website/
• Github repositoryhttps://github.com/bytedance/vidi
• arXiv technical paper: https://arxiv.org/pdf/2511.19529

Vidi2 application scenarios

• Intelligent video editingIt automatically extracts highlights from long videos and generates content suitable for short video platforms, improving creation efficiency.
• Video Q&AAnswer user questions based on video content, enhancing the user's interactive experience with the video content.
• Spatiotemporal positioning-assisted editingBy accurately locating time and objects in a video, it assists creators in performing complex editing tasks, such as multi-view switching and intelligent reconstruction of images.
• Automatic subtitle generationAutomatically generate subtitles for videos, improving video accessibility and user experience.
• Content creation assistanceIt helps creators turn simple prompts or themes into complete video scripts, including titles, hooks, and storyboards.