Long-VITA - A multimodal model open-sourced by Tencent YouTu in collaboration with Nanjing University and Xiamen University
Long-VITA is an open-source multimodal model developed by Tencent YouTu Lab, Nanjing University, and Xiamen University. It can handle long text inputs with over 1 million tokens and performs exceptionally well on short text tasks. Long-VITA is based on staged training...
What is Long-VITA?
Long-VITA is an open-source multimodal model developed by Tencent YouTu Lab, Nanjing University, and Xiamen University. It can handle long text inputs exceeding 1 million tokens and performs exceptionally well on short text tasks. Long-VITA is based on staged training, progressively expanding its visual and linguistic contextual understanding capabilities to support multimodal inputs including images, videos, and text. Long-VITA uses a dynamic block encoder to process high-resolution images and employs context-parallel distributed inference to support inputs of unlimited length. Trained on open-source datasets including long text data such as comic book summaries and movie plots, Long-VITA achieves new state-of-the-art (SOTA) performance on multiple multimodal benchmarks.
The main functions of Long-VITA
- Long text processing capabilitiesIt can handle inputs of over 1 million tokens and supports multimodal tasks such as long text, long video, and high-resolution images.
- Multimodal understandingIt supports input of images, videos, and text, and is suitable for tasks such as video understanding, high-resolution image analysis, and long text generation.
- Context extension capabilityBased on phased training, the model's context window is gradually expanded while maintaining high performance on short text tasks.
- Open source data trainingTraining with open-source datasets eliminates the need for internal data, lowering the development barrier.
- ScalabilityIt supports context-parallel distributed reasoning, can handle inputs of unlimited length, and is suitable for large-scale deployments.
The technical principle of Long-VITA
- Phased training:
- Visual-Language AlignmentFreeze the language model and visual encoder, train only the projector, and establish initial connections between visual and linguistic features.
- General knowledge learningUsing image-text data for multi-task learning improves the model's ability to understand general knowledge.
- Long sequence fine-tuningGradually expand the context length (from 128K to 1M), add long text and video understanding data, and optimize the model's ability to understand long content.
- Context-Parallel Distributed ReasoningBased on tensor parallelism and context parallelism, it supports inference on inputs of infinite length, solving the memory bottleneck in long text processing.
- Dynamic block encoderIt uses a dynamic block-based strategy to efficiently process high-resolution images and supports inputs with different aspect ratios.
- Masking Language Modeling HeadDuring the inference phase, logits are output based on a mask, which significantly reduces memory usage and supports the generation of large-scale long texts.
Long-VITA's project address
- GitHub repository:https://github.com/VITA-MLLM/Long-VITA
- HuggingFace model library:https://huggingface.co/VITA-MLLM
- arXiv technical paper:https://arxiv.org/pdf/2502.05177v1
Application scenarios of Long-VITA
- Video content generationAutomatically generate video summaries, subtitles, or answer video-related questions.
- Image analysis: Assists in artistic creation, medical imaging diagnosis, or satellite image analysis.
- Long text processingGenerate summaries of novels, academic reports, or documents.
- Intelligent dialogueInteract with users through text, images, and videos in customer service, education, or smart home applications.
- Real-time meeting assistanceProvides real-time translation, captioning, and meeting record generation.