Mini-InternVL - A lightweight, multimodal large-scale model launched by Shanghai AI Lab in collaboration with Tsinghua University and other institutions.
Mini-InternVL is a "mini" version of the Shusheng·Wanxiang large model. It is a lightweight, multimodal, large-scale language model series jointly launched by the Shanghai AI Lab, Tsinghua University, Nanjing University, and other institutions. It includes three parameter versions: 1B, 2B, and 4B...
What is Mini-InternVL?
Mini-InternVL is a "mini" version of the Shusheng·Wanxiang large-scale model. It's a lightweight multimodal large-scale language model series jointly developed by the Shanghai AI Lab, Tsinghua University, Nanjing University, and other institutions. It includes three parameter versions: 1B, 2B, and 4B, achieving high performance with a smaller number of parameters. Mini-InternVL-4B, in particular, achieves approximately 90% of the performance of InternVL2-76B with only 5% of the parameters. Mini-InternVL uses the InternViT-300M as the visual encoder and combines it with different pre-trained language models. It reduces the number of visual tags and improves processing efficiency based on a dynamic resolution input strategy and pixel shuffling operations. Mini-InternVL performs excellently in several general multimodal benchmarks and can be adapted to downstream tasks in specific domains using a simple transfer learning framework.
Main functions of Mini-InternVL
- Multimodal understanding and reasoningGiven an image and text input, understand and infer the semantic relationships between them.
- Cross-domain adaptabilityBased on knowledge distillation and transfer learning techniques, it can be adapted to different domains and tasks.
- Lightweight and efficientMini-InternVL achieves performance comparable to large models while maintaining a relatively small number of model parameters (100 million to 4 billion). This enables efficient operation in resource-constrained environments (such as consumer-grade GPUs or edge devices), reducing deployment costs and computing resource requirements.
- Visual instruction tuningIt has the ability to optimize based on visual instructions, and better understand and execute user image-based instructions.
- Dynamic resolution inputIt supports dynamic resolution input strategy, which divides the image into tiles of different sizes according to the aspect ratio and performs corresponding processing.
The technical principle of Mini-InternVL
- Visual encoder (InternViT-300M)As the core component of the model, the visual encoder is responsible for converting the input image into a feature representation that the model can understand. InternViT-300M is a lightweight vision model that inherits rich visual knowledge from the more powerful InternViT-6B model through knowledge distillation. This distillation process enables InternViT-300M to perform well across multiple vision domains while maintaining a small number of model parameters.
- Knowledge distillationThe goal is to transfer knowledge from a large teacher model to a small student model, allowing the student model to inherit the performance of the teacher model. In Mini-InternVL, the InternViT-6B acts as the teacher model, passing the knowledge of the hidden state to the InternViT-300M based on the negative cosine similarity loss.
- MLP projectorMLP (Multilayer Perceptron) projectors are used to connect visual encoders and language models. They project the feature vectors output by the visual encoder into a space suitable for the language model to process, enabling visual and textual information to be effectively fused and interacted.
- Pre-trained Language Models (LLMs)Mini-InternVL combines different pre-trained language models, such as Qwen2-0.5B, InternLM2-1.8B, and Phi-3mini.
- Dynamic resolution input strategyThe model employs a dynamic resolution input strategy. This strategy divides the image into 448×448 tiles based on its aspect ratio, combines these tiles into a fixed sequence, and ultimately generates a 2688×896 resolution image representation. Furthermore, the model adds a thumbnail to each tile to provide global contextual information.
- Pixel shuffle operationBased on pixel shuffling, the model reduces the image resolution to one-quarter of its original value, thereby reducing the number of visual markers.
Mini-InternVL project address
- GitHub repository:https://github.com/OpenGVLab/InternVL
- HuggingFace model library:https://huggingface.co/collections/OpenGVLab/internvl-adaptation
- arXiv technical paper:https://arxiv.org/pdf/2410.16261
Application scenarios of Mini-InternVL
- autonomous drivingIt is used in environmental perception, behavior prediction and path planning, to process multi-view images, identify and predict the behavior of traffic participants, and generate safe and efficient driving paths.
- Medical Image ProcessingIt assists in disease diagnosis, image annotation, and treatment recommendations; analyzes medical images, and provides diagnostic support and treatment suggestions.
- remote sensingTo conduct land use classification, disaster monitoring, and environmental monitoring; to identify different types of land use; to assess the impact of natural disasters; and to monitor environmental changes.
- Document and chart understandingExtract document content, parse tables and charts, generate document summaries and chart explanations, and support data visualization and analysis.
- Video UnderstandingExtract keyframes and content from videos, identify people's actions and events in the videos, generate video summaries, and answer video-related questions.