AB
AB
AiBoss

All Categories

project

InternVL3 - An open-source multimodal large language model from Shanghai AI Lab

InternVL3 is an open-source multimodal large-scale language model (MLLM) from the Shanghai Artificial Intelligence Laboratory, boasting exceptional multimodal perception and reasoning capabilities. The model series includes seven different sizes, ranging from 1B to 78B, and can process text simultaneously...

What is InternVL3?

InternVL3 is an open-source multimodal large-scale language model (MLLM) from the Shanghai Artificial Intelligence Laboratory, boasting exceptional multimodal perception and reasoning capabilities. The model series includes seven different sizes, ranging from 1B to 78B, and can simultaneously process various types of information, including text, images, and video. InternVL3 employs an innovative native multimodal pre-training method, integrating language and multimodal learning into a single pre-training stage, enhancing both multimodal and pure language capabilities. The model's reasoning ability is significantly improved through a hybrid preference optimization algorithm and multimodal testing enhancements.

Main functions of InternVL3

  • Multimodal perception and reasoningInternVL3 can process multiple types of information, such as text, images, and videos, simultaneously, demonstrating excellent multimodal perception and reasoning capabilities.
  • Extended multimodal capabilitiesThe model further expands its multimodal capabilities, covering more application scenarios such as tool usage, GUI agents, industrial image analysis, and 3D visual perception.
  • Native multimodal pre-trainingInternVL3 employs an innovative native multimodal pre-training method that integrates language and multimodal learning into the same pre-training stage, enhancing both multimodal and pure language capabilities.
  • Long context understandingBy integrating Variable Visual Position Encoding (V2PE), InternVL3 performs better in long context understanding.
  • Efficient deployment and invocationInternVL3 can be deployed as an OpenAI-compatible API via LMDeploy's api_server, allowing users to easily call models through OpenAI's API interface.

InternVL3 Technical Principles

  • Native multimodal pre-trainingInternVL3 employs an innovative native multimodal pre-training method that integrates language and visual learning into a single pre-training phase. Unlike traditional methods that train language models separately and then adapt them for multimodal tasks, InternVL3 directly trains by mixing large-scale multimodal data (such as image-text and video-text sequences) with plain text data. This unified training approach allows the model to learn both language and visual representations simultaneously, making it more efficient in handling visual-language tasks without requiring additional alignment modules.
  • Supervision and fine-tuningDuring the fine-tuning phase, InternVL3 employed techniques such as randomized JPEG compression, squared loss reweighting, and multimodal data packaging. Compared to InternVL2.5, InternVL3 further expanded the range of high-quality training samples, covering multiple domains including tool usage, 3D scene understanding, and GUI operations. This enhanced the model's robustness in complex scenarios.
  • Mixed Preference OptimizationInternVL3 introduces the MPO technique, which significantly improves the model's inference performance by combining preference loss, quality loss, and generation loss. MPO helps the model's output more closely approximate the true distribution by introducing additional supervision from both positive and negative samples, reducing bias during inference.
  • Dynamic preprocessing and multimodal input processingInternVL3 supports dynamic preprocessing, which can dynamically adjust the image size and segment it into multiple smaller blocks according to the aspect ratio of the input image to adapt to the model's input requirements. The model supports various multimodal dialogue scenarios such as multi-image input and video input, and can flexibly handle complex multimodal tasks.

InternVL3 project address

Application scenarios of InternVL3

  • Image and video understandingInternVL3 can be used for tasks such as image classification, object detection, and video description generation. It can generate detailed descriptions based on input images or videos, serving content creation and automated editing.
  • Intelligent Interaction and Tool UsageThe model supports tool usage and GUI agent functions, and can act as a graphical user interface (GUI) agent to follow instructions to operate professional software on computers or mobile phones.
  • Industrial Image Analysis and 3D Vision PerceptionInternVL3's multimodal capabilities extend to industrial image analysis and 3D vision perception, enabling it to process complex industrial scene images and support tasks such as architectural drawing understanding and spatial perception reasoning.
  • Intelligent Customer Service and Language Model ApplicationsBased on its powerful language generation capabilities, InternVL3 can be used to develop intelligent customer service systems, providing more efficient and accurate customer support.
Back