InternVL-U - an open-source multimodal integrated model from Shanghai AI Lab and other organizations.

What is InternVL-U?

InternVL-U is a lightweight, unified multimodal model with 4B parameters, open-sourced by the Shanghai Artificial Intelligence Laboratory in collaboration with several top universities. It achieves an end-to-end closed loop of "understanding—reasoning—generation—editing" for the first time. The model employs three core designs: unified contextual modeling, modality-specific modularization, and decoupled visual representation, overcoming the bottlenecks of high training costs and uneven capabilities inherent in traditional models. The model surpasses 14B-level models in complex scenarios such as text rendering, scientific reasoning, and spatial modeling. Its GenExam benchmark score of 22.9 for scientific image generation leads all open-source unified models, providing efficient and flexible multimodal solutions for scientific research and education, intelligent office work, and creative content creation.

Main functions of InternVL-U

• Multimodal understandingIt supports accurate analysis of visual information in images and answers various complex questions posed by users.
• Logical reasoningThe model uses the mind chain technique to break down abstract natural language instructions into specific, executable steps.
• Image generationGenerate high-fidelity, semantically accurate, and aesthetically pleasing visual images based on text descriptions.
• Image editingIt allows for precise modification of specified areas of an image while preserving the original background texture and lighting effects.
• Text renderingThe model can accurately generate Chinese and English characters, numbers, and mathematical symbols, completely eliminating character distortion and spelling errors.
• Scientific VisualizationIt supports the creation of professional scientific research diagrams that conform to disciplinary standards, such as molecular structures and algorithm flowcharts.
• Spatial modelingThe model can perform solid geometry calculations, CAD multi-view conversion, and arbitrary angle rotation of 3D objects.
• Fun CreationInternVL-U can quickly generate fun and creative content such as emoticons and memes that are suitable for online dissemination.

InternVL-U Technical Principles

• Decoupling visual representationInternVL-U employs an asymmetric visual representation strategy. In the understanding task, it uses pre-trained ViT to extract high semantic features to ensure the accuracy of understanding complex scenes. In the generation task, it uses independent VAEs to compress images to latent space to preserve pixel-level details. The model avoids the optimization conflict between semantic understanding and image reconstruction, enabling the model to maintain leading performance in both understanding and generation benchmarks.
• Dual-stream MMDiT generator headThe visual generation head employs a dual-stream structure to process multimodal contextual features and image latent features separately. It uses a sigmoid-gated attention mechanism to adjust weights to mitigate performance degradation in long context scenarios. It adopts a unified MSRoPE 3D positional encoding to ensure accurate preservation of spatial structure and supports multi-resolution generation from 512 to 1024 pixels to avoid stitching artifacts at high resolutions.
• Three-level progressive trainingThe model employs a three-stage strategy of pre-training, continuous pre-training, and fine-tuning. In the first stage, the backbone network is frozen to train the generation head and activate multimodal contextual understanding capabilities. In the second stage, the backbone network is fixed to train multi-resolution generation capabilities and select high-aesthetic samples. In the third stage, the entire model is unfrozen and integrated with the thought chain data to achieve deep collaboration between understanding, reasoning, and generation.

InternVL-U project address

• GitHub repositoryhttps://github.com/OpenGVLab/InternVL-U
• HuggingFace model libraryhttps://huggingface.co/InternVL-U/InternVL-U
• arXiv technical paper: https://arxiv.org/pdf/2603.09877

Application scenarios of InternVL-U

• Scientific Research and EducationIt provides researchers and students with professional visualizations such as molecular structures, algorithm flowcharts, and force analysis diagrams to assist in teaching demonstrations and the creation of illustrations for papers.
• Smart OfficeIt enables automated document generation, batch poster editing, and simultaneous modification of text in multiple regions, improving the efficiency of business document and marketing material production.
• Creative DesignIt supports designers in quickly generating high-fidelity concept art, stylized images, and multi-resolution visual materials, lowering the barrier to entry for professional design.
• Content OperationsIt helps new media operators generate fun content such as emojis and memes with one click, adapting to social media dissemination scenarios.
• Industrial manufacturingThe model can perform CAD multi-view conversion, solid geometry calculations, and 3D object rotation, assisting in engineering design and product prototype visualization.