dots.vlm1 - The first multimodal large model open-sourced by Xiaohongshu Hi Lab

What is dots.vlm1?

dots.vlm1 is the first multimodal large-scale model open-sourced by Xiaohongshu's hi lab. Based on a 1.2 billion parameter visual encoder NaViT trained from scratch and a DeepSeek V3 large language model (LLM), it possesses powerful visual perception and text reasoning capabilities. The model performs excellently on visual understanding and reasoning tasks, approaching the level of closed-source state-of-the-art models, and also remains competitive on text tasks. The NaViT visual encoder in dots.vlm1 was trained entirely from scratch, natively supports dynamic resolution, and adds pure visual supervision on top of text supervision, improving perception capabilities. The training data incorporates various synthetic data approaches, covering diverse image types and their descriptions, significantly improving data quality.

Main functions of dots.vlm1

• Strong visual comprehension abilityIt can accurately identify and understand the content in images, including complex charts, tables, documents, and graphics, supports dynamic resolution, and is suitable for a variety of visual tasks.
• Efficient text generation and reasoningBased on DeepSeek V3 LLM, it can generate high-quality text descriptions and performs well in text reasoning tasks such as mathematics and code.
• Multimodal data processingIt supports data processing with interwoven images and text, and can perform comprehensive reasoning by combining visual and textual information, making it suitable for multimodal application scenarios.
• Flexible adaptation and expansionIt connects the visual encoder and language model through a lightweight MLP adapter, making it easy to flexibly adapt and extend for different tasks.
• Open source and opennessIt provides complete open-source code and models to support developers in research and application development, and promotes the development of multimodal technology.

The technical principles of dots.vlm1

• NaViT visual encoderdots.vlm1 uses NaViT, a 1.2 billion parameter visual encoder trained from scratch, rather than fine-tuning existing mature models. It natively supports dynamic resolution, can handle image inputs of different resolutions, and adds pure visual supervision on top of text supervision, improving the model's ability to perceive images.
• Multimodal data trainingThe model employs diverse multimodal training data, including ordinary images, complex charts, tables, documents, graphs, and corresponding text descriptions (such as Alt Text, Dense Caption, Grounding, etc.). It incorporates synthetic data approaches and interleaved text and image data such as web pages and PDFs, improving data quality through rewriting and cleaning to enhance the model's multimodal understanding capabilities.
• Fusion of visual and language modelsThe dots.vlm1 module combines a visual encoder with the DeepSeek V3 Large Language Model (LLM) and connects them through a lightweight MLP adapter to achieve effective fusion of visual and linguistic information, supporting the processing of multimodal tasks.
• Three-stage training processThe model training consists of three stages: visual encoder pre-training, VLM pre-training, and VLM post-training. By progressively increasing image resolution and introducing diverse training data, the model's generalization ability and multimodal task processing capability are enhanced.

The project address for dots.vlm1

• GitHub repositoryhttps://github.com/rednote-hilab/dots.vlm1
• Hugging Face Model Libraryhttps://huggingface.co/rednote-hilab/dots.vlm1.inst
• Experience the demo onlinehttps://huggingface.co/spaces/rednote-hilab/dots-vlm1-demo

Application scenarios of dots.vlm1

• Complex Graph ReasoningIt can analyze and reason about complex charts, helping users better understand and interpret the information in the charts.
• STEM Problem SolvingIn the fields of science, technology, engineering, and mathematics (STEM), models can help solve related problems and provide solutions.
• Long tail recognitiondots.vlm1 also has good recognition capabilities for some categories or objects that appear less frequently.
• Visual reasoningIt can handle reasoning tasks involving visual information, such as obstacle recognition and product comparison analysis.
• Text and image Q&A and interactionIt supports question-and-answer tasks that combine text and images, and can conduct multi-turn dialogues, providing coherent answers based on the context.
• Content RecommendationBased on multimodal data, it provides users with personalized content recommendations, such as recommending relevant images, texts, or videos on the Xiaohongshu platform.