Kimi-VL - A lightweight, open-source multimodal visual language model from the Dark Side of the Moon.

What is Kimi-VL?

Kimi-VL is a lightweight multimodal visual language model open-sourced by Dark Side of the Moon. It is based on the lightweight MoE model Moonlight (16B total parameters, 2.8B activation parameters) and the native resolution MoonViT visual encoder (400M parameters). Kimi-VL supports multimodal inputs such as single images, multiple images, videos, and long documents. It performs exceptionally well in tasks including image perception, mathematics, multidisciplinary problems, and OCR, particularly excelling in long contexts (128K) and complex reasoning. It outperforms models like GPT-4o in tasks such as mathematical reasoning and long video understanding. Kimi-VL also offers a version supporting long-chain reasoning, Kimi-VL-Thinking, which is based on long-chain reasoning fine-tuning and reinforcement learning with only 2.8B activation parameters. In benchmark tests with higher reasoning difficulty, some of its performance is close to or even surpasses that of state-of-the-art models with very large dimensions.

The latest open-source version of Kimi-VL-A3B-Thinking-2506 demonstrates excellent performance in multimodal reasoning benchmarks, with significantly improved accuracy and a 20% reduction in think length. The model's visual understanding capabilities are significantly enhanced, supporting higher-resolution image processing and achieving remarkable progress in high-resolution perception and OS-agent grounding benchmarks (scoring 83.2 on V*Benchmark, 52.8 on ScreenSpot-Pro, and 52.5 on OSWorld-G). The model excels in multiple areas, including image understanding, graph reasoning, and mathematical computation, supporting specific response patterns and thought chains.

Kimi-VL's main functions

• Multimodal information inputSupports various input formats such as single image, multiple images, video, and long document.
• Fine-grained image perception: Perform detailed analysis on images to identify complex details and scenes within them.
• Mathematical and logical reasoningIt performs well in multimodal math problems and logical reasoning tasks, and supports complex calculations that incorporate visual information.
• OCR and text recognitionIt performs excellently in optical character recognition (OCR) tasks, accurately recognizing text content in images.
• Intelligent agent applicationsSupports agent tasks, such as screenshot parsing and intelligent agent navigation, and handles complex visual and text interaction scenarios.
• Long-term thinking abilityThe Kimi-VL-Thinking version supports deeper reasoning in complex tasks.

Kimi-VL's technical principles

•  Architecture Design:
  • Visual encoder (MoonViT)Based on the Vision Transformer architecture with 400M parameters, it processes images at native resolution without the need for image segmentation or stitching. It introduces the packing method from NaViT to divide the image into image blocks, flatten them, and concatenate them into a one-dimensional sequence, sharing the same operators and optimization methods as the language model.
  • Multilayer perceptron projection module (MLP Projector)This approach bridges the visual encoder and language model using a two-layer MLP. Pixel rearrangement compresses the spatial dimension of image features, projecting them onto the embedding dimension required by the language model.
  • Hybrid Expert (MoE) Language Model (Moonlight)A lightweight MoE architecture based on 6B total parameters and 2.8B activation parameters. It is initialized from an intermediate checkpoint in the pre-training phase, which has already processed 5.2T tokens of plain text data and activated 8K context lengths. A hybrid training scheme is then used to further pre-train on 2.3T tokens of multimodal and plain text data.
• Pre-training phase:
  • Independent ViT training: Train the visual encoder independently to establish a robust visual feature extraction capability.
  • Joint trainingIt includes three stages: pre-training, cooling, and long context activation, while simultaneously enhancing the model's language and multimodal capabilities.
  • Post-training phaseThe model performance was further optimized by performing tests in 32K and 128K contexts respectively. Long-CoT was used to fine-tune activation and enhance long-thinking capabilities, and reinforcement learning was used to further improve the model's reasoning ability.
• Performance optimizationAccurate information retrieval in long texts and videos using a 128K context window. The native resolution encoder MoonViT maintains high accuracy with low computational overhead in ultra-high resolution vision tasks. Kimi-VL-Thinking enables deeper reasoning in complex tasks based on Long CoT activation and reinforcement learning.

Kimi-VL's performance advantages

• High intelligenceKimi-VL excels in multimodal reasoning and multi-step agent tasks, and its text processing capabilities are comparable to those of pure text language models. In benchmark tests such as MMMU, MathVista, and OSWorld, Kimi-VL achieves remarkable results without relying on "long-term thinking" abilities, demonstrating outstanding intelligence.
• Long contextKimi-VL boasts an ultra-long context window of 128K, which significantly outperforms other models in the same class when handling long video and long document tasks, such as LongVideoBench and MMLongBench-Doc benchmark tests. It supports accurate retrieval and understanding of massive amounts of information, providing more comprehensive context support for complex tasks.
• Stronger visual abilityCompared to other open-source visual language models, Kimi-VL demonstrates comprehensive and significant competitive advantages in multiple visual scenarios, including visual perception, visual world knowledge, OCR, and high-resolution screen snapshot parsing. Kimi-VL can accurately and efficiently complete tasks in capturing details of complex images and deeply understanding visual information.

Kimi-VL's project address

• GitHub repository:https://github.com/MoonshotAI/Kimi-VL
• HuggingFace model library:https://huggingface.co/collections/moonshotai/kimi-vl
• Technical Papers:https://github.com/MoonshotAI/Kimi-VL/blob/main/Kimi-VL.pdf

Application scenarios of Kimi-VL

• Intelligent Customer ServiceUsed in multi-turn dialogues to answer user questions, supporting interaction that combines text and images.
• Educational guidanceTo assist students in their learning, providing illustrated solutions and teaching materials.
• Content creationGenerates text, images, and video content to assist in video editing and creative content creation.
• Medical assistanceAnalyze medical images to provide preliminary diagnostic suggestions and health consultations.
• Corporate OfficeProcess long documents, extract key information, and support intelligent meeting services.