AB
AB
AiBoss

All Categories

project

MiMo-V2-Flash - Xiaomi's open-source high-performance language model

MiMo-V2-Flash is a high-performance language model open-sourced by Xiaomi, boasting 309 billion total parameters and 15 billion activation parameters. Designed specifically for agent AI, it employs a hybrid attention architecture, combining sliding window attention (SWA) and global attention...

What is MiMo-V2-Flash?

MiMo-V2-Flash is a high-performance language model open-sourced by Xiaomi, boasting 309 billion total parameters and 15 billion activation parameters. Designed specifically for agent AI, it employs a hybrid attention architecture, combining sliding window attention (SWA) and global attention (GA), with a window size of 128 tokens. Through learnable attention sink bias technology, it significantly reduces KV cache storage while maintaining performance on long texts. Furthermore, it features a lightweight multi-token prediction (MTP) module, each containing only 33 million parameters, using a dense feedforward network (FFN) and SWA to significantly improve generation speed. This results in inference cost being only 2.5% of the benchmark closed-source model Claude 4.5 Sonnet, achieving a 2x speedup. MiMo-V2-Flash performs exceptionally well in various agent tasks and complex inference tasks, such as SWE-Bench and mathematical inference tasks. It supports FP8 mixed-precision inference and is recommended for deployment using the SGLang framework.

Main functions of MiMo-V2-Flash

  • High-performance inferenceIt adopts a hybrid attention architecture and lightweight multi-token prediction technology, which significantly improves inference efficiency, generates tokens faster, and greatly reduces inference costs.
  • Long text processing capabilitiesBy combining sliding window attention and global attention, it supports context lengths of up to 256K, making it suitable for handling long text tasks.
  • Agent task optimizationIt is designed specifically for agent AI and is suitable for complex reasoning and multi-turn dialogue tasks, through large-scale agent reinforcement learning and multi-teacher online policy distillation.
  • Code generation and understandingIt excels in code generation and understanding tasks, supports multiple programming languages, and can efficiently complete code completion, generation, and debugging tasks.
  • Multilingual supportIt possesses multilingual capabilities and can handle text generation, translation, and comprehension tasks in multiple languages.
  • Open source and ease of useThe model weights and inference code are fully open source, using the MIT license, making it easy for developers to use and perform secondary development.

The technical principle of MiMo-V2-Flash

  • Hybrid attention architectureIt adopts a hybrid design of sliding window attention (SWA) and global attention (GA), with a window size of 128 tokens. Through learnable attention sink bias technology, it reduces the storage of key-value cache while maintaining the ability to process long texts.
  • Lightweight Multi-Token Prediction (MTP)Equipped with a lightweight MTP module, using a dense feedforward network (FFN) and sliding window attention, each module has only 33 million parameters, significantly improving generation speed and optimizing inference efficiency.
  • High-efficiency pre-trainingIt uses 27 trillion tokens for pre-training, supports native 32K sequence lengths, and the context window can be expanded to 256K. It adopts FP8 mixed precision training to improve training efficiency.
  • Agent reinforcement learningThrough large-scale agent reinforcement learning and multi-teacher online policy distillation (MOPD), it performs well in complex reasoning and agent tasks, improving the model's generalization ability and task adaptability.
  • Inference optimizationIt supports FP8 mixed-precision inference and, combined with the SGLang framework, provides high-performance inference, making it suitable for large-scale deployments and applications.

MiMo-V2-Flash project address

  • Github repositoryhttps://github.com/xiaomimimo/MiMo-V2-Flash
  • HuggingFace model libraryhttps://huggingface.co/XiaomiMiMo/MiMo-V2-Flash
  • Technical Papers: https://github.com/XiaomiMiMo/MiMo-V2-Flash/blob/main/paper.pdf

Application scenarios of MiMo-V2-Flash

  • AI tasks for intelligent agentsSuitable for intelligent agent scenarios that require efficient reasoning and complex task processing, such as automated task execution, multi-turn dialogue management and tool invocation.
  • Long text processingIt can handle long text generation and comprehension tasks, such as long content creation, document summarization, and long text question answering.
  • Code generation and understandingIt supports code generation, code completion, and code comprehension, and is suitable for integration with developer tools and automated programming assistance.
  • Multilingual applicationsIt has multilingual capabilities and can be used for cross-language translation, multilingual content generation, and international application development.
  • Complex reasoning tasksIt excels in complex tasks such as mathematical reasoning, logical reasoning, and scientific problem solving, making it suitable for education and research.
  • Daily AssistantAs a general-purpose AI assistant, it can be used for daily question answering, information retrieval, and personalized content recommendation.
Back