MiniMax-01 - MiniMax's new open-source model series

What is MiniMax-01?

MiniMax-01 is a brand-new model series launched by MiniMax, including the basic language large-scale model MiniMax-Text-01 and the visual multimodal large-scale model MiniMax-VL-01. MiniMax-01 is the first to implement a linear attention mechanism on a large scale, breaking the limitations of the traditional Transformer architecture. It boasts 456 billion parameters and 45.9 billion activations per session, achieving performance comparable to top international models and efficiently handling contexts with up to 4 million tokens globally. The MiniMax-01 series offers API services at an extremely cost-effective price, with low standard pricing, and excels in long-text tasks, multimodal understanding, and other areas.

MiniMax-01 performance

• Parameter quantity and activation quantityThe model has a parameter count of up to 456 billion, with 45.9 billion parameters activated in a single session, and its overall performance is comparable to top overseas models.
• Extremely long context processing capabilityIt can efficiently handle the context of the world's longest collection of 4 million tokens, which is 32 times faster than GPT-4o and 20 times faster than Claude-3.5-Sonnet.
• Mainstream evaluation resultsIt matches the two most advanced models recognized overseas, GPT-4o-1120 and Claude-3.5-Sonnet-1022, in most tasks.
• Advantages of long text tasksIn long text tasks, it exhibits the slowest performance degradation and significantly outperforms Google's Gemini model.
• Efficiency in handling long inputsIt has near-linear complexity, which is a significant advantage compared to other top global models.

Main functions of MiniMax-01

• Language comprehension and generation:
  • Text SummaryIt can extract key information from long articles and generate concise and accurate summaries.
  • translateTo achieve accurate conversion between different languages while maintaining semantic integrity and accuracy.
  • Q&AOpen-domain question answering answers questions based on given text or existing knowledge. Closed-domain question answering targets specific text content, while open-domain question answering covers a broader range of knowledge.
• Multimodal understanding:
  • Image and text matchingThis function determines whether the text accurately describes the content of an image, and is used in scenarios such as image annotation and content review.
  • Image description generationGenerates fluent and accurate descriptive text from images to help understand the elements and layout relationships within the images.
  • Visual Q&AAnswer questions involving visual content by combining information from the images.
• Long context processingIt efficiently handles contexts of up to 4 million tokens, meeting practical needs such as reading professional books, assisting with programming projects, and analyzing long documents, providing fundamental capabilities for building complex agent systems.

The technical principles of MiniMax-01

•  Linear attention mechanism
  • Core ideaThe computationally intensive self-attention mechanism in the traditional Transformer is transformed into linear complexity using mathematical techniques, enabling the model to efficiently process long sequences.
  • Lightning AttentionAn optimized implementation of linear attention, using block-based techniques to divide attention computation into intra-block and inter-block computations. Intra-block computation uses left multiplication, while inter-block computation uses right multiplication, maintaining the overall computational complexity linear and improving the efficiency of processing long sequences.
• Hybrid architecture
  • Architecture DesignIn this model, 7 out of every 8 layers use linear attention, and 1 layer uses traditional SoftMax attention. This combines the efficiency of linear attention with the advantages of SoftMax attention, making it particularly effective for tasks with long contexts.
  • Performance verificationThis demonstrates the performance and efficiency advantages of the hybrid architecture in long-context tasks, such as in long-context retrieval tasks, where MiniMax-Text-01 exhibits the slowest performance degradation as the input length increases.
• Model optimization and training
  • MoE optimizationTo reduce communication overhead in the MoE architecture, a token grouping overlap scheme is introduced, where communication overlaps with token processing for different expert groups, reducing waiting time. An EP-ETP overlap strategy is also introduced to improve resource utilization, reduce communication overhead, and enhance training efficiency.
  • Long context optimizationThis paper employs a "data packing" technique to reduce computational waste. The Varlen Ring Attention algorithm is designed to directly apply ring attention calculations to the packed sequences, avoiding over-padded processing. The LASP+ algorithm is proposed to optimize the parallel computation process for linear attention sequences, achieving fully parallel computation and improving system efficiency.
  • Lightning Attention Inference OptimizationBased on strategies such as batch kernel fusion, separate pre-filling and decoding execution, multi-level filling, and strided batched matmul extension, memory access operations are reduced and inference speed is improved.

MiniMax-01 project address

• Project official website:
  • MiniMax Open Platform:https://www.minimaxi.com
  • MiniMax Open Platform Overseas Version:https://www.minimaxi.com/en
• GitHub repository:https://github.com/MiniMax-AI
• Technical Papers:https://filecdn.minimax.chat/_Arxiv_MiniMax_01

MiniMax-01 API pricing

• MiniMax-Text-01Powerful long text processing capability, context length 1000k, input 0.001 yuan/thousand tokens, output 0.008 yuan/thousand tokens.
• MiniMax-VL-01Powerful visual understanding capabilities, context length 1000k, input 0.001 yuan/thousand tokens, output 0.008 yuan/thousand tokens.

Application scenarios of MiniMax-01

• Enterprise usersIt covers content creators, marketers, customer service teams, technology teams, and knowledge managers, improving the efficiency of content creation, marketing effectiveness, customer satisfaction, project development, and knowledge sharing.
• Educators and studentsTeachers generate teaching materials, which students use to aid their learning and improve the quality of teaching and learning.
• Creative workers: Writers, poets, lyricists, designers and artists, to gain creative inspiration, assist in creative writing and art design, and stimulate creative potential.
• Researchers and scholarsIt is suitable for processing academic papers and conducting literature reviews, thereby improving the efficiency and depth of scientific research.
• Developers and engineersNatural language processing developers, multimodal application developers, and system integration engineers develop customized applications to improve system intelligence.