MiniCPM 4.1 - A hybrid thinking model launched by Wallfacer Intelligence

What is MiniCPM 4.1?

MiniCPM 4.1 is a hybrid thinking model developed by Wallfacer. It employs the InfLLM v2 sparse attention architecture, requiring each word to be correlated with less than 5% of words, significantly reducing the overhead of long text processing. In 128K long text scenarios, it drastically reduces cache storage space compared to traditional models. MiniCPM 4.1 supports an efficient dual-frequency switching mechanism, automatically switching attention modes based on task type to balance computational efficiency and output accuracy. It achieves top performance among models of similar size on multiple benchmarks, reaching best-in-class overall capabilities. MiniCPM 4.1 provides multiple deployment formats, such as GPTQ and AutoAWQ, facilitating efficient deployment on various edge devices.

Main functions of MiniCPM 4.1

• High-efficiency sparse architectureIt adopts the InfLLM v2 sparse attention architecture, where each word only needs to be calculated to be related to less than 5% of words. In the case of 128K long text, it significantly reduces the cache storage space compared with traditional models and significantly reduces the overhead of long text processing.
• Dual-frequency shifting mechanismAutomatically switch attention modes based on task type: use sparse for long texts and dense for short texts to balance computational efficiency and output accuracy.
• Excellent overall performanceIt achieved first place among models of the same size on multiple evaluation benchmarks, and its overall capabilities reached the best level in its class, demonstrating outstanding performance in knowledge, reasoning, programming, and instruction compliance.
• End-side friendlyIn tests such as LiveCodeBench and AIME, the inference speed is more than 3 times faster than open-source models of the same size. In scenarios with 128K long text, only 25% of the cache storage space is required, relieving pressure on the computing power of the edge.
• Multiple deployment formatsIt offers multiple deployment formats, such as GPTQ and AutoAWQ, to facilitate efficient deployment on different edge devices.

Technical Principles of MiniCPM 4.1

• Sparse attention mechanismMiniCPM 4.1 adopts the InfLLM v2 sparse attention architecture, which only needs to calculate the relevance of each word to less than 5% of words, greatly reducing the computational complexity and memory overhead of long text processing.
• Dual-frequency shifting mechanismThe attention mode is automatically switched according to the characteristics of the task. Sparse attention is enabled for long text tasks to reduce computational complexity, while dense attention is switched for short text tasks to ensure accuracy and speed.
• High-efficiency reasoning frameworkThe self-developed CPM.cu inference framework integrates technologies such as sparse attention, model quantization, and speculative sampling to achieve efficient pre-filling and decoding, thereby improving edge inference efficiency.
• Long text processing optimizationBy processing text in blocks and regions, it intelligently selects key areas for attention calculation, significantly reducing the computational load of long text processing and supporting text processing of 128K or even longer.
• Data and Training OptimizationBy employing high-quality pre-trained datasets and supervised fine-tuning datasets, combined with efficient training algorithms such as BitCPM extreme ternary quantization technology, we achieve extreme compression of model parameters, thereby improving training efficiency and model performance.

MiniCPM 4.1 project address

• Github repositoryhttps://github.com/openbmb/minicpm
• HuggingFace model libraryhttps://huggingface.co/openbmb/MiniCPM4.1-8B

Application Scenarios of MiniCPM 4.1

• End-side device applicationsMiniCPM 4.1 is designed for edge devices and is suitable for devices such as smartphones, tablets, and smart speakers. It can run efficiently on the device, protect user privacy, and reduce reliance on cloud computing.
• Long text processingSparse attention mechanisms and long text optimization can efficiently handle long text tasks, such as long document generation, story creation, and technical document writing, and are suitable for scenarios that require processing a large amount of contextual information.
• Knowledge Q&A and EducationWith its powerful knowledge understanding and reasoning capabilities, it can be used in educational applications such as knowledge question-and-answer systems and intelligent tutoring tools to help users quickly obtain accurate information and assist in learning and research.
• Programming aidsIt excels in code generation, code completion, and programming problem solving, and can serve as a programming assistant for developers, improving programming efficiency and quality.
• Content creationIt can assist users in various content creation tasks such as creative writing, copywriting, and news reporting, providing inspiration and writing support.