Mellum2 - JetBrains' open-source hybrid expert model

What is Mellum2?

Mellum2 is a new open-source machine learning model from JetBrains for software engineering systems. The model activates only 2.5B parameters per token, is optimized for AI workflows in software engineering, and supports code generation, intelligent routing, sub-agent invocation, and private deployment. With its low latency, high throughput, and low cost, Mellum2 performs exceptionally well in code benchmarks such as LiveCodeBench, making it a high-performance choice for building enterprise-grade AI programming assistants and automated workflows.

Mellum2's main functions

• Intelligent code generationIt is trained based on code and natural language data, and supports code completion, function generation, and refactoring suggestions.
• Dual-mode reasoningIt offers two modes: Thinking (deep thinking) and Non-thinking (rapid response), which can be switched as needed.
• Workflow routing and summarizationIt acts as an intelligent router for AI workflows, responsible for task distribution, document summarization, and intermediate inference.
• Tool Invocation and Agent CollaborationSupports sub-Agent invocation and toolchain integration, and can be embedded into complex automation processes.
• Local private deploymentIt can run independently on the enterprise intranet or local devices without relying on external APIs.

Mellum2's technical principles

• MoE Sparse ArchitectureEmploying a Mixture-of-Experts design with 64 experts and 8 experts activated per token, the total parameter count is 12B, with actual activations of only 2.5B, making the inference cost close to that of a 2.5B dense model. Combining Grouped-Query Attention and sliding window attention significantly reduces memory usage and accelerates decoding. A Multi-Token Prediction head is introduced as an auxiliary pre-training objective to improve performance, acting as a built-in draft model for speculative decoding, further reducing latency.
• Three-stage pre-trainingThe training was conducted on approximately 10.6 trillion tokens, with the data mix gradually transitioning from general web page data to carefully selected code and mathematical content, forming a progressive course. The Muon optimizer was used in conjunction with FP8 mixed precision, and a Warmup-Hold-Decay (linear decay to zero) learning rate scheduling was employed to achieve a balance between training efficiency and model stability.
• Long context expansionThe pre-trained base model expands the context window to 128K using Layer-Selective YaRN, performing positional encoding interpolation only on key layers to avoid performance loss caused by full-layer fine-tuning.

Mellum2's core advantages

• MoE High-Efficiency ArchitectureWith a total of 12B parameters, only 2.5B are activated, significantly reducing inference costs and achieving high throughput and low latency.
• Vertical specializationAbandoning multimodal approaches, focusing on code and natural language, resulting in more accurate and lightweight software engineering solutions.
• Production-grade performanceIn the LiveCodeBench v6 code evaluation, the Thinking mode scored 69.9, outperforming other open-source models in its class.
• Fully open source and commercially viableLicensed under the Apache 2.0 license, allowing for free experimentation, fine-tuning, and large-scale commercial deployment.
• Enterprise privacy friendlySupports fully localized operation, meeting code security and data compliance requirements.

Mellum2's Competitive Product Comparison

Application scenarios of Mellum2

• Intelligent code completion and generationIt provides real-time code completion, function generation, code refactoring, and editing suggestions within the IDE, replacing traditional auto-completion tools.
• Debugging and Error DiagnosisIt helps developers locate bugs, analyze stack traces, and generate fixes, reducing debugging time and costs.
• Multi-step reasoning and complex task decompositionIn the Agent workflow, it acts as an intermediate inference node, breaking down complex requirements into a sequence of executable subtasks.
• Tool calls and function orchestration: Connect to external APIs, databases, or development tools through Function Calling to automate the build, test, and deployment process.
• Conversational programming assistantUse natural language interaction to answer technical questions, explain code logic, recommend best practices, and act as a 24/7 technical advisor.