Instella-MoE - AMD's open-source hybrid expert language model series

What is Instella-MoE?

Instella-MoE is an open-source hybrid expert language model series from AMD, with a total of 16B parameters and 2.8B activation parameters, employing a 27-layer decoder architecture. The model is trained from scratch on AMD Instinct MI300X/MI325X GPUs and the ROCm software stack, providing six fully open-source versions: pre-trained, long context base, SFT, DPO, and RL optimization, achieving efficient collaboration between training and inference on AMD hardware.

Main functions of Instella-MoE

• Multi-stage model seriesIt offers six versions: pre-training, mid-training, long-context base, SFT, DPO, and RL optimization, covering the entire chain of needs from base to dialogue.
• High-efficiency MoE inferenceInference can be completed by activating only the 2.8B parameter, achieving performance close to that of a larger dense model while maintaining a small computational overhead.
• Long context supportThe Base version supports a 64K context length, enabling it to handle long document comprehension and reasoning tasks.
• Instruction compliance and thought chainThe SFT version implements instruction tracing and chained reasoning, while the Think version further improves response quality through reinforcement learning.
• AMD native optimizationBased on the ROCm ecosystem and the SGLang inference framework, it achieves efficient training and inference on AMD hardware through deep optimization.

Instella-MoE Technical Principles

• MoE Architecture and Attention MechanismThe model employs a hybrid expert architecture, activating only 2.8B parameters in each forward propagation out of a total of 16B parameters. A gated routing mechanism dynamically allocates tokens to different expert networks, achieving decoupling between high capacity and low inference cost. Furthermore, gated multi-head latent attention is introduced, adding gating control to the latent attention space to enhance long sequence modeling capabilities and reduce key-value caching overhead.
• AMD native training systemTraining is entirely based on the AMD ROCm™ software stack, using the Primus training framework and the Miles RL framework at the bottom layer. The pre-training phase utilizes FarSkip-Collective technology to achieve communication and computation overlap under expert parallelism, improving pre-training speed by 12.7%. The model is trained from scratch on AMD Instinct MI300X/MI325X GPUs, fully leveraging the interconnect bandwidth advantages of AMD hardware.
• Post-training and reinforcement learningPost-training employs a four-stage pipeline: SFT → DPO → two-stage RL. The RL stage is divided into: Stage 1 performs instruction compliance-specific reinforcement learning; Stage 2 uses MOPD, which routes IF Prompts to the IF-RL Teacher and other Prompts to the DPO Teacher via a Domain Router, using token-level inverse KL divergence to constrain student model policy updates, thereby enhancing instruction compliance and inference quality while maintaining general capabilities.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use Instella-MoE

• Obtain model weightsAccessing the Hugging Face model collection (amd/instella-moeDownload the required version (Base / SFT / DPO / Think) from the GitHub repository or GitHub repository.
• Environmental preparationInstall the AMD ROCm driver and PyTorch for ROCm, or use a Docker image that supports ROCm.
• Loading ModelLoad model weights using the Transformers library or the SGLang inference framework. SGLang can fully leverage the inference acceleration capabilities of AMD hardware.
• Execution reasoningSelect the appropriate version based on the task type: Base is used for text continuation and embedding, while SFT / DPO / Think are used for dialogue and instruction tasks.
• Fine-tuning and deployment: Continue training using your own data based on the Base model, or deploy as an API service via vLLM/SGLang.

Instella-MoE's core advantages

• Full-chain open source and transparentThe entire model, from pre-training to RL, is open source, with training data and code transparently available.
• Native AMD ecosystem compatibilityBased on ROCm and self-developed GPU training, it achieves optimal inference efficiency on AMD Instinct series hardware, with TTFT reduced by up to 39.2%.
• Activation parameters are highly efficientWith a total of 16B parameters, only 2.8B are activated, resulting in significantly lower inference costs compared to denser models with similar performance.
• Advanced post-training technologyMOPD multi-teacher distillation combined with token-level reverse KL performs exceptionally well in instruction compliance and mathematical reasoning.
• Communication computation deep overlapFarSkip-Collective and SGLang experts work in parallel to achieve communication hiding, resulting in superior training and inference efficiency.

Instella-MoE project address

• Project official website：https://rocm.blogs.amd.com/artificial-intelligence/instella-moe/README.html
• GitHub repository：https://github.com/AMD-AGI/Instella-MoE
• HuggingFace model library：https://huggingface.co/collections/amd/instella-moe

Comparison of Instella-MoE with similar products

Application scenarios of Instella-MoE

• Enterprise privatization deploymentDeploy dialogue and inference services on a local GPU cluster based on the AMD ROCm ecosystem to meet data compliance requirements.
• Long document analysisIt can handle long text understanding and summarization tasks such as legal contracts, scientific research papers, and financial statements with 64K long context capabilities.
• Code generation assistanceThe SFT/Think version supports programming instruction following and can be used with IDE plugins or code review tools.
• Scientific Research and TeachingThe fully open-source pipeline is suitable for academic research on MoE architecture, reproduction of training methods, and teaching of large-scale model courses.
• Edge and Hybrid Cloud InferenceLow activation parameter count is suitable for efficient inference in resource-constrained environments or as a base model for hybrid cloud AI services.