project
Moonlight-16B-A3B - The open-source MoE model for the dark side of the moon.
Moonlight-16B-A3B is a new Mixture-of-Expert (MoE) model from Moonshot AI, featuring 16 billion total parameters and 3 billion activation parameters. The model is trained using an optimized Muon optimizer...
What is Moonlight-16B-A3B?
Moonlight-16B-A3B is a new Mixture-of-Expert (MoE) model from Moonshot AI, featuring 16 billion total parameters and 3 billion activation parameters. The model is trained using an optimized Muon optimizer, achieving twice the computational efficiency of the traditional AdamW. In terms of performance, Moonlight excels on multiple benchmarks, outperforming other similar models in tasks such as English Language Understanding (MMLU) and HumanEval code generation. The model's training dataset comprises 5.7 trillion tokens, demonstrating higher sample efficiency.
Main functions of Moonlight-16B-A3B
- Efficient language understanding and generationThis model, trained with an optimized Muon optimizer, performs well on a variety of language tasks, such as language understanding, text generation, and code generation.
- Large-scale data trainingMoonlight-16B-A3B was trained using 5.7 trillion tokens of data, supporting highly efficient distributed training.
- High-efficiency optimizer and training efficiencyThe model uses an improved Muon optimizer, which is approximately twice as computationally efficient as the traditional AdamW optimizer. By optimizing weight decay and parameter update ratios, the Muon optimizer demonstrates higher stability and efficiency in large-scale training.
- Low computing costThe model requires only about 52% of the training FLOPs to achieve performance comparable to AdamW training.
- Low activation parameter designThe total number of parameters is 16B, and the activation parameters are only 3B, which significantly reduces the computing resource requirements while maintaining high performance.
Technical Principles of Moonlight-16B-A3B
- Improvements to the Muon optimizerMoonlight-16B-A3B uses an optimized Muon optimizer. The Muon optimizer optimizes model parameters through matrix orthogonalization techniques (such as the Newton-Schulz iteration), significantly improving training efficiency. Compared to the traditional AdamW optimizer, Muon's sample efficiency is approximately 2x higher, demonstrating greater stability and efficiency in large-scale training.
- Weight decay and update adjustmentTo improve Muon's performance in large-scale model training, the development team introduced a weight decay mechanism, adjusting the update scale of each parameter. This allows Muon to be directly applied to large-scale model training without requiring hyperparameter tuning.
- Distributed implementationThe training of Moonlight-16B-A3B employs a distributed optimization technique based on ZeRO-1. This reduces memory overhead and communication costs, enabling the model to be trained efficiently in a large-scale distributed environment.
- Model architecture and training dataMoonlight-16B-A3B is a 16B-parameter MoE model with 3B activation parameters, trained using 5.7 trillion labels. It significantly reduces computational resource requirements while maintaining high performance.
- Performance optimizationWith its optimized Muon optimizer and efficient distributed training, Moonlight-16B-A3B outperforms other models of similar size in multiple benchmark tests.
Project address for Moonlight-16B-A3B
- Github repository:https://github.com/MoonshotAI/Moonlight
- HuggingFace model library:https://huggingface.co/moonshotai/Moonlight-16B-A3B
- Technical Papers:https://github.com/MoonshotAI/Moonlight/blob/master/Moonlight.pdf
Performance of Moonlight-16B-A3B
- Language comprehension task
- MMLU (Multilingual Language Understanding)The performance of Moonlight-16B-A3B reached 70.0%, which is significantly better than LLAMA3-3B (54.75%) and Qwen2.5-3B (65.6%).
- BBH (BoolQ Benchmark)Moonlight achieved a score of 65.2% in the task, outperforming other similar models.
- TriviaQAMoonlight achieved a score of 66.3%, which is close to or surpasses other models.
- Code generation task
- HumanEvalMoonlight achieved a performance improvement of 48.1% in code generation tasks, outperforming LLAMA3-3B (28.0%) and Qwen2.5-3B (42.1%).
- MBPP (Mini-Benchmark for Program Synthesis)Moonlight's performance was 63.8%, significantly outperforming other models.
- Mathematical reasoning task
- GSM8KMoonlight achieved a score of 77.4% in this task, close to the best performance of Qwen2.5-3B (79.1%).
- MATHMoonlight's performance is 45.3%, which is better than other similar models.
- CMathMoonlight achieved 81.1% performance, outperforming Qwen2.5-3B (80.0%).
- Chinese task
- C-EvalMoonlight's performance was 77.2%, which is better than Qwen2.5-3B (75.0%).
- CMMLUMoonlight achieved a performance of 78.2%, outperforming other similar models.
- computational efficiency
- Training efficiencyThe Muon optimizer used by Moonlight is twice as computationally efficient as AdamW, requiring only about 52% of the training FLOPs to achieve performance comparable to AdamW.
- Memory and communication efficiencyThrough its improved distributed implementation, Moonlight demonstrates greater memory and communication efficiency during large-scale training.
| Benchmark (Metric) | Llama3.2-3B | Qwen2.5-3B | DSV2-Lite | Moonlight | |
|---|---|---|---|---|---|
| Activated Param† | 2.81B | 2.77B | 2.24B | 2.24B | |
| Total Params† | 2.81B | 2.77B | 15.29B | 15.29B | |
| Training Tokens | 9T | 18T | 5.7T | 5.7T | |
| Optimizer | AdamW | * | AdamW | Muon | |
| English | MMLU | 54.75 | 65.6 | 58.3 | 70.0 |
| MMLU-pro | 25.0 | 34.6 | 25.5 | 42.4 | |
| BBH | 46.8 | 56.3 | 44.1 | 65.2 | |
| TriviaQA‡ | 59.6 | 51.1 | 65.1 | 66.3 | |
| Code | HumanEval | 28.0 | 42.1 | 29.9 | 48.1 |
| MBPP | 48.7 | 57.1 | 43.2 | 63.8 | |
| Math | GSM8K | 34.0 | 79.1 | 41.1 | 77.4 |
| MATH | 8.5 | 42.6 | 17.1 | 45.3 | |
| CMath | – | 80.0 | 58.4 | 81.1 | |
| Chinese | C-Eval | – | 75.0 | 60.3 | 77.2 |
| CMMLU | – | 75.0 | 64.3 | 78.2 |
Application scenarios of Moonlight-16B-A3B
- Education and ResearchIn academic research, Moonlight can help researchers quickly understand and analyze large amounts of literature.
- Software developmentDevelopers can use Moonlight to automatically generate code snippets, improving development efficiency.
- Research and EngineeringResearchers and engineers can use Moonlight to solve mathematical problems in real-world applications.
- Chinese content creationIn the field of content creation, Moonlight can help creators generate high-quality Chinese content.
- Large-scale model trainingIn scenarios requiring large-scale model training, Moonlight can significantly reduce computational resource requirements and improve training efficiency.