Ling-V2 - A large-scale language model series launched by Ant Financial.

What is Ling-V2?

Ling-V2 is a family of large-scale language models based on the MoE architecture, launched by the Ant Financial team. The first version, Ling-mini-2.0, has 16 billion total parameters, with only 1.4 billion parameters activated per input tag. The model is trained on 20 trillion high-quality data tags and enhanced through multi-stage supervised fine-tuning and reinforcement learning, demonstrating excellent performance in complex inference and instruction following. Ling-mini-2.0 achieves a 7x equivalent dense performance leverage using a MoE architecture with a 1/32 activation ratio, boasting fast generation speed, high training and inference efficiency, and an open-source FP8 efficient training solution. It provides multiple pre-training checkpoints to support continuous training, making it an ideal starting point for MoE research.

Main functions of Ling-V2

• Strong reasoning abilityIt performs exceptionally well in coding, mathematical, and cross-domain knowledge-intensive reasoning tasks, demonstrating superior reasoning capabilities compared to dense models with fewer than 1 billion data points and larger-scale MoE models.
• High efficiencyThe MoE architecture, with an activation ratio of 1/32, achieves a 7x equivalent performance leverage in dense mode. Activating 1.4 billion parameters can provide the performance equivalent to 700-800 million dense models. In simple question-answering scenarios, the generation speed can reach 300+ tokens/s, and the relative speed improvement can reach more than 7x when processing 128K context length.
• Highly efficient training solutionsThe entire training process uses FP8 mixed precision training. An open-source FP8 training solution is provided, based on tile/blockwise FP8 scaling, and further introduces an FP8 optimizer to achieve extreme memory optimization. On 8/16/32 80G GPUs, the training throughput is significantly improved compared to LLaMA 3.1 8B and Qwen3 8B.
• Open source strategyIn addition to releasing the trained version, five pre-trained checkpoints have been open-sourced to support more in-depth research and wider applications.

The technical principle of Ling-V2

• MoE architectureBased on the Hybrid Expert (MoE) architecture, the model is decomposed into multiple expert networks, and only some experts are activated on each input label to achieve sparsity of the model, thereby improving computational efficiency while maintaining high performance.
• Optimized designEmpirical optimizations were made in aspects such as expert granularity, shared expert ratio, attention ratio, no-assistance loss + sigmoid routing strategy, MTP loss, QK-Norm, and semi-RoPE to further improve the performance and efficiency of the model.
• FP8 Mixed Precision TrainingDuring training, FP8 mixed precision is used. Compared with BF16, the loss curve and downstream benchmark performance are almost identical in experiments with more than 1 trillion training labels. At the same time, the FP8 training solution is open source, which supports the community to perform efficient continuous pre-training and fine-tuning with limited computing resources.
• Multi-stage trainingThe model is trained on over 20 trillion high-quality labeled data points and enhanced through multi-stage supervised fine-tuning and reinforcement learning, achieving significant improvements in complex reasoning and instruction following.

Ling-V2 project address

• GitHub repositoryhttps://github.com/inclusionAI/Ling-V2
• HuggingFace model library: https://huggingface.co/collections/inclusionAI/ling-v2-68bf1dd2fc34c306c1fa6f86

Application scenarios of Ling-V2

• Natural Language Processing (NLP) TasksIt efficiently handles various natural language processing tasks, such as text classification, sentiment analysis, and machine translation, providing accurate and efficient solutions.
• Intelligent Customer ServiceAs the core engine of intelligent customer service, it quickly responds to user questions, provides accurate answers, and improves user experience and customer service efficiency.
• Content creationIt assists in content creation, generating high-quality text content such as news reports, creative writing, and advertising copy, helping creators improve their efficiency and quality.
• EducationUsed in the education field, such as intelligent tutoring, automatic homework grading, and personalized learning plans, it provides personalized educational support for students and teachers.
• HealthcareLing-V2 can process medical text data, assisting doctors in case analysis, medical literature retrieval, etc., and improving the accuracy and efficiency of medical decision-making.