Ling-1T - A large-scale language model open-sourced by Ant Group

What is Ling-1T?

Ling-1T is a trillion-parameter language model open-sourced by Ant Group, positioned as a "flagship non-thinking model." Based on the MoE (Mixture of Experts) architecture, it boasts one trillion parameters, activating approximately 51 billion parameters per inference iteration, and supports 128K context lengths, making it suitable for handling long documents. Ling-1T focuses on delivering high-quality inference results directly with a limited number of output tokens, achieving extremely high inference efficiency. The model performs exceptionally well in tasks such as programming, mathematical reasoning, knowledge understanding, and creative writing, placing it among the top tier of open-source models.

Main functions of Ling-1T

• Efficient ReasoningIt provides high-quality reasoning results directly with a limited number of output tokens, resulting in extremely high reasoning efficiency, making it suitable for quickly solving problems.
• Long text processingIt supports 128K context length, can handle long document tasks, and is suitable for complex reasoning in fields such as law, finance, and scientific research.
• Creative WritingThe model can generate creative copy, scripts, poems, etc., to meet the needs of content marketing, advertising copywriting, and other creative tasks.
• Multilingual supportIt supports tasks that process English and other languages, and has a certain level of multilingual capability.
• Multitasking capabilityIt performs well in tasks such as programming assistance, mathematical problem solving, knowledge quizzes, and multi-turn dialogues, and can generate high-quality code and designs.
• Application integrationIt supports integration into various tools, such as payment software, financial management assistants, and health assistants, to enhance the level of intelligence.

Technical Principles of Ling-1T

• MoE architectureBased on the Mixture of Experts (MoE) architecture, it has a total of 1 trillion parameters and includes 256 experts. Only about 51 billion parameters are activated per inference, significantly reducing inference costs while maintaining high performance. The first few layers use a dense structure, then switch to MoE to reduce the load imbalance in shallower networks.
• High reasoning density corpusThe model was pre-trained using a high-quality, high-inference-density corpus of over 20T+ tokens to ensure strong logical density and depth of thought. Pre-training consisted of three stages:
  • Pretrain Stage 110T of high knowledge-density corpus.
  • Pretrain Stage 210T high-inference-density corpus.
  • Mid-trainingExpand the context to 128K and add the thought chain corpus.
• High-efficiency trainingThe entire training process uses FP8 precision. Compared with BF16, FP8 can significantly save GPU memory and improve training speed. In the comparative experiment with 1T tokens, the loss bias is only 0.1%.
• LPO optimizationUsing the Linguistics-Unit Policy Optimization (LPO) method, sentences are used as the optimization unit, which is more in line with semantic logic and improves the model's reasoning ability and generation quality.

Ling-1T project address

• HuggingFace model libraryhttps://huggingface.co/inclusionAI/Ling-1T

Application scenarios of Ling-1T

• Programming aidsThe model can generate high-quality code snippets, helping developers quickly implement functions and improve programming efficiency.
• Mathematical Problem SolvingIt excels in mathematical reasoning and problem-solving, providing support and assistance in solving complex mathematical problems, such as competition problems.
• Knowledge Q&AThe model possesses a strong knowledge understanding capability, can accurately answer various knowledge-based questions, and provide reliable information.
• Creative WritingThe model can generate creative copy, scripts, poems, etc., to meet the needs of content creation and advertising copywriting.