TeleChat3 - China Telecom's MoE Semantic Model with Hundreds of Billions of Parameters

What is TeleChat3?

TeleChat3 is a fine-grained MoE semantic model with hundreds of billions of parameters independently developed by the TeleAI Research Institute of China Telecom. Trained using domestically produced computing power, it supports tasks such as multi-turn dialogue, text creation, and code generation. The model contains 105B parameters (4.7B actually activated) and achieves accurate knowledge retrieval through 192 routing experts, excelling at handling complex tasks such as game design and web development. Its key feature is its fully domestically produced technology across the entire supply chain, achieving a training efficiency of 93%. It has been applied to scenarios such as intelligent customer service and government Q&A, and a 36B dense model version has been open-sourced.

TeleChat3's main functions

• Code generation and optimizationIt can efficiently break down task requirements, integrate multiple code capabilities, and deliver complete and runnable code in one go. It can perform various tasks such as code completion, code modification, website and web game generation, saving a lot of time spent on manual debugging, running smoothly and providing accurate results.
• Knowledge Questions and Logical ReasoningIt has improved its capabilities in areas such as knowledge-based Q&A and logical reasoning, enabling it to better understand and answer users' questions and provide accurate knowledge information and logical reasoning results.
• Text creation and semantic understandingIt possesses text creation capabilities, can generate high-quality text content, and has undergone targeted optimization in semantic understanding, enabling it to better understand the text information input by users.
• Role-playing and multi-turn dialogueIt supports role-playing functionality, allowing dialogue based on user-defined roles and scenarios. In multi-turn dialogues, it can automatically process historical information to maintain the coherence and consistency of the conversation.
• Introducing "Thinking Mode"By adding specific guiding symbols to the dialogue template, the model can automatically generate intermediate reasoning steps, significantly improving the logic and accuracy in complex tasks and making the AI reasoning process traceable.

TeleChat3's technical principles

• Model ArchitectureTeleChat3 adopts a fine-grained Mixture-of-Experts (MoE) architecture. For example, the TeleChat3-105B-A4.7-Thinking model contains 1 shared expert and 192 routing experts, with 4 experts activated each time.
• Reasoning mechanismTeleChat3 supports long text inference optimization. Through NTK-aware extrapolation and attention scaling, it can extrapolate models that have been fine-tuned at shorter lengths to longer lengths during inference.
• Training and optimizationThe series of models are trained entirely on domestic computing power (such as the Shanghai Lingang domestic 10,000-calorie computing power pool), with a basic data volume of 15T tokens. At the same time, the models have achieved deep adaptation to the Ascend Atlas 800T A2 training server and the Ascend MindSpore framework.
• Thinking patternsTeleChat3 fully supports the Thinking mode, which enhances the logic and accuracy of complex tasks by adding specific guiding symbols to the chat template to generate the reasoning process.
• Location encoding and attention mechanismThe model employs Rotary Embedding to inject positional information into tokens. By rotating the query (Q) and key (K), the model captures the sequential dependencies of the sequence. Simultaneously, it supports a causal self-attention mechanism to ensure that the prediction at each position depends only on the preceding token.
• Nonlinear enhancementThe model includes a multilayer perceptron (MLP) module, which performs non-linear transformations on the output of the attention layer to enhance the model's expressive power.

TeleChat3 project address

• Github repository: https://github.com/Tele-AI/TeleChat3
• ModelScope:
  • https://www.modelscope.cn/models/TeleAI/TeleChat3-105B-A4.7B-Thinking
  • https://www.modelscope.cn/models/TeleAI/TeleChat3-36B-Thinking

Application scenarios of TeleChat3

• Code generation and development assistanceIt can be used for code completion, code modification, and generating websites and web games, helping developers quickly complete programming tasks and improve development efficiency.
• Knowledge Q&A and EducationIt performs well in tasks such as knowledge quizzes and logical reasoning, providing users with accurate knowledge information. It is suitable for the education field, assisting in learning and answering questions.
• Text creation and content generationIt can generate high-quality text content that can be used in writing, copywriting, news reporting and other scenarios to meet different text creation needs.
• Multi-round dialogue and customer serviceIt supports multi-turn dialogues, can automatically process historical information, maintain dialogue continuity, and is suitable for scenarios such as intelligent customer service and virtual assistants.
• Role-playing and interactive entertainmentIt supports role-playing functionality, allowing dialogue based on user-defined roles and scenarios, and can be applied to games, interactive entertainment, and other fields.