JoyAI-LLM-Flash - JD.com's open-source hybrid expert architecture model

What is JoyAI-LLM-Flash?

JoyAI-LLM-Flash is a medium-sized instruction model open-sourced by JD.com. It employs a hybrid expert (MoE) architecture with a total of 48 bytes of parameters and only 3 bytes of activation parameters, supporting a 128K ultra-long context. The model innovatively introduces the FiberPO optimization framework—the first application of fiber bundle theory to reinforcement learning—combining it with the Muon optimizer for SFT, DPO, and RL training. It also utilizes dense multi-token prediction (MTP) technology, achieving a 1.3-1.7x throughput improvement compared to non-MTP versions. Based on 20 trillion tokens of pre-trained data, JoyAI-LLM-Flash demonstrates outstanding performance in cutting-edge knowledge understanding, logical reasoning, code generation, and agent interaction tasks, making it suitable for efficient edge-side reasoning, enterprise-level agent development, and long text processing scenarios.

Main functions of JoyAI-LLM-Flash

• Long context processingSupports 128K token context length, enabling it to handle long documents, long conversations, and complex multi-turn interaction tasks.
• High-efficiency MoE inferenceThe total number of parameters is 48B, with only 3B of activation parameters. There are 256 expert dynamic routes, and 8 experts are activated per token, balancing performance and low inference cost.
• Multi-Token Prediction (MTP)Employing dense MTP technology, the throughput is increased by 1.3-1.7 times compared to the non-MTP version, significantly improving generation efficiency.
• Code generation capabilityBased on 20 trillion tokens pre-trained, it excels in logical reasoning and code generation, and supports multiple programming languages to assist in development.
• Intelligent agent interactionOptimized for Agent scenarios, it supports complex task planning, tool invocation, and multi-step inference.
• Understanding cutting-edge knowledgeIt covers a wide range of knowledge fields and has powerful semantic understanding and question-answering capabilities.
• Instruction compliance optimizationThrough multi-stage training using SFT, DPO, and RL, it accurately understands and executes user commands.

The technical principle of JoyAI-LLM-Flash

• Hybrid Expert Architecture (MoE)It adopts a sparse activation MoE design with a total of 48B parameters, but only 3B parameters are activated per token. Through dynamic routing of 256 expert modules (8 experts are selected per token), it significantly reduces inference costs while maintaining high performance.
• Dense Multi-Token Prediction (Dense MTP)It innovatively introduces a dense MTP mechanism, which predicts multiple future tokens in parallel during a single forward propagation, solving the instability problem of traditional models when scaling up, and improving throughput by 1.3-1.7 times.
• FiberPO optimization frameworkThis study is the first to introduce fiber bundle theory into reinforcement learning optimization, combining it with the Muon optimizer for parameter updates to improve training stability and convergence efficiency.
• Multi-stage training strategyThe training process employs a three-stage approach: SFT (Supervised Fine-tuning), DPO (Direct Preference Optimization), and RL (Reinforcement Learning) to progressively optimize the model's instruction compliance and output quality.
• MLA attention mechanismIt uses multi-head latent attention, with 2048 hidden dimensions, supports 32-head attention, and efficiently handles long sequence dependencies.
• SwiGLU activation functionSwiGLU is used as a non-linear activation, combined with a 129K vocabulary size, to improve the model's expressive power and training stability.

JoyAI-LLM-Flash project address

• HuggingFace model library: https://huggingface.co/jdopensource/JoyAI-LLM-Flash

Application Scenarios of JoyAI-LLM-Flash

• Efficient edge-side inferenceWith only 3B activation parameters, it is suitable for deployment on resource-constrained edge scenarios such as mobile phones and IoT devices to realize localized AI assistants.
• Intelligent Customer Service SystemIt supports 128K ultra-long contexts, can handle complex multi-turn dialogues, and is suitable for large-scale customer service automation in e-commerce, finance and other fields.
• Code-assisted developmentProficient in logical reasoning and code generation, providing developers with real-time code completion, bug fixing, and technical document interpretation.
• Enterprise-grade Agent PlatformOptimized for intelligent agent interaction, it supports task planning, tool invocation, and multi-step reasoning, and is suitable for scenarios such as RPA and automated office.
• Long document processing128K contextual capabilities support professional scenarios that require processing large amounts of text, such as paper analysis, contract review, and research report summaries.
• Content creation assistanceBased on 20 trillion tokens pre-trained, it can assist in writing marketing copy, product descriptions, press releases and other business content.