LLaDA 2.0 - A Discrete Diffusion Large Language Model Open Source by Ant Group

What is LLaDA 2.0?

LLaDA 2.0 is an open-source Discrete Diffusion Large Language Model (dLLM) from Ant Group, available in 16B (mini) and 100B (flash) versions. It breaks through the scalability bottleneck of diffusion models, scaling the parameter size to the 100B level for the first time. Through its innovative Warmup-Stable-Decay (WSD) continuous pre-training strategy, LLaDA 2.0 achieves a smooth transition from autoregressive (AR) models to diffusion models, inheriting the knowledge of AR models and avoiding the high cost of training from scratch.

Main features of LLaDA 2.0

• Large-scale parameter expansionLLaDA 2.0 offers two versions, 16B and 100B, and is currently the largest diffusion language model, breaking through the limitation of diffusion models being difficult to scale on a large scale.
• High-efficiency reasoning accelerationWith the help of a parallel decoding mechanism, LLaDA 2.0 achieves an inference speed of up to 535 tokens/s, which is 2.1 times faster than the same level of autoregressive model, significantly improving the generation efficiency.
• Smooth transition and knowledge inheritanceThe Warmup-Stable-Decay (WSD) strategy is adopted to achieve a smooth transition from the autoregressive model to the diffusion model, inheriting the knowledge of the AR model and avoiding the cost of training from scratch.
• Excellent performanceIt demonstrates significant advantages in structured generation tasks such as code generation, mathematical reasoning, and agent tasks, while maintaining parity with open-source AR models in other fields.
• Fully open source and sharedThe model weights (16B/100B) and related training code for LLaDA 2.0 have been fully open-sourced on Hugging Face, making it convenient for developers to use and conduct further research.

Technical principles of LLaDA 2.0

• Diffusion Model ArchitectureLLaDA 2.0 is based on the diffusion model and generates text through progressive denoising. Unlike traditional autoregressive generation methods, it can decode multiple tags in parallel, thereby improving the generation speed.
• Hybrid Expert Architecture (MoE)By combining the MoE architecture, only a portion of the parameters (approximately 1.44 billion) are activated in each inference, significantly reducing computational costs while maintaining high performance.
• Warmup-Stable-Decay (WSD) StrategyThrough three-stage pre-training (gradually increasing block size, full sequence training, and decreasing block size), a smooth transition from autoregressive model to diffusion model is achieved, inheriting existing knowledge and optimizing inference efficiency.
• Confidence-Aware Parallel Training (CAP)During parallel decoding, an auxiliary loss function is used to reward "correctly predicted and highly confident" labels, thereby improving decoding efficiency and enabling high-speed inference.
• Diffusion Model Version DPOBy using the evidence lower bound (ELBO) to approximate conditional probability, preference learning (DPO) is adapted to the diffusion model to optimize the model output to conform to human preferences.
• Document-level attention maskDuring multi-document concatenation training, a document-level attention mask is designed to avoid erroneous connections between irrelevant documents and ensure the semantic coherence of long texts.

LLaDA 2.0 project address

Application scenarios of LLaDA 2.0

• Code generationLLaDA 2.0 performs exceptionally well in code generation tasks, producing high-quality code snippets to help developers quickly implement features.
• Mathematical reasoningThe model demonstrates powerful capabilities in solving mathematical problems and complex reasoning tasks, and can be used in fields such as education and scientific research.
• Intelligent agent taskIt supports complex agent invocation and long text tasks, making it suitable for scenarios requiring multi-step reasoning and tool invocation.
• Text generationIt can generate high-quality text content and is suitable for creative writing, content generation, and other scenarios.
• Knowledge Q&AIt performs well in knowledge understanding and question answering tasks and can be used in fields such as intelligent customer service and knowledge graphs.