LLaDA - A diffusion-based large language model jointly launched by Renmin University Hillhouse AI and Ant Financial.
LLaDA (Large Language Diffusion with mAsking) is a novel large-scale language model jointly developed by Professors Li Chongxuan and Wen Jirong's team at Renmin University of China's Gaoling School of AI and Ant Group. It is based on a diffusion model framework...
What is LLaDA?
LLaDA (Large Language Diffusion with mAsking) is a novel large-scale language model jointly developed by Professors Chongxuan Li and Jirong Wen's team at the Gaoling School of AI, Renmin University of China, and Ant Group. It is based on a diffusion model framework rather than the traditional autoregressive model (ARM). LLaDA models text distribution based on forward masking and backward inference processes, using a Transformer as the masking predictor and optimizing the likelihood lower bound to achieve generation tasks. LLaDA uses 2.3 trillion labeled data points during pre-training and improves instruction compliance through supervised fine-tuning (SFT). LLaDA excels in scalability, context learning, and instruction compliance, solving the "curse of inversion" problem of traditional ARM in inverse inference tasks. Its 8-parameter version is comparable to strong models like LLaMA3 in multiple benchmark tests, demonstrating the significant potential of diffusion models as an alternative to autoregressive models.
Main functions of LLaDA
- Efficient text generationIt can generate high-quality, coherent text content, suitable for scenarios such as writing, dialogue, and content creation.
- Powerful context learning capabilities: Quickly adapt to new tasks based on context.
- Instruction compliance capabilityIt can better understand and execute human instructions, and is suitable for scenarios such as multi-turn dialogue, question and answer, and task execution.
- Two-way reasoning abilityIt solves the "curse of reversal" of traditional autoregressive models and performs well in both forward and backward reasoning tasks, such as poetry completion tasks.
- Multi-domain adaptabilityIt performs exceptionally well in multiple fields, including language comprehension, mathematics, programming, and Chinese language comprehension, and has broad applicability.
LLaDA's technical principles
- Diffusion Model FrameworkThis model models text distribution based on a forward masking process (gradually masking markers in the text) and a backward recovery process (gradually restoring masked markers). It supports the model in generating text in a non-autoregressive manner, avoiding the sequential generation limitations of traditional autoregressive models.
- Masking predictorUsing a standard Transformer architecture as the mask predictor, the model takes a partially masked text sequence as input and predicts all masking markers. This allows the model to capture bidirectional dependencies, rather than just unidirectional left-to-right generation.
- Optimize the likelihood lower boundTraining the model based on the optimized likelihood lower bound is a fundamental principle in generative modeling, ensuring the scalability and generative capability of the model under large-scale data and model parameters.
- Pre-training and supervised fine-tuningLLaDA is based on a combination of pre-training and supervised fine-tuning (SFT). The pre-training stage uses large-scale text data for unsupervised learning, while the SFT stage uses labeled data to improve the model's ability to follow instructions.
- Flexible sampling strategyDuring the generation process, LLaDA supports a variety of sampling strategies (such as random masking, low-confidence masking, semi-autoregressive masking, etc.) to balance generation quality and efficiency.
LLaDA's project address
- Project official website:https://ml-gsai.github.io/LLaDA
- GitHub repository:https://github.com/ML-GSAI/LLaDA
- arXiv technical paper:https://arxiv.org/pdf/2502.09992
Application scenarios of LLaDA
- Multiple rounds of dialogueUsed for intelligent customer service, chatbots, etc., supporting smooth multi-turn communication.
- Text generationIt is suitable for writing assistance, creative copywriting, etc., and can generate high-quality text.
- Code generationIt helps developers generate code snippets or fix errors, improving programming efficiency.
- Mathematical reasoningIt solves mathematical problems, provides solution steps, and can be used in the field of education.
- Language translationTo achieve multilingual translation and promote cross-cultural communication.