ELF - The first diffusion language model launched by He Kaiming's team

What is ELF?

ELF (Embedded Language Flows) is the first language flow library launched by Kaiming He's team.diffusionThis language model employs a continuous diffusion paradigm instead of the traditional autoregressive approach. The model generates text by denoising within a continuous embedding space throughout the process, discretizing it into tokens in the final step. Using 10⁵M parameters, 45B training tokens, and 32 sampling steps, the model achieves a generation perplexity of 24 on OpenWebText, outperforming mainstream discrete diffusion models with nearly 10 times less training data. It demonstrates excellent performance on tasks such as unconditional generation, translation, and summarization.

Main functions of ELF

• Continuous spatial text generationThe entire process is performed in a continuous embedding space for denoising. At the final time step (t=1), the continuous representation is projected into a discrete token through an unembedding layer.
• Unconditional Language GenerationStarting from pure Gaussian noise, natural, low-perplexity human-style text is generated through 32 iterations.
• Conditional text taskIt supports conditional generation tasks such as machine translation (WMT14) and text summarization (XSum), and its performance surpasses existing diffusion language models and some autoregressive baselines.
• Training-Inference Unified ArchitectureThe denoising network and the final decoding network share the same set of parameters and switch between denoising/decoding modes using a binary mode token, eliminating the need to train an independent decoder.

ELF Technical Principles

• Continuous Embedding encodingDiscrete tokens are first mapped to bidirectional contextual embeddings by a T5 pre-trained encoder. These embeddings are used only during the training phase and do not add any additional modules during inference.
• Flow Matching + x-predictionDefines a rectified flow trajectory from noise to clean data in a continuous space; the network directly predicts clean embeddings (x-prediction), which are not velocity fields, and the training objective is MSE loss, which is more stable in high-dimensional representations.
• Final-step DiscretizationThe final step projects the continuous embeddings into token logits through a learnable unembedding matrix. Token-level corruption is added during training to prevent the task from being too simplistic, and the loss is cross-entropy.
• Self-conditioning CFGIt introduces the classifier-free guidance concept from image generation, and uses training-time CFG as the conditional signal, eliminating inference overhead.

How to use ELF

• Access code repositoryAccess the cloned project code on GitHub.
• Prepare the environmentInstall dependencies (PyTorch, etc.) and configure the GPU environment.
• Data preprocessingThe text data is converted into a continuous embedding format using a T5 encoder.
• Model training: Train the denoising network using the Flow Matching + x-prediction objective function, supporting MSE or CE loss.
• Text generationStarting with Gaussian noise, the ELF denoising network is called for 32 iterations, and the final step switches to decoding mode to output the token.
• Downstream task fine-tuningFine-tuning of conditional generation on datasets such as WMT14 and XSum.

ELF's core advantages

• Extremely high data efficiencyIt is trained with 45B tokens, which is an order of magnitude less than the 500B+ tokens of its competitors such as MDLM, Duo, and FLM.
• Very few sampling stepsIt can achieve or even surpass the generation quality of its competitors' 1024 steps in 32 steps.
• Higher qualityOpenWebText generates text with a perplexity as low as 24, resulting in more natural text and less AI artifacts.
• Simple and unified architectureThe denoising and decoding share a single network, eliminating the need for an additional decoder module and avoiding the overhead and error accumulation associated with decoder training in latent diffusion.

ELF's project address

• GitHub repositoryhttps://github.com/lillian039/ELF
• arXiv technical paper: https://arxiv.org/pdf/2605.10938

Comparison of ELF's similar competing products

Application scenarios of ELF

• Low-resource, high-efficiency text generationHigh-quality models can be trained using 45B tokens, making them suitable for businesses and researchers with limited data budgets to quickly deploy natural language generation capabilities.
• Machine translationIt outperforms existing diffusion models and some autoregressive baselines on translation tasks such as WMT14, and can serve as the core engine for non-autoregressive translation systems.
• Text summarization and content rewritingIt performs stably in summarizing tasks such as XSum and is suitable for scenarios that require retaining key information, such as news summarization and document extraction.
• Creative Writing and Dialogue GenerationIt generates perplexity as low as 24, with high text naturalness and weak AI traces, making it suitable for creating human-style content such as long novels and marketing copy.
• Baseline of diffusion language model researchThis study is the first to verify the feasibility of the "continuous to the end" route in text generation, providing an important reference and basic framework for subsequent exploration of large model architecture.