Nemotron-Labs-Diffusion - NVIDIA's open-source three-mode language model

What is Nemotron-Labs-Diffusion?

Nemotron-Labs-Diffusion is a three-mode language model from NVIDIA that unifies autoregressive, diffusion, and self-inferential decoding within a single architecture. Through joint AR-diffusion target training, the model can switch modes to maintain high throughput under different concurrent scenarios. This series includes 3B, 8B, and 14B parameter base, instruction, and visual language models, surpassing existing open-source AR and diffusion language models in both accuracy and speed.

Main functions of Nemotron-Labs-Diffusion

• Three-mode decoding switchingIt supports three decoding modes: AR, diffusion, and self-inference. It can be seamlessly switched within a single model by changing the attention mode.
• Block-level diffusion parallel generationThe block-level diffusion formula is used to divide the sequence into blocks and then decode them bidirectionally in parallel within each block, generating multiple tokens in a single forward pass.
• Self-deduction draft verificationThe algorithm generates multiple token drafts in parallel, and AR is validated within the same model, sharing a key-value cache and requiring no auxiliary prediction head.
• Visual Language UnderstandingIt provides visual language model variants to support image understanding and text-image reasoning, expanding the boundaries of multimodal applications.
• Efficient inference deploymentSupports FP8 low-precision format and enables high-throughput services on GB200 GPUs by combining with the SGLang framework.

The technical principles of Nemotron-Labs-Diffusion

• Joint AR-Diffusion Target TrainingThe model simultaneously optimizes the autoregressive next-token loss and the block-level diffusion denoising loss, balancing the two with a weighted combination (α=0.3). The AR objective provides left-to-right linguistic priors for diffusion, preventing diffusion training from wasting capacity on arbitrary token permutations; the diffusion objective enhances the model's look-ahead planning ability.
• Two-stage training strategyThe first stage uses only AR target pre-training to establish a strong left-to-right linguistic inductive bias; the second stage enables joint training to supplement the diffusion gradient and achieve a harmonious fusion of the two targets.
• Structured attention patternDuring training, a dual-stream input is used. In the noisy stream, tokens undergo bidirectional attention within blocks and causal attention across blocks, while attention is paid to the generated prefixes in the clean stream. The clean stream maintains a strict causal mask, allowing AR targets and diffusion targets to be jointly computed in the same forward-backward propagation.
• Global loss averageTo address the gradient variance issue caused by variations in the number of noise tokens in different samples within the diffusion target, a global token-level loss averaging method is adopted, which equally weights all tokens in the batch to avoid disproportionately affecting the batch gradient by a few high-weight noise samples.
• Optimal Sampling and LoRA EnhancementThe diffusion mode, combined with a sampler optimized based on sampling trajectory, improves parallelism; the self-predictive mode can be equipped with a LoRA draft adapter to enhance draft quality, improve acceptance rate, and increase actual device efficiency.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use Nemotron-Labs-Diffusion

• Install dependencies:implement pip install "transformers>=5.0" torch peft Commands such as `--` prepare the Python and GPU environments.
• Get the modelPull from HuggingFace nvidia/Nemotron-Labs-Diffusion-8B Equal model weights and corresponding word segmenters.
• Select modeChoose AR mode (high concurrency), diffusion mode (maximum parallelism), or self-predictive mode (low latency) based on the concurrency level.
• Run script:use chat_ar.py,chat_dlm.py or chat_linear_spec.py Wait for the official script to perform single-round or multi-round dialogue reasoning.
• Service Deployment: Start the inference service based on the SGLang framework, configure Linear Self-Speculation and FP8 accuracy, and connect to the production environment.

The core advantages of Nemotron-Labs-Diffusion

• Three-mode unified architectureA single model can simultaneously master AR language priors and diffusion parallel planning capabilities, eliminating the need to maintain multiple models and pipelines.
• Significantly improved throughputThe 8B model decodes 6 times more tokens per forward pass than Qwen3-8B, and the throughput of SPEED-Bench on GB200 is increased by 4 times.
• Self-prediction is better than MTPThe self-predictive model outperforms the multi-token prediction (MTP) method in both acceptance rate and actual device efficiency.
• Dual-objective harmony trainingThe global loss averaging and two-stage strategy effectively balance the gradients of the two objectives and avoid capacity competition between modes.
• Scene adaptive switchingBy switching attention modes, it can adapt to high-concurrency cloud-side and low-concurrency edge-side scenarios without requiring architectural modifications.

Nemotron-Labs-Diffusion project address

• HuggingFace model libraryhttps://huggingface.co/collections/nvidia/nemotron-labs-diffusion
• arXiv technical paper: https://arxiv.org/pdf/2607.05722

Comparison of Nemotron-Labs-Diffusion with similar products

• Real-time conversation assistantIn low-concurrency scenarios, a self-predictive mode is used to achieve low-latency interactive AI dialogue and improve user experience.
• Cloud inference serviceAR mode adapts to high-concurrency batch requests, making full use of GPU computing density and reducing cloud deployment costs.
• Code and Mathematical ReasoningThe diffusion model enhances forward planning and is suitable for programming-assisted and mathematical problem-solving tasks that require multi-step logical deduction.
• End-side local inferenceThe 8B model can run on personal devices, with three modes to adapt to different workloads and ensure data privacy.
• Visual language applicationsThe visual language model variant supports image understanding and image-text question answering, and is suitable for intelligent document analysis and multimodal interaction.

Application scenarios of Nemotron-Labs-Diffusion

• Real-time conversation assistantIn low-concurrency scenarios, the self-predictive mode is used to achieve low-latency interactive AI dialogue and improve user experience.
• Cloud inference serviceAR mode adapts to high-concurrency batch requests, making full use of GPU computing density and reducing cloud deployment costs.
• Code and Mathematical ReasoningThe diffusion model enhances forward planning and is suitable for programming-assisted and mathematical problem-solving tasks that require multi-step logical deduction.
• End-side local inferenceThe 8B model can run on personal devices, with three modes to adapt to different workloads and ensure data privacy.
• Visual language applicationsThe visual language model variant supports image understanding and image-text question answering, and is suitable for intelligent document analysis and multimodal interaction.