Nemotron-Labs-TwoTower - NVIDIA's open-source dual-tower architecture diffusion language model - AiBoss

What is Nemotron-Labs-TwoTower?

NVIDIA Nemotron-Labs-TwoTower is an open-source dual-tower diffusion language model from NVIDIA, with approximately 60 bytes of total parameters and 3 bytes of active parameters. The model decouples context understanding and denoising generation into two independent towers: a frozen AR context tower cleans the tokens, while a trainable diffusion denoising tower refines noise blocks through cross-attention. Built on Nemotron-3-Nano-30B-A3B, the model trains on approximately 2.1TB of tokens, retains 98.7% of the quality of the autoregressive baseline, and achieves a 2.42x improvement in generation throughput. It supports Mask Diffusion, Mock-AR, and AR-only inference modes.

Main functions of Nemotron-Labs-TwoTower

Twin Towers Separate Reasoning: Decompose context encoding and denoising generation into two independent modules to avoid the performance bottleneck of a single network "doing both jobs".
Switching between three reasoning modesA single checkpoint supports Mask Diffusion, Mock-AR, and AR-only, flexibly adapting to different scenario requirements.
High-quality text generationIt achieves parallel iterative generation while retaining 98.7% of the quality of the aggregated baseline of the autoregressive baseline.
High throughput generation accelerationIn a 2×H100 environment, the throughput is increased by 2.42 times, significantly reducing inference latency and computational cost.
Commercial-grade open-source deploymentIt uses the NVIDIA Nemotron Open Model License for open source, supporting enterprise commercial use and secondary development.

The technical principles of Nemotron-Labs-TwoTower

Core ideaTraditional diffusion language models use a single network to handle both "contextual understanding" and "iterative denoising," causing these two functions to hinder each other. TwoTower decouples these two responsibilities into a dual-tower architecture:
- Context TowerThe frozen Nemotron-3-Nano-30B-A3B uses causal attention to process clean tokens and is responsible for high-quality context representation encoding.
- Denoiser Tower: A trainable module that uses bidirectional block attention to process noisy token blocks and obtains semantic guidance from the context tower through cross attention to gradually refine and denoise the tokens.
Training methodsBased on a 30B hybrid Mamba-Transformer MoE backbone, it is trained on approximately 2.1T tokens and uses MoE sparse activation to achieve efficient computation.
Reasoning advantageThe parallel iterative nature of the diffusion model naturally supports accelerated decoding. After the dual towers are separated, the denoising tower does not need to repeatedly encode the context, which greatly improves the wall-clock throughput while ensuring quality.

How to use Nemotron-Labs-TwoTower

Visit the HuggingFace model pageOpen the official Nemotron-Labs-TwoTower-30B-A3B-Base-BF16 repository page.
Reading Model Cards and LicensesConfirm the commercial terms of the NVIDIA Nemotron Open Model License and understand the model architecture and hardware requirements.
Cloning/Download Weight:use git lfs Or HuggingFace transformers Download the model weights and configuration files from the library.
Configure the runtime environmentPrepare an environment with at least 2 x H100 GPUs, and install PyTorch and its dependencies (refer to the repository). requirements.txt).
Loading Model and Tokenizer:pass AutoModelForCausalLM and AutoTokenizer Load the model and select Mask Diffusion / Mock-AR / AR-only inference mode.
Execution reasoning generationInput prompt to call the model generation interface and use a dual-tower architecture to complete efficient parallel denoising generation.
Fine-tuning adaptationThe trainable denoising tower is further fine-tuned based on our own data, while the frozen context tower remains unchanged.

Nemotron-Labs-TwoTower's core advantages

The two towers are decoupled, each performing its own function.: Separate context encoding and denoising generation into two independent towers to avoid performance bottlenecks caused by conflicts in a single network role.
The quality is almost undamaged.The aggregation baseline quality is retained at 98.7% of the autoregressive baseline, and diffusion generation no longer comes at the expense of output quality.
Reasoning speed doubledAchieve 2.42 times the wall-clock generation throughput in a 2×H100 environment, significantly reducing latency and computing power costs.
One model, three modesA single checkpoint supports three inference methods: Mask Diffusion, Mock-AR, and AR-only, flexibly adapting to different latency and quality requirements.
Open source and commercially usable: Publish weights using the NVIDIA Nemotron Open Model License, supporting free deployment and commercial secondary development by enterprises.

Nemotron-Labs-TwoTower project address

HuggingFace model libraryhttps://huggingface.co/collections/nvidia/nemotron-labs-twotower
arXiv technical paper: https://arxiv.org/pdf/2606.26493

Nemotron-Labs-TwoTower Competitive Product Comparison

Comparison Dimensions	Nemotron-Labs-TwoTower	LLaDA
Issuing agency	NVIDIA	MIT
Architecture Design	Twin towers separated: Frozen AR context tower + trainable diffusion denoising tower (cross-attention connection)	Single Tower UnificationA single Transformer performs both context encoding and mask denoising.
Total number of parameters	~60B (Active 3B, MoE sparse)	8B (Dense)
Base model	Nemotron-3-Nano-30B-A3B (Mamba-Transformer MoE)	Self-developed Transformer
Training data	~2.1T tokens	~2T tokens
Baseline quality retention	98.7%(Relative to the autoregressive baseline)	~95% (relative to AR models of the same size)
Throughput Enhancement	2.42×(2×H100, wall-clock)	~1.5× (Standard GPU environment)
Reasoning patterns	Three types：Mask Diffusion/Mock-AR/AR-only	singleMask diffusion (random/semi-autoregressive sampling)
Attention mechanism	Context tower: causal attention; Denoising tower: bidirectional block attention + cross attention	Unified bidirectional attention + positional encoding processing
Core Innovation	Role decouplingAvoid the performance bottleneck of a single network "doing two jobs".	Simple scalable: Prove that the diffusion model can scale to 8B and approximate the mass of GPT-4.

Application scenarios of Nemotron-Labs-TwoTower

High-concurrency online servicesThe 2.42x throughput improvement makes it suitable for real-time text generation scenarios that require low latency and high concurrency, such as search engines and intelligent customer service.
Long document generationThe parallel iterative nature of the diffusion model makes it suitable for tasks that require multiple refinement steps, such as long text continuation, report generation, and code completion.
Flexible deployment in multiple modesThe ability to switch between three inference modes allows businesses to weigh cost/quality trade-offs, using AR-only at the edge and Mask Diffusion in the cloud.
Commercial product developmentThe commercial licensing of the model allows businesses to integrate it into commercial products such as writing assistants, marketing copy generators, and code assistance tools.
Scientific research and secondary innovationThe open-source weights and paper details facilitate researchers' exploration of diffusion language model architecture, training strategies, and cross-modal expansion directions.