NVIDIA's open-source Nemotron-Labs-TwoTower diffusion language model

NVIDIA has open-sourced Nemotron-Labs-TwoTower, a discrete diffusion language model based on a pre-trained autoregressive backbone, which addresses the bottleneck of token generation speed in large models. The model has a total of 60 bytes of parameters and employs a dual-tower architecture—a 30-byte autoregressive context tower and a 30-byte diffusion denoising tower, with each tower activating 3 bytes of parameters. The context tower maintains the autoregressive context frozen, while the denoising tower handles noise removal. The two collaborate through layer-by-layer cross-attention.