Editorial illustration for NVIDIA launches Nemotron‑Labs‑TwoTower diffusion model with 128‑expert MoE
NVIDIA Launches Nemotron 128-Expert MoE Diffusion Model
NVIDIA just dropped a diffusion language model that thinks in stereo. Nemotron‑Labs‑TwoTower marries a frozen autoregressive backbone with a second tower that refines noisy blocks, and it does so through a 128‑expert mixture of experts, with six experts active per token plus two shared ones. Both towers begin as carbon copies of the same checkpoint.
The denoiser, however, was trained on a mere 2.1 trillion tokens, a sliver of the backbone’s 25‑trillion‑token pretraining. The result? A model that preserves causal generation while injecting bidirectional refinement, block by block.
NVIDIA has released Nemotron-Labs-TwoTower , a diffusion language model built on a pretrained autoregressive backbone.
The real innovation here isn’t just the scale, 128 experts, 6 active, two shared, it’s the elegant division of labor. One tower keeps its autoregressive crown, preserving the backbone’s causal muscle. The other learns to diffuse, refining noise with bidirectional precision.
And it does all that on a fraction of the training data, just 2.1 trillion tokens against the backbone’s 25 trillion. That’s not a shortcut. That’s a signal.
NVIDIA has shown you can bolt a new capability onto a frozen foundation without rebuilding the engine. The result is a model that generates text not by predicting the next word, but by sculpting the whole block. It’s a different kind of fluency, one that could redraw the line between generation and revision, between prompt and polish.
The TwoTower architecture isn’t just another release. It’s a blueprint for how to make massive pretrained models learn new tricks without forgetting who they are.
Related Reading
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
Open Source OpenAI's GPT Transcribe Cuts Error Rate to 3.31%, Improving on GPT-4o
Open Source Study: 6.7% of Deepfake Requests on Hugging Face Targeted Children
Related Topic NVIDIA and Google Cloud let developers scale AI from prototype to production
Market Trends
Related Topic NVIDIA NeMo powers telco reasoning model for autonomous network workflows
Industry Applications
From Archives Yan calls context engineering the #1 job for AI agent builders, per Martin
From Archives Qwen3.6 Trained on MCP Benchmarks to Prevent Context Loss and Hallucinations
From Archives NVIDIA BioNeMo Agent Toolkit speeds AI for life‑science researchers
LLMs & Generative AI
From Archives NVIDIA Nsight tools boost neural reconstruction efficiency, cutting GPU time
Industry Applications
Common Questions Answered
What is the name of the new NVIDIA model mentioned in the article?
The article states that NVIDIA launched the Nemotron-Labs-TwoTower diffusion model. This model utilizes a Mixture-of-Experts architecture with 128 experts. The 'TwoTower' likely refers to a dual-encoder design.
What is the key architectural feature of the Nemotron-Labs-TwoTower model?
The model employs a Mixture-of-Experts (MoE) architecture with 128 experts. This allows the model to activate only a subset of parameters for each input. The article highlights the 128-expert MoE as a defining characteristic.
How many experts does the MoE in the new NVIDIA model have?
The new NVIDIA Nemotron-Labs-TwoTower diffusion model features a Mixture-of-Experts with 128 experts. This large number of experts enables efficient scaling of model capacity. The article specifically mentions the '128-expert MoE' in the headline.
What type of generative model did NVIDIA launch?
NVIDIA launched a diffusion model, as indicated in the article headline. Diffusion models are used for generating high-quality data such as images or video. The model is named Nemotron-Labs-TwoTower and includes a 128-expert MoE.
Further Reading
- NVIDIA AI Releases Nemotron-Labs-Diffusion: A Tri-Mode Language Model with 6x Tokens Per Forward Over Qwen3-8B — MarkTechPost
- NVIDIA's Nemotron Diffusion: One Model, Three Generation Modes, 6x Faster — Dev.to
- The Full Story of the 6x Faster Text Generation Achieved by NVIDIA Nemotron-Labs Diffusion — note.com
- Nemotron-Labs-Diffusion from NVIDIA — Reddit (LocalLLaMA)
- NVIDIA has released Nemotron-TwoTower-30B-A3B-Base-BF16, an unusual diffusion-based language model — Reddit (LocalLLaMA)
