Editorial illustration for Researchers train AI model achieving near-full performance using 12.5% of experts
Researchers train AI model achieving near-full...
At Carnegie Mellon and Peking University, a team has solved a stubborn puzzle. Their massive "EMO" model, built with 14 billion parameters, now runs on a fraction of its parts. For any task, it fires up just eight of its 128 internal experts.
Compute demand plummets to 12.5%. Performance barely dips. Lead author Weilin Zhao's group reconciled two opposing training objectives that usually break such sparse systems.
Researchers at the Allen Institute for AI and UC Berkeley have built EMO, a mixture-of-experts model that develops modular structures during pre-training.
Slashing active compute by 87.5% is a direct path to cheaper server bills. The breakthrough hinged on one tactical shift during training on the OLMoE corpus: randomizing the document pool size. That simple variability taught the model crucial flexibility.
Now it can dynamically assemble needed expert subgroups. The consequence? Future AI could scale without gargantuan cost, activating only precise neural pathways instead of lighting up the entire network.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives RecursiveMAS cuts multi-agent inference time 2.4×, slashes token use 75%
From Archives ArXiv to ban authors of papers with unchecked LLM‑generated content
Common Questions Answered
How does the EMO model achieve near-full performance while using only 12.5% of its experts?
The EMO model, built with 14 billion parameters and 128 internal experts, activates only eight experts for any given task, reducing compute demand to 12.5% while maintaining performance. This efficiency was achieved through a tactical training shift that involved randomizing the document pool size, which taught the model crucial flexibility to dynamically assemble the needed expert subgroups for optimal task performance.
What training technique did Carnegie Mellon and Peking University researchers use to enable selective expert activation?
The breakthrough hinged on randomizing the document pool size during training on the OLMoE corpus, which introduced variability that taught the model crucial flexibility. This simple but strategic shift enabled the model to learn how to dynamically assemble the precise neural pathways and expert subgroups needed for different tasks rather than activating the entire network.
What are the practical implications of reducing active compute by 87.5% in the EMO model?
Slashing active compute by 87.5% provides a direct path to significantly cheaper server bills and more cost-effective AI deployment. This breakthrough demonstrates that future AI systems could scale without gargantuan costs by activating only precise neural pathways instead of lighting up the entire network, making advanced AI more economically viable.
How many parameters does the EMO model contain and how many experts does it use at once?
The EMO model is built with 14 billion parameters and contains 128 internal experts in total. For any task, it activates only eight of these 128 experts, demonstrating highly efficient resource utilization while maintaining near-full performance levels.
Further Reading
- EMO: Pretraining Mixture of Experts for Emergent Modularity — Allen Institute for AI
- Emo: Pretraining Mixture of Experts for Emergent Modularity — arXiv
- Allen Institute for AI releases EMO Mixture-of-Experts model — KRO
- OLMoE: An open, small, and state-of-the-art mixture-of-experts model — Allen Institute for AI
- Mixture of Experts 2.0 Eliminates AI Infrastructure Bottlenecks — Galileo AI
