Editorial illustration for AMD's Instella-MoE-16B Hits 12.7% Speedup With New FarSkip Training Technique
AMD's Instella-MoE-16B Achieves 12.7% Speedup
AMD put out Instella-MoE-16B-A3B this week, a Mixture-of-Experts language model built from the ground up on its own Instinct MI300X and MI325X GPUs. The headline number is 16 billion total parameters, but only 2.8 billion of those fire on any given token, which is the whole point of routing work through a small subset of experts instead of the full network. AMD isn't just handing over a checkpoint either. The company published weights from every stage of training, the data mixtures that fed them, the training configs, and the inference code needed to actually run the thing.
That openness comes with a catch on the licensing side, which matters more for some teams than others. But before getting into who can actually deploy this and how, there are two architectural decisions baked into Instella-MoE that explain why AMD is talking about it as more than another open-weights release: Gated Multi-head Latent Attention and something called FarSkip-Collective connectivity. Both are aimed at squeezing more out of expert-parallel training on Instinct hardware, and they're worth understanding before looking at what the model can and can't do in practice.
AMD released Instella-MoE-16B-A3B, a fully open Mixture-of-Experts language model trained from scratch on Instinct MI300X and MI325X GPUs. The model holds 16B total parameters but activates only 2.8B per token.
Why this matters
AMD is betting that openness, not just raw benchmark scores, is what pulls developers off Nvidia's CUDA stack. Publishing every checkpoint, the data mixture, and inference code for Instella-MoE-16B-A3B gives researchers something Meta and Mistral rarely offer: a full paper trail on how a mixture-of-experts model actually gets built on MI300X and MI325X hardware. That matters for anyone trying to reproduce results or debug training instability, which is usually the hardest part of MoE work.
The FarSkip-Collective numbers deserve scrutiny rather than applause. A 12.7% pre-training speedup and a 39.2% cut in time to first token are real engineering wins if they hold up outside AMD's own benchmarks, but "up to" language always invites questions about typical versus best-case results. The Research-only license is the bigger catch.
Founders eyeing this for production should treat it as a proof of concept for AMD's software maturity, not a deployable asset. The real test is whether independent teams can replicate the speedups on their own MI300X clusters without AMD's tuning.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Gemini Robotics ER 2 Improves Robot Tool Workflow
From Archives Deepseek's New AI Model Matches GPT-5.6 at 60% Lower Cost
Common Questions Answered
What is the key difference between Instella-MoE-16B's total parameters and active parameters?
Instella-MoE-16B-A3B has 16 billion total parameters, but only 2.8 billion of those parameters activate on any given token. This Mixture-of-Experts architecture routes work through a small subset of experts rather than using the full network, which enables the reported 12.7% speedup in performance.
What hardware was used to train AMD's Instella-MoE-16B model?
AMD's Instella-MoE-16B-A3B was trained from scratch on AMD's own Instinct MI300X and MI325X GPUs. This represents AMD's effort to demonstrate that developers can build and train advanced language models on their own hardware stack rather than relying on Nvidia's CUDA platform.
How does AMD's approach to releasing Instella-MoE-16B differ from competitors like Meta and Mistral?
AMD published not only the model weights but also weights from every stage of training, the data mixtures used, and inference code, providing a complete paper trail of how the mixture-of-experts model was built. This level of transparency is rarely offered by competitors and helps researchers reproduce results and debug training instability.
Why does AMD emphasize openness as a strategy for Instella-MoE-16B?
AMD believes that openness and transparency, not just raw benchmark scores, will convince developers to move away from Nvidia's CUDA stack. By providing full training checkpoints and documentation, AMD enables researchers to understand the complete development process and makes it easier to reproduce and debug the model on MI300X and MI325X hardware.
Further Reading
- Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Model - AMD ROCm Blog
- Accelerating Mixture-of-Experts Execution with FarSkip-Collective - AMD ROCm Blog
- FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models - arXiv
- AMD Instella-MoE: Fully Open 16B MoE Model on AMD GPUs - InformerTech
- AMD Instella-MoE, a fully open MoE model with 16 billion parameters trained on Instinct GPUs - 24 AI
