Editorial illustration for Zhipu AI employs Muon Optimizer and Muon Split in GLM-4.5 and GLM-5 pretraining
Zhipu AI employs Muon Optimizer and Muon Split in...
The wall isn't made of silicon. At Zhipu AI, engineers hit it while training GLM-4.5. The problem was the optimizer—the software that tweaks a model's billions of internal knobs.
Adam, the industry standard, had stalled. So they built an exit called Muon and baked it into their next model, the colossal 744-billion-parameter GLM-5. They paired it with a surgical fix dubbed Muon Split.
That technique isolates a specific performance lag when mixing certain attention mechanisms. It doesn't broadly boost efficiency. It orthogonally repairs the up-projection matrix for each attention head, one by one.
Muon Optimizer has gained great momentum with significant adoption from frontier AI Labs.
DeepSeek-V4, a 1.6-trillion-parameter beast, runs on Muon too. The goal there is faster convergence and steadier training. A trend is solidifying.
But integration is gritty work. Plugging Muon into standard frameworks like DeepSpeed reveals a clash of philosophies. Legacy optimizers like SGD and Adam see gradients as simple, flattened buffers—just lists of numbers.
Muon demands a more structured view.
Progress now hinges on this re-navigation of fundamentals. The labs defining the frontier have flagged the optimizer itself as the critical bottleneck. Their next breakthroughs won't be measured in raw parameter counts alone. They'll be tallied in weeks of saved training time and in costly failed runs that never happen.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Anthropic says Claude writes >90% of its code; AI pause button urged
From Archives Choosing AI Models: Prioritize Real‑World Needs Over Benchmark Rankings
Common Questions Answered
What is the Muon Optimizer and why did Zhipu AI develop it for GLM-4.5?
The Muon Optimizer is a custom optimizer that Zhipu AI developed to overcome limitations with Adam, the industry-standard optimizer that had stalled during GLM-4.5 training. Rather than accept the performance plateau, Zhipu AI built Muon as an alternative approach to tweaking a model's billions of internal parameters more effectively during pretraining.
How does Muon Split address performance issues in GLM-5 pretraining?
Muon Split is a surgical technique that isolates and fixes a specific performance lag that occurs when mixing certain attention mechanisms during model training. By targeting this particular bottleneck, Muon Split helps optimize the training process for GLM-5, Zhipu AI's massive 744-billion-parameter model.
What fundamental difference exists between legacy optimizers like Adam and the Muon Optimizer?
Legacy optimizers like SGD and Adam treat gradients as simple, flattened buffers—just lists of numbers—while Muon demands a more structured view of gradient data. This philosophical difference creates integration challenges when plugging Muon into standard frameworks like DeepSpeed, requiring a re-navigation of fundamental optimization principles.
Which other large language models are using the Muon Optimizer besides GLM-5?
DeepSeek-V4, a 1.6-trillion-parameter model, also runs on the Muon Optimizer to achieve faster convergence and steadier training. This trend of adoption across different labs indicates that Muon is becoming an increasingly viable alternative to traditional optimizers in large-scale model pretraining.
Further Reading
- Using Muon Optimizer with DeepSpeed — PyTorch
- GLM-5: from Vibe Coding to Agentic Engineering — arXiv
- GLM-5: Top-1 Open-Weight Model for Code and Text Generation Competing with Claude and GPT on Agentic Tasks — NeuroHive
- A Review of GLM-5: 744B-Parameter LLM Built for 200K Context — The Salt
- GLM-4.5: The Open-Source Model That Challenges Proprietary AI Dominance — United Nations University
