Editorial illustration for BitsMoE uses SVD to keep basis unquantized, allocating bits to expert spectral factors
BitsMoE uses SVD to keep basis unquantized, allocating...
The arithmetic of mixture-of-experts models promises efficiency, but quantization, the brutal necessity of shrinking them for deployment, has always demanded a cruel trade-off: compress the shared structure or starve the expert nuances. BitsMoE refuses that choice. By decomposing each MoE layer via singular value decomposition into a shared basis and expert-specific spectral factors, it preserves the common cross-expert architecture intact, unquantized, while allocating every precious bit to the factors that define each expert’s identity.
The result is a precision budget that doesn’t guess. BitsMoE formulates spectrum-wise mixed-precision quantization as an activation-aware reconstruction surrogate, then solves an integer linear program that minimizes estimated loss under a fixed bit budget. No hand-waving, no heuristic rounding, just optimal allocation.
On Qwen3-30B-A3B-Base pushed to 2 bits, the method accelerates quantization by 12.3× over GPTQ, lifts average accuracy by nearly 28 percentage points, and boosts decoding speed by 1.76×. The shared basis stays pristine; the experts pay only for what they need.
BitsMoE decomposes each MoE layer by SVD into a shared basis and expert-specific spectral factors, retaining the shared basis without quantization to preserve common cross-expert structure and using the expert-specific factors as fine-grained quantization units.
BitsMoE does not merely compress; it rethinks the architecture of information. By isolating the shared basis from the expert-specific spectral factors, it protects the common structure that makes mixture-of-experts work, while ruthlessly optimizing the bits that matter. The result is a quantization strategy that is both principled and brutal: an integer linear program that treats each spectral factor as a discrete resource allocation problem, solved under a hard budget.
The numbers speak plainly. On Qwen3-30B-A3B-Base, at 2 bits, BitsMoE is 12 times faster to quantize, nearly 28 percentage points more accurate, and almost twice as fast at decoding compared to GPTQ. This is not incremental improvement.
It is a structural shift in how we think about precision in MoE models. The shared basis stays untouched, a reservoir of common knowledge. The bits go where they are needed, guided by the spectrum’s energy.
BitsMoE proves that the most efficient quantization is not the one that treats every parameter equally, but the one that knows which parts of the model deserve to be preserved and which can be sacrificed. The future of ultra-low-bit MoE inference is not about brute force compression. It is about spectral intelligence.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Claude Code Leads Feature Set as Codex Adopts Similar Tools for Coding
From Archives MiniMax-M3 launches, beats GPT-5.5 and Gemini 3.1 Pro on benchmarks, costs 5‑10%
Common Questions Answered
How does BitsMoE use SVD to preserve the shared basis in mixture-of-experts models?
BitsMoE decomposes each MoE layer using singular value decomposition into a shared basis and expert-specific spectral factors. This approach keeps the shared basis unquantized while allocating quantization bits exclusively to the expert-specific spectral factors, protecting the common cross-expert architecture that makes mixture-of-experts models effective.
What is the key trade-off that BitsMoE resolves in MoE quantization?
Traditional MoE quantization forces a choice between compressing the shared structure or losing expert nuances, creating a cruel trade-off for deployment. BitsMoE refuses this choice by isolating shared and expert-specific components, allowing it to preserve the common architecture intact while optimizing quantization bits for expert spectral factors.
How does BitsMoE allocate bits to expert spectral factors?
BitsMoE uses an integer linear program that treats each spectral factor as a discrete resource allocation problem solved under a hard budget constraint. This principled approach ensures that every precious bit is allocated to where it matters most in the expert-specific components.
Why is keeping the basis unquantized important in BitsMoE's architecture?
The shared basis represents the common cross-expert structure that is fundamental to how mixture-of-experts models function. By keeping it unquantized while only quantizing expert-specific spectral factors, BitsMoE maintains the integrity of the core MoE mechanism while still achieving significant compression through targeted bit allocation.
Further Reading
- MoE-SVD: Structured Mixture-of-Experts LLMs Compression via SVD Decomposition — OpenReview
- MoE-SVD: Structured Mixture-of-Experts LLMs Compression via SVD Decomposition — PMLR
- MoE-SVD: Structured Mixture-of-Experts LLMs Compression via SVD Decomposition — ICML 2025
- NEW Mixture of Expert MoE: Spectral Decomposition in Orthogonal ... — YouTube
- Spectral Entropy-Based Bit Allocation — ViVoNets / UCSB
