Editorial illustration for Multiverse reduces inference cost by favoring low‑cost prefill over decoding
Multiverse reduces inference cost by favoring low‑cost...
Most AI cost analysis misses the point. The expensive part isn't starting a conversation with the model. It's letting it finish.
Inference cost splits in two. Prefill is cheap and fast: you throw your prompt in, the model builds its initial state. Decoding is the money pit.
It's the slow, memory-bound grind of generating each token one by one. For years, everyone focused on making decoding faster. A team called Multiverse asks a different question: what if you just didn't do as much of it?
Their method is deliberately wasteful. Instead of letting a single, expensive decoding chain run on forever, they stop it. They restart the whole process with a fresh, cheap prefill.
They run multiple cheap jobs instead of one long, costly one. It feels backwards. But prefill is so much cheaper than decoding that the math works out.
And because this second prefill uses causal attention, it's a drop-in replacement for existing models. No special hardware required.
While this introduces computational redundancy that Multiverse tries to avoid, the cost of prefill is significantly lower than decoding. In addition, this does not require special attention handling during inference, as the second prefill uses causal attention (threads see each other), making it easier to adapt sequential autoregressive models for this task. Figure 9: ThreadWeaver's Prefill and Decode Strategy How should we train a model to learn this behavior?
Naively, for each parallel trajectory, we can break it down into multiple sequential pieces following our inference pattern. For instance, we would train the model to output the subtasks given prompt, individual threads given prompt+subtask assignment, and conclusion given prompt+subtasks+corresponding threads.
The trick is exploiting a price asymmetry everyone else treats as a fact of life. You don't need a new model. You just need to change how you use the old one.
Train it to break a problem into cheaper chunks, solve those, then synthesize an answer. You think in parallel. You pay in cheap, serial prefills.
This is more than an optimization. It's a different philosophy. Stop trying to win the decoding race.
Reroute the entire problem onto cheaper ground. Let the expensive part of the process become a footnote.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives AI agents solve neuroscience pipeline tasks on datasets larger than benchmarks
From Archives ML models predict World Cup outcomes, but miss draws, capture team strength
Common Questions Answered
What is the key difference between prefill and decoding costs in AI inference?
Prefill is the cheap and fast initial phase where the model processes your prompt and builds its initial state, while decoding is the expensive, memory-bound process of generating each token one by one. According to Multiverse, decoding represents the true money pit of inference costs, not the initial conversation start as most analyses suggest.
How does Multiverse reduce inference costs by exploiting price asymmetry?
Multiverse trains models to break problems into cheaper chunks that can be solved in parallel, then synthesize an answer by routing the work onto low-cost prefill operations instead of expensive decoding. This approach doesn't require a new model, just a different philosophy of how to use existing ones by favoring serial prefills over the decoding race.
Why has the AI industry traditionally focused on making decoding faster rather than reducing its usage?
The industry has treated the price asymmetry between prefill and decoding as a fact of life, leading most cost optimization efforts to target decoding speed improvements. Multiverse's insight is that instead of trying to win the decoding race, the real opportunity lies in rerouting problems onto cheaper prefill ground entirely.
What is the philosophical shift that Multiverse proposes beyond just optimization?
Multiverse proposes a fundamental change in how AI inference problems are approached, moving away from trying to optimize the expensive decoding phase toward restructuring problems to use cheaper prefill operations. This represents a different philosophy where you think in parallel but pay only for cheap, serial prefills, rather than accepting decoding as an unavoidable bottleneck.
Further Reading
- How To Reduce Inference Costs While Running LLMs — To Data Beyond
- Prefill vs Decode: LLM Inference Phases Explained — Redis
- LLM Inference Optimization: Cut Cost & Latency at Every Layer (2026) — Morph LLM
- Decoding the Energy of LLM Inference in Software Development — arXiv
- Mastering LLM Techniques: Inference Optimization — NVIDIA Developer Blog
