Editorial illustration for M* introduces overlapped scheduling to streamline multimodal model serving
M* introduces overlapped scheduling to streamline...
Multimodal models are slow. The real problem isn't the math. It's the wait.
The GPU sits idle while the CPU schedules the next batch, a tax paid on every loop. M* cuts that tax. Its overlapped scheduling runs the GPU and prepares the next batch at the same time, on different streams.
The check to see if a request is done gets pushed back by one cycle. The loop never really stops.
This works for anything because it's built over a generic primitive called Loop. Text, images, speculative decoding. The goal is simple: keep the GPU from stalling.
The method is direct. And it's only part of the fix.
In M*, a model is declared as a graph of model component nodes connected by tensor edges, plus a set of named Walks. Each walk is a labeled subgraph for one phase of behavior. A request is a series of Walks, chosen by a small state machine the model author writes.
The results are not subtle. On text-to-image tasks, M* cuts latency by 1.3 times. On image editing, the advantage balloons to 2.6 times.
These wins come from treating the whole stack as one scheduling problem. The sharding is baked in. You set a single tp_size parameter in a placement file.
That automatically handles parallel linears, vocab-parallel embeddings, and shards the MoE and KV cache across GPUs using NCCL. No single component is forced to fit on one chip anymore.
Most systems are built for one thing. They are specialized, brittle, and slow down when you change the model. M* is built for the loop itself.
It makes the scheduler the bottleneck, then eliminates it. The future of serving these massive models looks less like a collection of custom tools and more like one very smart, very fast engine. This is that engine.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Benchmark shows Claude Fable 5 passes only 3% of tasks, 31 of 91 fail 50%
From Archives Nobel laureate John Jumper departs DeepMind for Anthropic after AlphaFold win
Common Questions Answered
How does M*'s overlapped scheduling reduce GPU idle time in multimodal model serving?
M* overlapped scheduling runs the GPU and prepares the next batch simultaneously on different streams, eliminating the idle wait time that occurs when the CPU schedules batches. By pushing back the completion check by one cycle, the processing loop never truly stops, continuously keeping the GPU occupied with useful work.
What latency improvements does M* achieve on text-to-image and image editing tasks?
M* cuts latency by 1.3 times on text-to-image tasks and achieves an even more significant 2.6 times latency reduction on image editing tasks. These substantial improvements demonstrate the effectiveness of treating the entire serving stack as a unified scheduling problem.
How does M*'s single tp_size parameter simplify distributed GPU configuration?
M* uses a single tp_size parameter in a placement file to automatically handle all parallel distribution concerns including parallel linears, vocab-parallel embeddings, and sharding of the MoE and KV cache across GPUs using NCCL. This eliminates the need to manually fit individual components onto single chips, streamlining the configuration process.
What generic primitive does M* use to support multimodal model serving across different input types?
M* is built over a generic primitive called Loop that works for any modality type including text, images, and speculative decoding. This universal foundation allows the overlapped scheduling approach to benefit all multimodal tasks regardless of input type.
Further Reading
- M*: A Modular, Extensible, Serving System for Multimodal Models — arXiv
- M* (M-star): A Modular, Extensible, Serving System for Multimodal ... — Stanford University
- TriInfer: Hybrid Disaggregated Scheduling for Multimodal LLM Serving — MLSys 2026
- Modality-aware Scheduling for Multimodal Large Language Model Serving — arXiv
- SpaceServe: Spatial Multiplexing of Complementary Encoders and LLMs for Multimodal Large Model Serving — NeurIPS 2025
