Editorial illustration for Google Cloud Ships llm-d 1.0 With Cache-Aware TPU and GPU Routing
Google Cloud llm-d 1.0: Smart TPU/GPU Routing
Google Cloud pushed llm-d to version 1.0 this week at the PyTorch Conference in San Francisco, moving its distributed inference project out of preview and into production territory. The open-source, Kubernetes-native stack takes individual vLLM or SGLang model servers and knits them into a single coordinated pool, which matters most for self-hosted agent and reinforcement learning workloads where standard round-robin load balancing falls apart under uneven, bursty traffic.
The project isn't a Google-only effort. Founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, llm-d now has more than 500 contributors across over 25 organizations and sits in the CNCF Sandbox, accepted this past March under Apache 2.0 licensing. Red Hat, IBM and Google lead maintenance.
Features like prefix-aware routing, KV cache offloading, prefill/decode disaggregation and wide expert parallelism already existed before this release. What 1.0 adds is stability for actually running those pieces in production, what IBM engineers refer to as day 2 operations: multi-tenant quality of service controls, fairness-aware flow control and a settled API surface meant to stop changing underneath operators.
Google team have released llm-d 1.0, the first stable version of its distributed inference stack. This open-source, Kubernetes-native layer turns vLLM or SGLang model servers into one coordinated pool.
Why this matters
llm-d 1.0 matters less as a feature list and more as an admission: serving large models at scale has become its own engineering discipline, separate from training them. The 17,000-node GKE benchmark is the real headline here, not the individual features like prefix-aware routing or KV cache offloading, most of which were already in earlier releases. What's new is Google and IBM calling this "day 2 operations," a tacit acknowledgment that running these systems reliably in production is harder than building them in the first place.
For teams running self-hosted agents or RL pipelines, where request patterns are bursty and unpredictable, round-robin load balancing was never going to cut it, and that's the gap this fills. The multi-tenant quality-of-service controls are worth watching closely. Priority and fairness-aware flow control sounds good on paper, but these mechanisms tend to get tested hardest when multiple teams share a cluster and someone's workload starts starving everyone else's.
Whether llm-d holds up under that kind of contention, outside Google's own benchmark, is the open question.
Related Reading
Browse all Google news →
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
Open Source Box CEO Aaron Levie Says AI Will Drive "Biggest Cybersecurity Upgrade
Open Source Study: 6.7% of Deepfake Requests on Hugging Face Targeted Children
Related Topic Google to End Free Access to Gemini AI for Non-Subscribers
LLMs & Generative AI
Related Topic NVIDIA and Google Cloud let developers scale AI from prototype to production
Market Trends
From Archives Deepseek open-sources inference engine powering its fastest AI model
From Archives Google Joins Amazon in Nvidia Fight Over AI App Developers and PyTorch
From Archives Google open-sources TorchTPU, a native PyTorch backend 50% faster than Strict
From Archives Anthropic, Google block OpenCode API keys after hackers ran up user charges
LLMs & Generative AI
Common Questions Answered
What is llm-d 1.0 and how does it improve distributed inference?
llm-d 1.0 is Google Cloud's open-source, Kubernetes-native distributed inference stack that coordinates multiple vLLM or SGLang model servers into a single unified pool. Unlike standard round-robin load balancing, it uses cache-aware TPU and GPU routing to handle uneven, bursty traffic patterns common in agent and reinforcement learning workloads, making it significantly more efficient for production deployments.
Why is cache-aware routing important for llm-d 1.0's performance?
Cache-aware routing optimizes how requests are distributed across TPU and GPU clusters by considering KV cache state and prefix-aware routing patterns. This prevents the inefficiencies of traditional round-robin load balancing, which fails under the uneven traffic patterns typical of self-hosted agent and reinforcement learning applications.
What does the 17,000-node GKE benchmark demonstrate about llm-d 1.0?
The 17,000-node Google Kubernetes Engine benchmark demonstrates that llm-d 1.0 can reliably scale and manage large distributed inference clusters in production environments. This benchmark is positioned as the most significant achievement, highlighting that serving large models at scale has become a distinct engineering discipline requiring specialized operational capabilities.
How does llm-d 1.0 address 'day 2 operations' for large language models?
llm-d 1.0 acknowledges that running large language model systems reliably in production goes beyond initial deployment, requiring ongoing operational management and optimization. The stack provides the infrastructure and routing intelligence needed to maintain stable, efficient performance across distributed model serving clusters over extended periods.
What is the relationship between llm-d 1.0 and vLLM or SGLang model servers?
llm-d 1.0 acts as a coordination layer that takes individual vLLM or SGLang model servers and integrates them into a single coordinated pool with intelligent request routing. Rather than replacing these model servers, llm-d 1.0 enhances their capabilities by providing distributed load balancing and cache-aware routing across multiple instances.
Further Reading
- IBM-backed llm-d framework adds controls for running AI models in-house - SiliconANGLE
- llm-d 1.0 Ships the Serving Layer DeepSeek Withheld. The Sponsors Sell the Managed Version. - Forkast
- Announcing llm-d 1.0: Open distributed inference for agents on Google Cloud TPUs and GPUs - Google Cloud
- About GKE Inference Gateway powered by llm-d - Google Cloud Documentation
