AMD Launches ROCm KV Cache Tech for LLM Inference

AMD put a name to a problem that's been quietly eating GPU budgets across the inference stack: KV cache management. On Wednesday, the company introduced ROCm AMD Infinity Context, a storage tier built specifically to hold the key-value cache data that large language models generate during inference, and opened it up for use with existing AMD Instinct GPU deployments. Support starts with the Instinct MI300X and MI350 Series, with the MI450 Series and Helios platform coming next.

The timing tracks with where inference workloads are headed. Models like GPT-OSS-120B and Qwen3 235B now handle context windows far larger than what shipped even a year ago, and agentic workflows with multi-turn conversations keep piling more state into memory that GPU High-Bandwidth Memory was never sized to hold. Until now, teams dealing with that overflow had three options, and AMD says none of them work well: recompute the cache from scratch at real cost, pin it to node-local NVMe storage that doesn't travel, or offload it to general-purpose network file systems too slow for production inference. ROCm AIC is AMD's answer to that gap, built as an open, AI-native tier for distributed serving.