Editorial illustration for ATOM Serves AgentX Workloads on AMD Instinct MI355X GPUs With 35.0K-Token Calls
ATOM Runs 195K-Token Agent Calls on AMD MI355X
ATOM Serves AgentX Workloads on AMD Instinct MI355X GPUs With 35.0K-Token Calls
Call #48 in a recorded coding-agent session sends 195.5K tokens to the model. Only 1.3K of those are new. The other 194.2K are history the agent already sent before, resent because that's how the conversation works: every tool call, every sub-agent result, every prior turn gets packed back into the context window. Multiply that across 198 calls in a single session and the waste compounds fast.
This is the shape of agentic inference, and it breaks the assumptions most serving stacks are built on. Coding agents, research assistants, and multi-step tool-using systems don't look like chatbots. They generate long shared histories, fire off bursts of parallel sub-agent calls, then sit idle waiting on a tool to return. A GPU that's great at single-shot generation can still choke on this pattern if it can't keep a session's history cached and ready, or if it stalls decode every time a new request shows up mid-session.
AMD's ATOM engine, running on Instinct MI355X GPUs, was built to handle exactly this. Tested against SemiAnalysis's AgentX benchmark using the Kimi-K3 model, ATOM's approach to memory, scheduling, and compute gets put through two recorded sessions that expose where naive serving falls apart.
On eight AMD Instinct™ MI355X GPUs, ATOM serves Kimi-K3 at up to 17.5K total tok/s per chip with 56 concurrent sessions (input plus output tokens, cached input included) and a P90 interactivity of 152 tok/s per user with a single session on the SemiAnalysis InferenceX AgentX benchmark (run of 2026-10-15, chart as of 2026-10-23; Figure 9).
Why this matters The headline number here isn't GPU throughput, it's waste: 195.5K tokens resent for 1.3K tokens of new work, repeated across three sub-agents that start from nearly identical prompts. That's the real cost center in agentic systems, and it's easy to miss if you're only watching FLOPS or benchmark scores. For teams building coding agents or multi-agent pipelines, ATOM's approach, caching and sharing history across forked calls rather than recomputing it, points to where serving infrastructure actually needs to evolve.
Buying faster silicon like the MI355X helps, but it doesn't fix architectural waste baked into how agents call models. If three sub-agents all open with the same 35K-token prefix, that's a serving-layer problem, not a hardware one. Founders evaluating agent infrastructure should ask less about raw compute specs and more about how a system handles redundant context across parallel calls.
Researchers studying agent efficiency now have a concrete pattern to test against. The gap between "fast GPU" and "cheap agent session" is where the next round of infrastructure competition will actually play out.
Common Questions Answered
What is the main inefficiency problem that ATOM addresses in agentic inference workloads?
ATOM tackles the problem of massive token reuse in agent conversations, where only 1.3K out of 195.5K tokens sent in a single call are new, with the remaining 194.2K being historical context resent repeatedly. This waste compounds across multiple tool calls and sub-agent results, making it a critical cost center that traditional serving stacks don't optimize for. By caching and sharing history across forked calls rather than recomputing it, ATOM significantly reduces this redundant token processing.
What performance metrics does ATOM achieve on AMD Instinct MI355X GPUs for the AgentX benchmark?
On eight AMD Instinct MI355X GPUs, ATOM serves Kimi-K3 at up to 17.5K total tokens per second per chip while handling 56 concurrent sessions. The system achieves a P90 interactivity of 152 tokens per second per user with a single session on the SemiAnalysis InferenceX AgentX benchmark, demonstrating strong performance for multi-agent workloads.
Why is monitoring GPU throughput alone insufficient for evaluating agentic inference systems?
Traditional metrics like FLOPS and benchmark scores don't capture the real cost center in agentic systems, which is token waste from redundant context resending. In coding agents and multi-agent pipelines, the efficiency of handling repeated historical context is more important than raw GPU throughput, since most tokens processed are cached or reused rather than new computations.
How does ATOM's caching approach differ from traditional serving stacks for multi-agent workloads?
ATOM caches and shares history across forked calls rather than recomputing it each time, which is fundamentally different from how most serving stacks handle agent conversations. This approach directly addresses the problem where three sub-agents starting from nearly identical prompts would normally resend the same context multiple times, eliminating this redundant processing.
Further Reading
- Single Node and Distributed Inference Performance on AMD Instinct MI355X GPU - AMD
- Serving LLMs for Agents on AMD Instinct: Long Context, Fast Tokens, Real Workloads - AMD Developer Community
- Serving 64Mi-Token Contexts on One AMD Instinct MI355X Node - AMD ROCm Blog
- Don’t Break the Cache: An Evaluation of Prompt Caching - arXiv
- How We Investigated Prompt Cache Misses with Datadog - Datadog