Editorial illustration for Tsinghua AI Open-Sources AKV Cache Server for LLM Agents on SGLang
Tsinghua Open-Sources AKV Cache Server for LLM Agents
Researchers at Tsinghua University and the Chinese University of Hong Kong have open-sourced AKV, a serving system that changes how large language model agents manage memory during long tasks. The code is Apache-2.0 licensed and built on top of SGLang. Tested on three models, GPT-OSS-120B, MiniMax-M2.7, and MiroThinker-1.7-Mini, across four agent benchmarks, AKV cut average cost by 27.7% and raised average task score by 1.7 points compared to threshold summarization. The biggest gain showed up on GPT-OSS-120B running SWE-bench Verified, where cost per task dropped from 20.0 to 9.7 cents while the score climbed from 40.0 to 43.2.
The system runs on NVIDIA GPUs, with the paper's benchmarks done on 8 A800-80GB cards in BF16, and a quick-start setup that serves GPT-OSS-120B on just two GPUs using tensor parallelism. AKV ships as a serving system plus a policy library, no model weights included.
The core idea is what the researchers call cache engineering, a departure from the usual approach of summarizing an agent's conversation history to save memory. That distinction, and why it matters for how agents retain information from earlier in a task, is where things get interesting.
Tsinghua University and CUHK researchers open-sourced AKV, a serving system for KV cache engineering in LLM agents. Instead of summarizing old history, agents drop selected KV cache entries and keep reusing the rest. Across 3 models and 4 agent benchmarks, average cost falls 27.7%.
Why this matters
For anyone running agents on long horizons, the KV cache has become the actual bottleneck, not model weights. AKV's approach, dropping stale cache entries instead of compressing them into summaries, sidesteps the information loss that plagues Text Engineering baselines while still cutting cost 21.9% against that same baseline and 27.7% against the Standard 96K-token summarization setup. A 1.7 point bump in task score on top of lower cost is the part worth sitting with: usually you trade accuracy for savings, not get both.
The fact that it's Apache-2.0 and built directly on SGLang matters more than the benchmark numbers themselves. Teams already running SGLang in production don't need to rearchitect their serving stack to try this, they can test it against BrowseComp-Plus or SWE-bench Verified style workloads directly. Tsinghua and CUHK testing across GPT-OSS-120B and MiniMax-M2.7 at very different parameter scales suggests the policy generalizes rather than being tuned to one model's quirks. Worth watching whether this gets adopted into mainline serving frameworks, or stays a research curiosity.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google to End Free Access to Gemini AI for Non-Subscribers
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Nvidia open-sources Flora, robot video labeling 125x cheaper than GPT-6 Astra
From Archives Grok's AI chooses Republicans, OpenAI's picks Democrats when forced to pick
Common Questions Answered
What is AKV and how does it improve KV cache management for LLM agents?
AKV is a serving system developed by Tsinghua University and CUHK researchers that manages KV cache memory for large language model agents during long tasks. Instead of summarizing old history like traditional approaches, AKV selectively drops stale KV cache entries while keeping the rest reusable, which reduces information loss and improves performance.
What are the performance improvements AKV achieved across the tested models and benchmarks?
AKV demonstrated significant improvements when tested on three models (GPT-OSS-120B, MiniMax-M2.7, and MiroThinker-1.7-Mini) across four agent benchmarks. The system achieved an average cost reduction of 27.7% compared to threshold summarization and raised the average task score by 1.7 points, while also cutting costs 21.9% against Text Engineering baselines.
How does AKV's approach differ from traditional KV cache summarization methods?
AKV drops selected KV cache entries instead of compressing old history into summaries, which avoids the information loss that commonly affects summarization-based baselines. This selective dropping approach maintains better information fidelity while still achieving substantial cost reductions, making it more effective for long-horizon agent tasks.
What is the licensing and technical foundation of the open-sourced AKV system?
AKV is open-sourced under the Apache-2.0 license and is built on top of SGLang, making it accessible for researchers and developers to integrate into their projects. The open-source release enables the community to leverage this KV cache engineering solution for their own LLM agent implementations.
Why does AKV address a critical bottleneck in running long-horizon LLM agents?
For long-running agents, the KV cache has become the actual bottleneck rather than model weights, making efficient cache management essential for performance. AKV solves this problem by intelligently managing cache entries, allowing agents to operate efficiently over extended task horizons without the typical performance degradation or prohibitive costs associated with traditional approaches.
Further Reading
- Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang - MarkTechPost
- Cache Engineering for Agents - HyperAI
- Cache Engineering for Agents: Tsinghua and CUHK Introduce AKV Serving Engine - AICoder
- SGLang vs vLLM: Your Agent Is Slow Because You Picked the Wrong Cache - DEV Community
- llm-d 1.0, an Open Source Stack for Distributed Inference - Google Cloud Blog
