Editorial illustration for CUDA Agent Hits 98.8% Pass Rate, Outperforms PyTorch Benchmarks
CUDA Agent Hits 98.8% Pass Rate, Beats PyTorch
A large language model can write CUDA that runs. Getting it to write CUDA that runs fast is a different problem, and it's the one ByteDance Seed and Tsinghua AIR set out to fix with a new system called CUDA Agent. The team's baseline model, Seed1.6, already passes 74% of tasks on KernelBench, a standard suite for testing GPU kernel generation.
But passing isn't the same as winning: those kernels beat PyTorch's own compiler, torch.compile, on just 27.2% of tasks, running at a 0.69x geometric-mean speedup. In practice, that means the model's code was slower than what an automated compiler produces without any AI involved at all.
CUDA Agent's fix is to stop treating kernel generation as a one-shot text prediction task and instead put the model inside an actual development loop, complete with profiling tools, correctness checks, and a locked-down sandbox, then train it with reinforcement learning over long, 131,072-token contexts. ByteDance and Tsinghua ran that training for 150 PPO steps. What came out the other end is a system that doesn't just write correct kernels. It writes fast ones.
ByteDance Seed and Tsinghua AIR have released CUDA Agent, an agentic reinforcement learning system that trains a large language model to write GPU kernels that beat a compiler.
Why this matters
The jump from a 0.69× to a 2.11× geomean speedup against torch.compile is the kind of number that should make anyone writing CUDA kernels by hand pay attention. ByteDance and Tsinghua AIR aren't claiming their model writes more correct code, Seed1.6 was already passing 74% of KernelBench. They're claiming it writes faster code, reliably, at a 98.8% pass rate. That's a different problem, and a more commercially useful one: compilers are conservative by design, and there's real money sitting in the gap between "correct" and "fast" on expensive GPU clusters.
The weak spot is Level 3, where pass rate drops to 94% and the speedup gains shrink too. Composite kernels and full model graphs are harder than single operators or short operator chains, which tracks with how these benchmarks are usually structured. For teams evaluating this, the Level 2 numbers (100% pass, 2.80× over torch.compile) are the headline, but Level 3 is where the real infrastructure workloads live. Worth watching whether that gap closes with scale or whether it's a structural ceiling for agentic RL on kernel generation.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
Related Topic NVIDIA and Google Cloud let developers scale AI from prototype to production
Market Trends
Related Topic NVIDIA NeMo powers telco reasoning model for autonomous network workflows
Industry Applications
From Archives Claude Sonnet 5 Launches With User-Adjustable Effort Level
From Archives Anthropic Watermarks AI Output, Some Clients Request It
From Archives Nvidia Cuts OpenAI Deal Amid USD 350 Billion Chip Talks
Business & Startups
From Archives Indonesia Opens First University AI Center with UGM, Indosat, NVIDIA
Research & Benchmarks
Common Questions Answered
What is CUDA Agent and how does it improve upon the baseline Seed1.6 model?
CUDA Agent is an agentic reinforcement learning system developed by ByteDance Seed and Tsinghua AIR that trains a large language model to write GPU kernels optimized for performance. While the baseline Seed1.6 model passes 74% of tasks on KernelBench, CUDA Agent achieves a 98.8% pass rate and improves performance from 0.69x to 2.11x geometric-mean speedup against PyTorch's torch.compile compiler.
How does CUDA Agent's performance compare to PyTorch's torch.compile on KernelBench?
CUDA Agent demonstrates significant improvements over torch.compile, achieving a 2.11x geometric-mean speedup compared to the baseline Seed1.6's 0.69x speedup. This represents a substantial leap in GPU kernel generation performance, with CUDA Agent-generated kernels beating torch.compile's output on a much higher percentage of tasks.
What is the key difference between CUDA Agent's focus and traditional compiler optimization?
CUDA Agent prioritizes writing faster code reliably rather than simply writing more correct code, which is the traditional focus of compilers. Since compilers are conservative by design, CUDA Agent's approach of generating optimized kernels at a 98.8% pass rate addresses a more commercially useful problem for GPU kernel development.
What benchmark suite is used to test CUDA Agent's GPU kernel generation capabilities?
KernelBench is the standard suite used to test GPU kernel generation performance for CUDA Agent and other systems. The benchmark allows researchers to measure both the correctness of generated kernels and their performance relative to existing compilers like torch.compile.
Further Reading
- CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation - arXiv
- ByteDance Seed and Tsinghua AIR Introduces CUDA Agent: A Large-Scale Agentic RL System for CUDA Kernel Generation - MarkTechPost
- ByteDance Trained an AI Agent That Writes Faster CUDA Kernels Than torch.compile - AwesomeAgents.ai
- Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization - arXiv
- ByteDanceのAIエージェントが業界標準のコンパイラよりも高速にCUDAカーネルを生成：「CUDA Agent」が示す次世代のGPU最適化とは - Gnews.jp
