Editorial illustration for TensorRT Edge-LLM Runs MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor
TensorRT Edge-LLM Runs MLPerf Edge Agentic Benchmark...
NVIDIA's TensorRT Edge-LLM finished the MLPerf Inference v6.1 Edge Agentic benchmark in 24 minutes and 36 seconds on a single Jetson AGX Thor Developer Kit, compared to 2 hours and 37 minutes for the llama.cpp reference submission. That's a 6.4x gap on the same hardware, running the same Qwen3.6-27B model, at 52.33 tokens per second across all 1,007 turns in the workload.
The benchmark itself is new territory for MLPerf. Instead of scoring a single prompt-response pair, Edge Agentic replays actual software-engineering agent trajectories: 20 conversations, 1,007 turns, with input length climbing to roughly 23.5K tokens by the end. Each turn requires the model to read a growing conversation history, issue a tool call, read back the tool's result, and keep reasoning without losing track of earlier context. That's a different kind of pressure than chatbot-style inference, and it's meant to reflect what agents built for robots, vehicles, and other edge devices actually have to do once they're deployed outside a data center.
NVIDIA's result leaned on NVFP4 quantization, tree-based multi-token prediction, and KV cache reuse to hit that speed on device hardware with fixed power and memory limits.
In the MLPerf Inference v6.1 Edge Agentic benchmark, NVIDIA TensorRT Edge-LLM ran Qwen3.6-27B on a single NVIDIA Jetson AGX Thor Developer Kit. The system achieved 52.33 tokens per second and completed all 1,007 turns of the performance workload in 24 minutes and 36 seconds, 6.4x faster than the llama.cpp reference submission of 2 hours and 37 minutes.
Why this matters
Agentic workloads are a different beast than chatbot inference, and this benchmark is the first real signal of how that difference plays out on hardware you can actually bolt onto a robot or a vehicle. A 6.4x speedup on Qwen3.6-27B running locally on Jetson AGX Thor, with 128 GB of unified memory, tells us NVIDIA is optimizing for the specific pain points of agent loops: long shared context, repeated tool calls, tight power budgets. That's a narrower and more useful signal than another cloud throughput chart.
For developers building on-device agents, robotics teams, autonomous systems, this is the number to watch when deciding whether reasoning stays local or gets shipped to a data center. We'd caution against reading one SingleStream result on one dev kit as a generalizable win. MLPerf submissions are optimized benchmarks, not production deployments, and MLCommons hasn't published wider comparative data across vendors yet.
Still, if edge agentic inference becomes a real category with real competition, this is the opening bid, and it puts pressure on Qualcomm, Intel, and AMD to answer with their own numbers.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives NVIDIA's Vera Rubin NVL72 Leads MLPerf Inference Debut
From Archives Former OpenAI Researcher Builds AI That Judges Options, Not Writes Text
Further Reading
- Accelerating LLM and VLM Inference for Automotive and Robotics with NVIDIA TensorRT Edge-LLM - NVIDIA Technical Blog
- MLPerf Inference v6.1: pioneering agent, VLM benchmarks - Lambda
- Call for Submission: Edge Agentic Inference Benchmark ... - MLCommons
- TensorRT Edge-LLM on Jetson - Jetson AI Lab
- Build Next-Gen Physical AI with Edge-First LLMs for Autonomous Vehicles and Robotics - NVIDIA Technical Blog
