Editorial illustration for LightSeek launches TokenSpeed, cutting LLM latency by half vs TensorRT-LLM
LightSeek launches TokenSpeed, cutting LLM latency by...
Large language models are only as fast as their inference engine. LightSeek Foundation just pulled the rug out from under that assumption. With the release of TokenSpeed, a new MIT-licensed, open-source inference engine, they’re targeting the very workloads that define the next generation of AI: agentic, real-time, batch-heavy.
The headline number is stark: on typical decode tasks with speculative decoding at batch sizes 4, 8, and 16, TokenSpeed nearly halves latency compared to TensorRT-LLM. That’s not a marginal gain. It’s a structural shift.
Under the hood, a C++ finite-state machine enforces KV cache safety at compile time, while the execution plane stays in Python, a deliberate trade-off that buys usability without sacrificing speed. On an NVIDIA B200, TokenSpeed already beats TensorRT-LLM by 9% in min-latency and 11% in throughput at 100 tokens per second per user on Kimi K2.5. And the MLA kernel alone cuts decode latency in half.
This is open-source performance that doesn’t ask for compromises.
Combined with other optimizations, this nearly halves latency relative to TensorRT-LLM on typical decode workloads with speculative decoding at batch sizes 4, 8, and 16 with long prefix KV cache.
Key Takeaways
- TokenSpeed is a new MIT-licensed, open-source LLM inference engine by LightSeek Foundation, built specifically for agentic workloads. (Available in preview mode)
- Its scheduler uses a C++ finite-state machine to enforce KV cache safety at compile time, while keeping the execution plane in Python for usability.
- On NVIDIA B200, TokenSpeed outperforms TensorRT-LLM by ~9% in min-latency and ~11% in throughput at 100 TPS/User on Kimi K2.5.
- The TokenSpeed MLA kernel nearly halves decode latency vs.
This is not a modest gain. It is a halving of latency on the decode workloads that define modern agentic inference, long prefixes, speculative decoding, batch sizes that actually matter. LightSeek’s architecture bet on a C++ finite-state machine to enforce KV cache safety at compile time, then handed the execution back to Python for flexibility.
That bet paid off. On Kimi K2.5 at 100 TPS per user, TokenSpeed delivers 9% lower minimum latency and 11% better throughput than TensorRT-LLM. But the real edge?
The MIT license. Open source, preview mode, built for the stack that demands responsiveness. TensorRT-LLM just got a benchmark it cannot ignore.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives CLIP-FP8 Model Matches CLIP-FP16 Quality; Patch Embedding Quantizers Matter
From Archives Automation updates AI context morning with active threads, key dates, note
Common Questions Answered
How much latency reduction does TokenSpeed achieve compared to TensorRT-LLM?
TokenSpeed nearly halves latency on typical decode tasks with speculative decoding at batch sizes 4, 8, and 16 compared to TensorRT-LLM. On Kimi K2.5 at 100 TPS per user, TokenSpeed delivers 9% lower minimum latency and 11% better throughput than TensorRT-LLM, making it significantly faster for modern agentic inference workloads.
What is the architectural approach that enables TokenSpeed's performance gains?
TokenSpeed uses a C++ finite-state machine to enforce KV cache safety at compile time, then hands execution back to Python for flexibility. This hybrid architecture approach allows the engine to optimize performance-critical operations while maintaining the ease of development that Python provides.
What types of AI workloads is TokenSpeed designed to optimize?
TokenSpeed is specifically designed for agentic, real-time, and batch-heavy workloads that define the next generation of AI applications. The engine excels at handling decode tasks with long prefixes, speculative decoding, and meaningful batch sizes that are critical for modern inference scenarios.
What license does TokenSpeed use and how is it distributed?
TokenSpeed is released as an MIT-licensed, open-source inference engine by LightSeek Foundation. This open-source approach makes the technology accessible to developers and organizations looking to optimize their LLM inference performance.
Further Reading
- TokenSpeed: A Speed-of-Light LLM Inference Engine for Agentic ... — LightSeek
- TokenSpeed is a speed-of-light LLM inference engine. - GitHub — GitHub - LightSeek
- LLM Inference Benchmarking: Performance Tuning with TensorRT-LLM — NVIDIA Developer Blog
- Overview — TensorRT-LLM - GitHub Pages — NVIDIA TensorRT-LLM Docs
