Editorial illustration for LiveKit Updates Voice AI Benchmark to 10k Token Prompts, Citing Real-World Use
LiveKit Voice AI Benchmark Expands to 10k Tokens
LiveKit pushed its voice AI benchmark to 10,000-token prompts this month, telling developers that shorter test cases don't match how production agents actually get used. The company's reasoning ties into a bigger problem with how teams pick inference APIs in the first place: they lean on time to first token, or TTFT, a metric borrowed straight from chat applications where it works fine. Voice is a different animal.
A text-to-speech engine can't start talking on a fragment; it needs a full clause before it generates audio, which means the gap between "the model started responding" and "the user heard something" can stretch far beyond what TTFT alone would suggest. LiveKit's own deployment notes on Gemma 4 introduced a companion metric, time-to-first-sentence, to capture that gap. The distinction matters because a voice agent is really a latency budget with a language model wedged inside it, and every stage of that budget, from speech-to-text through the LLM to text-to-speech, spends milliseconds a caller can actually hear.
A provider that posts a fast TTFT but a slow token-per-second rate can still leave an agent sounding sluggish or getting talked over.
A text-to-speech model cannot synthesize half a word. It needs a complete clause or sentence before it produces audio. LiveKit calls the resulting metric time-to-first-sentence (TTFS), and argues in its Gemma 4 deployment post that TTFS is what users actually feel.
Why this matters
A 1k-token benchmark was measuring a fantasy. Nobody ships a voice agent with a bare system prompt; by the time you've loaded persona instructions, escalation policy, retrieved context, and tool schemas, you're at 10k tokens before the user says a word. LiveKit's update matters because it forces a harder question: not "how fast does this API respond," but "how fast does it respond once you've fed it what production actually requires." That's a less flattering number for a lot of vendors, and a more useful one for anyone building.
For developers and founders picking an inference provider, this is a reminder that headline latency figures are marketing artifacts until you test them against your own prompt size. TTFT alone was never the full story, since a TTS layer can't speak from a token, it needs a clause. If you're benchmarking voice stacks, run your own prompt lengths through the comparison rather than trusting a vendor's best-case number. The gap between "looks fast in a demo" and "doesn't get interrupted in production" is exactly what this benchmark is trying to close.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Google AI's EnvHarness Makes Static AI Training Worlds Adaptable
From Archives AI Agents Solved Tasks but Couldn't Track Time Accurately
Common Questions Answered
Why did LiveKit update its voice AI benchmark from 1k to 10k token prompts?
LiveKit updated the benchmark because shorter test cases don't reflect how production voice agents are actually used in real-world scenarios. A typical production deployment requires system prompts, persona instructions, escalation policies, retrieved context, and tool schemas, which collectively add up to approximately 10,000 tokens before users even speak.
What is the difference between TTFT and TTFS metrics for voice AI?
Time to first token (TTFT) is a metric borrowed from chat applications that measures how quickly an API responds to input. However, LiveKit introduced time-to-first-sentence (TTFS) as a more appropriate metric for voice AI, since text-to-speech engines cannot synthesize partial words and require complete clauses or sentences before producing audio output.
Why can't text-to-speech models start producing audio on incomplete tokens?
Text-to-speech models cannot synthesize half a word or incomplete fragments because they need linguistic context to properly generate audio. A text-to-speech engine requires a complete clause or sentence before it can produce intelligible audio output, making TTFS a more meaningful performance measurement than TTFT for voice applications.
How does the 10k token benchmark change how developers should evaluate inference APIs for voice agents?
The 10k token benchmark forces developers to ask a more realistic question: not just how fast an API responds in ideal conditions, but how fast it responds when loaded with all the production requirements voice agents actually need. This results in less flattering performance numbers for many APIs compared to benchmarks using minimal token counts, providing a more accurate representation of real-world performance.
Further Reading
- Benchmarking models for voice - LiveKit Community
- A Real World Benchmark for Evaluating Voice AI Systems - arXiv
- Voice Showdown: The First Arena for Voice AI - Scale AI
- Most voice AI benchmarks still don’t reflect how ... - LinkedIn
- The State of Voice AI Instruction Following in 2026 - Coval
