Editorial illustration for Research Team Launches Open ASR Leaderboard to Benchmark 60+ Speech Recognition Models
Open ASR Leaderboard Reveals Top Speech Recognition Models
Toss the marketing slides. The choice of a speech recognition model is now clear, definitive, and frankly unpleasant. A new public benchmark—from Hugging Face, Nvidia, Cambridge, and Mistral AI—has put over 60 models from 18 companies through a standardized wringer.
It measures English transcription, performance in five other languages, and handling of long audio. The results aren't revolutionary. They are a blunt verdict: you can have speed, or accuracy, or multilingual skill.
You cannot have the best of all three.
A research group from Hugging Face, Nvidia, the University of Cambridge, and Mistral AI has released the Open ASR Leaderboard, an evaluation platform for automatic speech recognition systems. The leaderboard is meant to provide a clear comparison of open source and commercial models. According to the project's study, more than 60 models from 18 companies have been tested so far.
The evaluation covers three main categories: English transcription, multilingual recognition (German, French, Italian, Spanish, and Portuguese), and long audio files over 30 seconds. The last category highlights how some systems perform differently on long versus short recordings. Two main benchmarks are used: - Word Error Rate (WER) measures the number of incorrect words.
- Inverse Real-Time Factor (RTFx) measures speed. For example, an RTFx of 100 means one minute of audio is transcribed in 0.6 seconds. To keep comparisons fair, transcripts are normalized before scoring.
The process removes punctuation and capitalization, spells out numbers, and drops filler words like "uh" and "mhm." This matches the normalization standard used by OpenAI's Whisper. speed The leaderboard shows clear differences between model types in English transcription. Systems built on large language models deliver the most accurate results.
Nvidia's Canary Qwen 2.5B leads with a WER of 5.63 percent. However, these accurate models are slower to process audio. Simpler systems, like Nvidia's Parakeet CTC 1.1B, transcribe audio 2,728 times faster than real time, but only rank 23rd in accuracy.
Multilingual models lose some specialization Tests across several languages show a trade-off between versatility and accuracy. Models narrowly trained on one language outperform broader multilingual models for that language, but struggle with others. Whisper models trained only on English beat the multilingual Whisper Large v3 at English, but can't reliably transcribe other languages.
In multilingual tests, Microsoft's Phi-4 multimodal instruct leads in German and Italian.
Forget a universal solution. This is the task's fundamental physics, and the leaderboard is the proof. Nvidia's Canary Qwen 2.5B hits a 5.63% Word Error Rate but lags.
Its Parakeet CTC 1.1B sibling is blisteringly fast yet ranks 23rd for accuracy. Microsoft's Phi-4 leads in German and Italian but cedes ground elsewhere. So you pick your primary pain.
Need a pristine archive transcript and have minutes? Choose accuracy. Live-captioning a meeting where two seconds is too long?
Choose speed. Parsing calls from Lisbon and Berlin with one tool? Choose versatility, and own the errors.
The data kills the guesswork. It also kills the fantasy. Your priorities are there, quantified.
Now pick one.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Indian Prodigy's AI "Supermemory" Attracts Top Tech Investors
From Archives AI Code Benchmarks Fail the "Vibe Check," Says New DeepMind Study
Common Questions Answered
Which research organizations are behind the Open ASR Leaderboard project?
The Open ASR Leaderboard was developed by a collaborative research team including Hugging Face, Nvidia, the University of Cambridge, and Mistral AI. This multi-institutional effort aims to provide a transparent and comprehensive evaluation of automatic speech recognition models.
How many speech recognition models are currently included in the Open ASR Leaderboard?
The leaderboard currently features over 60 models from 18 different companies, providing an extensive comparative framework for speech recognition technology. This comprehensive approach allows for a detailed and objective assessment of various AI speech recognition systems.
What languages are being tested in the Open ASR Leaderboard's multilingual recognition category?
The multilingual recognition category of the Open ASR Leaderboard includes tests for four languages: English, German, French, and Italian. This approach provides a broader understanding of speech recognition performance across different linguistic contexts.
Further Reading
- Open ASR Leaderboard tests more than 60 speech recognition models for accuracy and speed - Gnoppix Forum
- Open ASR Leaderboard Reveals Trends in Multilingual and Long-Form Speech Recognition - Hyper.ai
- ASR Leaderboard: Towards Reproducible and Transparent Benchmarking for Speech Recognition - arXiv
- Best open source speech-to-text (STT) model in 2026: Benchmarks and comparisons - Northflank
- Open ASR Leaderboard - A reproducible, transparent benchmarking platform for ASR systems - Emergent Mind
