Editorial illustration for 16B-Parameter Diffusion Models Generate Multi-Speaker, Multilingual Speech
Apple's 16B Diffusion Model Generates Multilingual Speech
16B-Parameter Diffusion Models Generate Multi-Speaker, Multilingual Speech
Apple researchers have trained a continuous diffusion speech model with 16 billion parameters on tens of millions of hours of conversational audio, producing a system that can generate emotive, multi-speaker, multilingual speech without ever converting sound into discrete tokens. The work, described in a paper titled "Scaling Properties of Continuous Diffusion Spoken Language Models" from Jason Ramapuram, Eeshan Gunesh Dhekane, Amitis Shidani, Dan Busbridge, Bogdan Mazoure, Zijin Gu, Russ Webb, Tatiana Likhomanenko and Navdeep Jaitly, tackles a problem that has dogged speech-only language models for years: they trail far behind text and text-speech systems, and the usual fix, discretizing audio for autoregressive training, creates its own bottlenecks in computation and data.
The team's answer is to test whether continuous diffusion offers a cleaner path. To measure how well these models actually capture spoken language, they introduce a new metric, phoneme Jensen-Shannon divergence, or pJSD, which scores linguistic quality directly rather than relying on proxies borrowed from text models. What they find when they push these models to scale, and how their behavior compares to autoregressive systems, sets up the rest of the paper's results.
Scaling CD SLMs to 16B parameters with tens of millions of hours of conversational data enables generation of emotive, prosodic, multi-speaker, multilingual speech, though achieving long-form coherence remains a significant challenge. Figure 1: Scaling law fit for validation loss. Training (â¢) and testing (Ã) points are shown alongside compute-optimal points (â ).
Figure 2: The curvature κ of isoFLOPs at their optima decreases as compute increases: flattening corresponds to approximately two orders of magnitude expansion in the range of model (ÎN) and dataset (ÎD) sizes yielding a loss within ε of the optimum L*. Thus, higher compute allows near-optimal performance across a much wider variety of parameter-to-data allocations, opening up an efficient inference frontier.
Why this matters
The team behind this work, Ramapuram, Dhekane, Shidani, and colleagues at Apple, just showed that continuous diffusion spoken language models scale the way text transformers do: bigger models and more data buy you predictable gains, up to 16B parameters trained on tens of millions of hours of conversational audio. That's the real news for anyone building voice products. Emotive, multi-speaker, multilingual generation isn't a party trick anymore, it's an emergent property of scale, following a fitted scaling law rather than clever architecture tricks alone.
For researchers, that's a useful signal: the diffusion approach to speech is now on the same predictable curve autoregressive text models rode a few years back, which means the next gains are mostly a compute-and-data problem. For founders, the honest caveat matters more than the headline number. The paper is explicit that long-form coherence remains unsolved even at 16B parameters, so anyone pitching production-ready long conversational agents off this result should be pressed on exactly how they're handling that gap.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Guide Shows How Python Connects to Existing AI Models via Custom Requests
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives PathMoE Study Shows More Concentrated, Robust Routing Paths
From Archives Five Small, Open-Weight Models Built for Agentic Tool Calling
Common Questions Answered
What is the key advantage of Apple's continuous diffusion speech model over token-based approaches?
Apple's 16-billion parameter continuous diffusion model generates speech without converting sound into discrete tokens, enabling more natural and emotive speech synthesis. This approach allows the model to capture the full nuance of speech patterns while maintaining prosodic qualities across multiple speakers and languages.
How does scaling affect the performance of continuous diffusion spoken language models according to the research?
The research demonstrates that continuous diffusion spoken language models follow predictable scaling laws similar to text transformers, where larger models and more training data consistently produce measurable improvements in performance. The team achieved these gains by scaling to 16 billion parameters trained on tens of millions of hours of conversational audio data.
What are the current limitations of the 16B-parameter diffusion model despite its capabilities?
While the model excels at generating emotive, multi-speaker, and multilingual speech, achieving long-form coherence remains a significant technical challenge. This limitation suggests that further research and optimization is needed for applications requiring extended, contextually consistent speech generation.
Why is the emergence of multi-speaker, multilingual generation capabilities significant for voice product developers?
Multi-speaker, multilingual generation is no longer a specialized feature but an emergent property of scaling continuous diffusion models, making it accessible to voice product developers building at scale. This democratization of advanced speech synthesis capabilities enables more sophisticated and diverse voice applications without requiring specialized architectural innovations.
