Editorial illustration for Apple Silicon Mac Studio Cluster Benchmarked for Distributed LLM Inference
Apple Silicon Cluster Runs Trillion-Param LLM Fast
Apple Silicon Mac Studio Cluster Benchmarked for Distributed LLM Inference
Apple's MLX team put four Mac Studios on a desk, wired them together with RDMA over Thunderbolt, and ran a model with nearly a trillion parameters without touching a data center. The result: 36 tokens per second out of Muse Spark Open, a frontier-class model that would normally need rack-mounted GPUs to move. That's the headline number in a new benchmarking post from the team behind MLX, the open-source array framework built for Apple silicon's unified memory architecture.
The setup starts smaller than four machines, though. A single Mac Studio, loaded with up to 512GB of unified memory, can already handle large open-weight models on its own. The scaling story comes in when that one machine isn't enough: Thunderbolt-based RDMA lets multiple Studios pool resources and split the work, turning a desktop into something closer to a small cluster.
The benchmarks also pit two generations of hardware against each other: the new M5 Ultra, with its GPU Neural Accelerator built for matrix multiplication, against the older M3 Ultra. Using the MLX LM package, the team tested a range of popular models to see how much that architecture shift actually changes in practice.
Time to first token (TTFT) is essentially compute bound, and that is where the Neural Accelerators in the M5 Ultra GPU shine: the specialized hardware support for fast matrix-multiplication of the GPU Neural Accelerator directly translates into significant speedups for TTFT, up to 4.6Ã faster compared to an M3 Ultra (see Figure 2).
Why this matters
For anyone building or testing large models outside the cloud, this benchmark is a useful data point: a cluster of Mac Studios running MLX with RDMA over the jaccl backend can handle distributed inference on an 816B-parameter model without renting a GPU farm. That's a real shift for researchers and small teams who want to prototype at scale on hardware they own and control, keeping weights and prompts private in the process. The command-line simplicity here, launching a multi-machine benchmark with a single mlx.launch call and a hostfile, suggests Apple is lowering the barrier to distributed experimentation rather than keeping it locked to specialized clusters.
We'd push back on reading too much into one architecture's numbers, though. Apple silicon's unified memory is well suited to this kind of workload, but the real test is whether these setups hold up against production-grade GPU clusters on throughput and cost per token, not just feasibility. Still, for founders weighing whether to buy Mac Studios instead of cloud credits, this is the kind of concrete benchmark worth watching closely.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google to End Free Access to Gemini AI for Non-Subscribers
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Cracking AI's 'Random' Number: Why One Guess Is Not Enough, Toy Case Shows
From Archives YouTube adds "Made with AI" labels to altered or fully generated songs
Common Questions Answered
What throughput did Apple's Mac Studio cluster achieve when running the Muse Spark Open model?
Apple's MLX team achieved 36 tokens per second when running Muse Spark Open, a frontier-class model with nearly a trillion parameters, across a cluster of four Mac Studios connected via RDMA over Thunderbolt. This performance level is typically associated with rack-mounted GPU setups in data centers, demonstrating that Apple silicon can handle distributed LLM inference at scale without traditional enterprise infrastructure.
How much faster is the M5 Ultra GPU compared to the M3 Ultra for time to first token according to the benchmark?
The M5 Ultra GPU is up to 4.6x faster than the M3 Ultra for time to first token (TTFT) performance. This speedup is attributed to the specialized Neural Accelerators in the M5 Ultra GPU, which provide direct hardware support for fast matrix-multiplication operations that are essential for computing the first token.
What is the key advantage of using MLX with RDMA over Thunderbolt for distributed inference on Mac Studios?
Using MLX with RDMA over Thunderbolt allows researchers and small teams to run distributed inference on large models like the 816B-parameter model without renting expensive GPU farms or relying on cloud infrastructure. This approach enables users to keep model weights and prompts private while maintaining control over the hardware they own, making large-scale model prototyping more accessible and cost-effective.
What is MLX and how is it designed for Apple silicon?
MLX is an open-source array framework specifically built for Apple silicon's unified memory architecture, enabling efficient distributed computing across multiple Mac devices. The framework leverages Apple's hardware capabilities to facilitate large-scale model inference without requiring traditional data center infrastructure.
Why does the Mac Studio cluster setup represent a significant shift for researchers developing large language models?
The Mac Studio cluster demonstrates that frontier-class models with hundreds of billions of parameters can be run on owned hardware rather than rented cloud GPU infrastructure, providing researchers with cost savings, data privacy, and hardware control. This capability shifts the economics and accessibility of large-scale AI development, allowing smaller teams and independent researchers to prototype and test models at scale without enterprise-level cloud commitments.
Further Reading
- Explore distributed inference and training with MLX - Apple Developer
- Four Mac Studios Run a Trillion-Parameter AI Model Over Thunderbolt 5 - BitcoinVersus
- MLX Distributed Training Arrives: 3x Faster Fine-Tuning and 1 Trillion-Parameter Models - Sean Kim Blog
- MLX Distributed Inference: Multi-Mac Cluster Setup for Large Language Models - Contra Collective
- Production-Grade Local LLM Inference on Apple Silicon - arXiv
