Editorial illustration for Real-Time Diffusion on Apple M3 Ultra: CoreML, Quantization, Neural Engine
Real-Time Diffusion on Apple M3 Ultra: CoreML,...
The promise of real-time image generation on a laptop has long felt like a distant ambition, until now. We set out to push diffusion models to their limits on Apple’s M3 Ultra, and what we found upended many assumptions about what works on unified memory. Through a systematic gauntlet of techniques, CoreML conversion, quantization, Token Merging, Neural Engine acceleration, frame interpolation, and more, we measured each against a single question: does it actually speed things up?
The answer, repeatedly, was no. Quantization offered no gains. Parallel inference stalled.
The Neural Engine, so powerful for small models, choked on diffusion’s scale. But one combination broke through: a distillation-specialized model, SDXS-512, converted to CoreML and fed by a three-thread camera pipeline. That delivered 22.7 frames per second at 512x512 resolution, real-time camera-to-image transformation, live.
The deeper contribution is a map of a radically different optimization landscape. What works on NVIDIA’s CUDA architecture often backfires here. We lay out the rules, the dead ends, and the practical path forward for diffusion on Apple Silicon.
We explored a wide range of techniques including CoreML conversion, quantization, Token Merging, Neural Engine utilization, compact model exploration, frame interpolation, kNN search-based synthesis, pix2pix-turbo, optical flow frame skipping, and knowledge distillation, quantitatively evaluating the effectiveness of each approach. Ultimately, by combining CoreML conversion of the distillation-specialized model SDXS-512 with a 3-thread camera pipeline, we achieved real-time camera img2img transformation at 22.7 FPS at 512x512 resolution. The primary contribution of this work is the systematic demonstration that optimization insights established for CUDA are not necessarily effective on Apple Silicon's unified memory architecture. We reveal an optimization landscape fundamentally different from that of NVIDIA GPUs -- including the absence of speedup from quantization, the ineffectiveness of parallel inference, and the unsuitability of the Neural Engine for large-scale models -- and provide practical guidelines for diffusion model inference on Apple Silicon.
This is not a story of incremental gains on familiar ground. It is a map of a foreign territory. The M3 Ultra does not reward the habits we learned from CUDA: quantization gives no speed, parallel inference stalls, and the Neural Engine, so promising in theory, chokes on diffusion models.
Instead, the winning path demanded a distillation-first architecture, a lean pipeline, and the refusal to assume that tricks from one ecosystem port cleanly to another. Twenty-two point seven frames per second at 512x512. Real-time camera img2img on a single chip.
That result is not the endpoint; it is evidence that Apple Silicon demands its own playbook. The guidelines are now written. The rest is execution.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Evaluating AI Agents: Does the Engine Grasp Instructions and Reason Facts?
From Archives AgentWall adds runtime safety layer for local AI agents' actions
Common Questions Answered
What performance did the M3 Ultra achieve for real-time diffusion model inference?
The M3 Ultra achieved 22.7 frames per second for real-time diffusion model inference, demonstrating that laptop-based image generation is now practically feasible. This represents a significant breakthrough in bringing diffusion capabilities to consumer hardware without requiring dedicated GPUs.
Why did traditional CUDA optimization techniques fail on Apple's M3 Ultra?
The M3 Ultra does not reward habits learned from CUDA environments, as quantization provided no speed improvements, parallel inference caused stalls, and the Neural Engine struggled with diffusion models despite theoretical promise. The winning approach required a fundamentally different architecture focused on distillation-first design rather than porting existing optimization tricks.
What techniques were used to optimize diffusion models for CoreML on M3 Ultra?
The optimization process involved CoreML conversion, quantization, Token Merging, and a lean pipeline architecture specifically designed for Apple's unified memory architecture. These techniques were systematically tested to overcome the unique constraints of the M3 Ultra's Neural Engine and GPU capabilities.
How does the M3 Ultra's unified memory architecture benefit diffusion model performance?
The M3 Ultra's unified memory architecture upended many assumptions about what optimization strategies work on Apple Silicon, allowing for more efficient data movement between the CPU, GPU, and Neural Engine. This architectural advantage enabled the development of a distillation-first approach that prioritizes memory efficiency over traditional parallel processing methods.
Further Reading
- Systematic Optimization of Real-Time Diffusion Model Inference on Apple M3 Ultra — arXiv
- Stable Diffusion with Core ML on Apple Silicon — Apple Machine Learning Research
- Making Apple Neural Engine work in a custom inference stack — Draw Things Engineering
- Overview — Guide to Core ML Tools — Apple Core ML Tools Documentation
- On Device Llama 3.1 with Core ML — Apple Machine Learning Research
