Editorial illustration for ZML releases free tool to speed AI inference across mixed chips
ZML Releases Free Tool for Multi-Chip AI Inference
ZML releases free tool to speed AI inference across mixed chips
ZML, a Paris-based AI startup backed by Turing Award winner Yann LeCun, has released a free inference server called ZML/LLMD built to run open-source large language models across a mix of chips, Nvidia, AMD, Google's TPU, Apple Metal and Intel Arc among them. Founder Steeve Morin told TechCrunch the goal is to break down the walls that currently lock companies into a single chipmaker's ecosystem, letting them run models at peak speed, or faster, no matter what hardware sits underneath.
The timing matters. Inference, the actual work of processing prompts once a model is trained, has overtaken training as the costlier and more constant AI workload for most companies. Morin argues the infrastructure behind it remains messy, full of software mismatches and architecture quirks that push buyers toward whichever vendor they started with. A tool that runs equally well on several chip types could let enterprises and cloud providers shop around, mixing in cheaper or more power-efficient hardware where it makes sense.
That flexibility could ripple beyond cost savings. Morin pointed to newer chipmakers, many based in Europe, as potential beneficiaries if ZML's approach catches on.
With ZML/LLMD, the newly launched LLM inference server, the company’s ambition is to break existing silos and make different chips available for AI use cases at their maximum available speed, and sometimes faster, ZML founder Steeve Morin told TechCrunch.
Why this matters
ZML's bet is that inference, not training, is where the next real cost battle gets fought, and giving developers a free tool to route workloads across Nvidia, AMD, Google TPU, Apple Metal and Intel Arc chips is a direct shot at CUDA lock-in. LeCun's endorsement gives it credibility, but the real test is whether ZML/LLMD performs well enough on non-Nvidia silicon that teams actually switch, rather than just running benchmarks and going back to what works. For founders watching cloud bills climb, hardware flexibility is worth real money if the performance holds.
For researchers, this is another sign that the open-source LLM ecosystem is pushing hard against single-vendor dependency, chip by chip. We'd temper the excitement with a plain question: does mixing chips introduce enough complexity, in debugging, in latency spikes, in maintenance overhead, that the savings get eaten up anyway? Morin's framing about giving people "the power to create their own system" sounds right, but adoption numbers, not slogans, will tell us if enterprises actually trust it in production.
Related Reading
Open Source Trump cracks down on Anthropic after Amazon tip; staff largely foreign
Open Source SDOF Adds Two Defensive Layers via Intent Router and StateAwareDisp
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
From Archives Tencent's Hy3 Model Matches Larger Rivals With 21 Billion Parameters
From Archives AI Agent Skips Unneeded Tool Call After Observing Zero Precipitation
Common Questions Answered
What is ZML/LLMD and what problem does it solve?
ZML/LLMD is a free inference server released by Paris-based AI startup ZML that enables large language models to run across multiple chip types including Nvidia, AMD, Google TPU, Apple Metal, and Intel Arc. The tool breaks down vendor lock-in by allowing companies to run AI models at peak speed on any hardware, rather than being restricted to a single chipmaker's ecosystem.
Who is backing ZML and what is their background?
ZML is backed by Yann LeCun, a Turing Award winner and prominent figure in artificial intelligence. LeCun's endorsement provides significant credibility to the startup's mission of democratizing AI inference across different hardware platforms.
How does ZML/LLMD address CUDA lock-in?
ZML/LLMD provides a free tool that allows developers to route AI workloads across multiple chip architectures, directly challenging Nvidia's CUDA dominance. By enabling teams to achieve comparable or better performance on non-Nvidia silicon, the tool aims to break the dependency that currently locks companies into Nvidia's ecosystem.
Why does ZML believe inference is more important than training for cost optimization?
ZML's strategy focuses on inference rather than training because the company believes the next major cost battle in AI will be fought at the inference stage. By providing a free tool to optimize inference across multiple chips, ZML targets where organizations can achieve the most significant cost savings and operational flexibility.
What is the key challenge ZML/LLMD must overcome to succeed?
The real test for ZML/LLMD is whether it performs well enough on non-Nvidia hardware that development teams actually switch to using it in production, rather than just running benchmarks and reverting to familiar solutions. Performance parity across different chip types will determine whether the tool achieves meaningful adoption beyond theoretical demonstrations.
Further Reading
- ZML: A Zig-Based Inference Engine Bringing LLMs to AMD GPUs - Rits (NYU Shanghai)
- Test-Driving the LLMD Inference Engine by ZML - Hugging Face
- ZML's Hardware-Agnostic Distributed Inference - Chipstrat
- AI Anywhere: One Server, Any Chip, Forget about CUDA - YouTube
- ZML - Model to Metal - ZML.ai
