Editorial illustration for Liquid AI's Pipette Benchmarks 1,000+ On-Device Model Configurations
Liquid AI Benchmarks 1,000+ On-Device Model Configs
A model card says a model hits a certain quality score. It usually doesn't say what happens when that same model gets quantized down and shoved onto a Snapdragon chip inside a Galaxy S26 Ultra. Liquid AI is betting that gap matters more than the industry has admitted.
This week the company open-sourced Pipette, a benchmarking suite built with Artificial Analysis acting as an independent check on the methodology. The core idea is simple to state and expensive to execute: stop measuring models in isolation and start measuring the full configuration that actually ships, meaning the model paired with a specific quantization, a specific runtime, and a specific device. Liquid AI's launch dataset already runs deep, covering more than 1,000 combinations across 30-plus models, llama.cpp builds on macOS, iOS, Windows and Android, and context windows from 256 up to 8,192 tokens.
Verified numbers so far come from three real devices: a MacBook Pro with an M5 Max chip, an iPhone 17 Pro, and a Galaxy S26 Ultra. The question Pipette is built to answer is blunt: does a model that looks fine on paper actually hold up once it's running on the hardware someone will actually use it on?
Pipette treats on-device behavior as a property of the deployed system, not the model in isolation. Its unit of measurement is a full configuration: model + quantization + runtime + device.
Why this matters
Model cards have always been a poor proxy for what happens once a model lands on someone's phone, and Pipette is an admission of that from a company that builds edge models for a living. For developers, the value is in the specificity: 30+ models, multiple quantization formats, four operating systems, context windows from 256 to 8,192 tokens. That's the kind of granularity you need to answer a question model cards never could, like whether a 4-bit quantized model on an aging Android chipset actually holds up at 4,096 tokens of context.
Bringing in Artificial Analysis as an outside validator is a smart move if Liquid AI wants the benchmark to be trusted rather than treated as marketing for its own hardware story. We'd still want to see how other model makers respond, whether they submit their own configurations or wait to see if Pipette becomes a standard. Open-sourcing the suite lowers the bar to check, which is the point.
Watch whether third parties start publishing Pipette numbers unprompted.
Related Reading
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
Open Source OpenAI's GPT Transcribe Cuts Error Rate to 3.31%, Improving on GPT-4o
Open Source Study: 6.7% of Deepfake Requests on Hugging Face Targeted Children
From Archives Report Details Missed Warnings in OpenAI Security Incident
From Archives OpenAI Details Hack Timeline, Says Incidents Unrelated
Common Questions Answered
What is Pipette and why did Liquid AI open-source it?
Pipette is a benchmarking suite that measures the actual performance of on-device AI models by testing complete system configurations rather than models in isolation. Liquid AI open-sourced Pipette with Artificial Analysis as an independent methodology check because the company recognized that traditional model cards fail to accurately represent how models perform once deployed on consumer devices like smartphones.
How does Pipette's approach to benchmarking differ from traditional model cards?
Pipette treats on-device behavior as a property of the entire deployed system, measuring model + quantization + runtime + device as a single unit, rather than evaluating models in isolation as traditional model cards do. This comprehensive approach reveals performance gaps that model cards never capture, providing developers with realistic expectations for edge deployment scenarios.
What specific configurations does Pipette benchmark across?
Pipette benchmarks over 1,000 on-device model configurations, testing 30+ models with multiple quantization formats across four operating systems and context windows ranging from 256 to 8,192 tokens. This granularity allows developers to find optimal configurations for specific hardware constraints, such as determining whether a 4-bit quantized model performs adequately on older devices.
Why is the gap between model cards and actual on-device performance significant?
Model cards typically report quality scores for unoptimized models, but they don't account for what happens when those models are quantized and deployed on specific hardware like Snapdragon chips in smartphones. Liquid AI believes this gap matters more than the industry has acknowledged, and Pipette addresses this by providing realistic performance data for the actual deployed configurations developers will use.
Further Reading
- Introducing Pipette: A benchmarking suite for on-device ... - Liquid AI
- Pipette Overview | On-Device Model Benchmarks - Liquid AI
- Liquid AI's Pipette Tests 1000+ On-Device AI Setups—and ... - Superpower Daily
- Liquid AI Launches Pipette Because On-Device AI Benchmarks Need Adult Supervision - Silicon Snark
- LFM2.5 Q4_0: Quantization-Aware Distillation for Edge Deployment - Liquid AI
