Editorial illustration for PrismML's Bonsai 27B Runs Qwen3.6 on Laptops With 1-bit and Ternary Builds
Bonsai 27B Runs Qwen on Laptops With 1-bit Compression
PrismML shipped Bonsai 27B this week, a pair of low-bit rebuilds of Qwen3.6-27B that fit on a laptop or, in the smaller case, a phone. Nothing about the underlying architecture changed. What did change is how the weights are stored.
Ternary Bonsai 27B packs each weight into {−1, 0, +1} at 1.71 bits, landing at 5.9GB. 1-bit Bonsai 27B goes further, using binary {−1, +1} weights at 1.125 bits for a 3.9GB footprint. Both releases are Apache 2.0 and both keep the model's multimodal setup intact, with roughly 24.8B language weights, a 0.46B vision tower held at 4-bit HQQ, and 2.5B parameters in embeddings and the LM head.
The 262K-token context survives the compression too, helped by the fact that about 75% of Qwen3.6-27B's attention is already linear.
The numbers only mean something next to what's already out there. Standard "4-bit" and "2-bit" quantizations of the same base model don't actually hit those bit counts once you measure the true average, and Bonsai's approach diverges sharply from BitNet's from-scratch training strategy. That comparison is where things get interesting.
Ternary Bonsai 27B uses {−1, 0, +1} weights at a true 1.71 bits per weight. Its ideal size is 5.9GB. 1-bit Bonsai 27B uses binary {−1, +1} weights at 1.125 bits per weight, for 3.9GB.
Why this matters
For developers building on-device apps, Bonsai 27B is a concrete answer to a real constraint: iOS caps a single app at roughly half of physical memory, and a 262K context window in FP16 KV cache blows past that at 17.2GB. Cutting to a 4-bit KV cache drops it to 4.3GB, which actually fits. That's the story here, not the model's intelligence but its footprint.
PrismML isn't claiming a smarter Qwen3.6, just a smaller one, and the 0.0011 nats output divergence against the FP16-KV baseline suggests the ternary compression holds up well against its own baseline. Whether that holds against harder benchmarks or longer real-world sessions is still an open question worth watching.
For founders shipping local-first products, the 3.9GB and 5.9GB sizes matter more than the benchmark scores. That's the difference between "runs on a phone" and "needs a server." We'd still want independent evals before trusting a 1-bit model in production, but the engineering here, keeping the vision tower separate, targeting real memory limits like Apple's, points at a practical trend: quantization work aimed at specific hardware ceilings rather than generic leaderboard chasing.
Related Reading
Open Source Trump cracks down on Anthropic after Amazon tip; staff largely foreign
Open Source SDOF Adds Two Defensive Layers via Intent Router and StateAwareDisp
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
From Archives Half of Fortune 500 Firms Use Hugging Face for Private, Open Source AI
From Archives ACRouter's AI model selector beats Opus-only setups by 2.6x on cost
Common Questions Answered
What are the key differences between Ternary Bonsai 27B and 1-bit Bonsai 27B?
Ternary Bonsai 27B uses weights stored as {−1, 0, +1} at 1.71 bits per weight, resulting in a 5.9GB model size, while 1-bit Bonsai 27B uses binary {−1, +1} weights at 1.125 bits per weight for a smaller 3.9GB footprint. The 1-bit version achieves greater compression but with slightly different weight quantization, allowing it to fit on phones and smaller devices more easily.
How does PrismML's Bonsai 27B maintain multimodal capabilities while reducing model size?
PrismML kept the multimodal setup intact in both Bonsai 27B variants by only changing how weights are stored through low-bit quantization, rather than modifying the underlying architecture of Qwen3.6-27B. This approach allows the models to retain their multimodal processing abilities while achieving significant size reductions through 1-bit and ternary weight compression.
Why is the reduced KV cache footprint important for on-device iOS applications?
iOS caps a single app at roughly half of physical memory, and the original FP16 KV cache with a 262K context window requires 17.2GB, which exceeds this limit and prevents deployment. By cutting to a 4-bit KV cache, the footprint drops to 4.3GB, which actually fits within iOS memory constraints and enables developers to build on-device applications with Bonsai 27B.
What is the output divergence of Bonsai 27B compared to the original FP16-KV version?
Bonsai 27B maintains minimal output divergence of 0.0011 nats against the FP16-KV baseline, demonstrating that the aggressive quantization to 1-bit and ternary weights does not significantly compromise the model's output quality. This low divergence indicates that the compression techniques preserve the model's performance characteristics despite the dramatic reduction in model size.
Under what license are the Bonsai 27B models released?
Both the Ternary Bonsai 27B and 1-bit Bonsai 27B are released under the Apache 2.0 license, making them freely available for commercial and non-commercial use with minimal restrictions. This open licensing approach allows developers to integrate these compressed models into their own projects and applications.
Further Reading
- The 27 billion-parameter AI 'Qwen3.6-27B' runs on iPhone 17 Pro alone after PrismML compression - GIGAZINE
- iPhone 17 Proで270億パラメータのAI「Qwen3.6-27B」が動作、AppleはPrismMLと圧縮技術の活用を協議か - GIGAZINE
- Running a 1-Bit LLM on Windows — Bonsai 8B Setup Guide - Dan Marshall's Blog
- Bonsai AI Tutorial: Run a 1-Bit LLM Locally On an Old Laptop - DataCamp
- ArmanJR/PrismML-Bonsai-vs-Qwen3.5-Benchmark - GitHub
