Perplexity's Lily inference engine running Qwen3.6-35B on Apple Silicon, showcasing AI model efficiency.

Editorial illustration for Perplexity Open Sources Lily Inference Engine for Qwen3.6-35B on Apple Silicon

Perplexity Open Sources Lily Inference Engine

By AI Daily Post Edited by Brian Petersen, Editor-in-Chief

September 3, 2026 • 4 min read

Perplexity has put its local inference engine out in the open. Lily, the runtime that powers Hybrid Compute in Perplexity Computer, now sits in the pplx-garden repository for anyone to inspect or run. It's a single-process setup: Rust loads the checkpoint and drives generation, an OpenAI-compatible chat-completions API streams the output, and hand-written Metal kernels do the actual compute work.

No PyTorch. No MLX. Just one model, Qwen3.6-35B-A3B, tuned for one hardware family, Apple silicon.

That narrowness is the whole point. Most Mac-based inference runs through MLX and MLX-LM, a general-purpose stack built to stay flexible across different model architectures. Lily throws that flexibility away on purpose, hardcoding model structure and kernel choices into the runtime itself. The 4-bit checkpoint runs 19.4 GB, which puts 32 GB of unified memory as the practical minimum, in line with what Perplexity already recommends for Hybrid Compute (macOS 15+, 24 GB floor, 32 GB for real headroom).

Qwen3.6-35B-A3B itself is an unusual target: 35 billion parameters total, but only about 3 billion active per token, split across a mixture-of-experts router and a hybrid attention design.

On one 40-core, 128 GB M5 Max at batch 1, loading identical 4-bit checkpoint bytes against MLX-LM’s fastest direct-generation path across ten lengths from 256 to 128K tokens, Lily averaged 4,156 prefill tokens/s versus 3,388 (1.23x) and 170.0 decode tokens/s versus 126.4 (1.35x).

Perplexity Open Sources Lily: A Rust + Metal Inference Engine for Qwen3.6-35B-A3B on Apple Silicon - MarkTechPost

Why this matters

Perplexity just handed developers a working example of what it looks like to abandon the "one runtime for every model" assumption that PyTorch and MLX both bake in. By writing Metal kernels directly against Qwen3.6-35B-A3B's specific routing pattern, 256 experts, eight active plus one shared, they're betting that generality has a real performance cost, and that betting on a single model and hardware pair can beat frameworks built to handle anything. For teams shipping a fixed model on fixed hardware, that's a legitimate architecture question worth stealing.

We'd temper the enthusiasm, though. Lily's narrowness is the whole trick: it works because Perplexity isn't trying to support your model, your GPU, or your next fine-tune. That's a fine tradeoff for a product team running one checkpoint in production, and a bad template for anyone who needs flexibility. Open sourcing it is genuinely useful as a reference for hand-tuned Metal kernel work on Apple Silicon, but the interesting test is whether anyone outside Perplexity forks it for a different model rather than just reading it for ideas.

Common Questions Answered

What is Lily and how does it differ from PyTorch and MLX frameworks?

Lily is Perplexity's open-source local inference engine that powers Hybrid Compute in Perplexity Computer, now available in the pplx-garden repository. Unlike PyTorch and MLX which aim to be general-purpose frameworks supporting any model, Lily is purpose-built as a single-process Rust runtime with hand-written Metal kernels specifically optimized for the Qwen3.6-35B-A3B model on Apple silicon hardware.

What performance improvements does Lily achieve compared to MLX-LM on Apple silicon?

According to benchmarks on a 40-core, 128 GB M5 Max, Lily averaged 4,156 prefill tokens/s versus MLX-LM's 3,388 tokens/s (a 1.23x improvement) and 170.0 decode tokens/s versus MLX-LM's 126.4 tokens/s (a 1.35x improvement). These performance gains were measured across ten different token lengths ranging from 256 to 128K tokens using identical 4-bit checkpoints.

How is Lily's architecture structured and what components does it use?

Lily uses a single-process setup where Rust loads the model checkpoint and drives generation, an OpenAI-compatible chat-completions API streams the output, and hand-written Metal kernels perform the actual compute work. This architecture deliberately avoids relying on PyTorch or MLX, instead leveraging direct Metal kernel optimization for Apple silicon.

Why did Perplexity choose to optimize Lily for only one model and hardware pair instead of building a general framework?

Perplexity's strategy demonstrates that specialization can outperform generality by writing Metal kernels directly against Qwen3.6-35B-A3B's specific routing pattern with 256 experts (eight active plus one shared). By betting on a single model and hardware pair rather than supporting every model on any hardware, Lily achieves measurable performance advantages over general-purpose frameworks.

Ship an AI product this weekend — no engineers required.

Structured, in-depth lessons on the exact no-code tools — not scattered tutorials.

The exact platforms, taught in depth
Build real, working projects
Our honest review + a reader discount

Read the review →