Perplexity Open Sources Lily: A Rust + Metal Inference Engine for Qwen3.6-35B-A3B on Apple Silicon

Perplexity has open sourced Lily, the native inference engine behind Hybrid Compute in Perplexity Computer. It is a single-process runtime: a Rust layer masses the checkpoint and drives the era loop, an OpenAI-compatible chat-completions API streams tokens, and hand-written Metal kernels execute the mannequin. Neither PyTorch nor MLX sits within the execution path. Lily is intentionally slim with one mannequin, Qwen3.6-35B-A3B, on one {hardware} household and that narrowness is the efficiency argument.

Is it deployable? Yes. A standalone demo is public within the pplx-garden repository. A Rust and Metal inference server providing grasping textual content era via a minimal OpenAI-compatible HTTP API. The 4-bit checkpoint is nineteen.4 GB, so an Apple silicon Mac with 32 GB or extra of unified reminiscence is the reasonable flooring; Perplexity’s transport Hybrid Compute product lists macOS 15+, 24 GB minimal and 32 GB for finest outcomes.

Why specialize in any respect?

The default Mac stack is MLX plus MLX-LM, which already ships a Qwen implementation with grouped knowledgeable work, a fused recurrent Metal kernel, and GQA-aware consideration. But its operations should keep reusable throughout architectures. Lily offers that up and places mannequin construction, execution plans, and kernel choice in a single runtime.

Three workload shapes

Qwen3.6-35B-A3B shops 35B parameters and prompts roughly 3B per token. A router scores 256 specialists and picks eight, alongside one shared knowledgeable that sees each token. It additionally mixes 10 full-attention layers utilizing grouped-query attention (16 question heads, two KV heads) with 30 Gated DeltaNet layers. That yields three patterns: uneven knowledgeable teams, consideration over a rising KV cache, and a fixed-size recurrence.

Prefill: hold weights packed, hold routing on the GPU

The checkpoint makes use of groupwise affine 4-bit quantization, each group of 64 weights sharing a bfloat16 scale and bias, about 70 GB of bfloat16 weights compressed to 19.4 GB. Metal 4 tensor operations devour bfloat16, so weights have to be reconstructed first. Lily does that one tile at a time contained in the grouped GEMM, holding ends in threadgroup reminiscence and accumulating in FP32, so the expanded array by no means reaches unified reminiscence. In Perplexity’s ablation that fusion raised end-to-end prefill 77.4% at a 512-token immediate.

Keeping the routing histogram, prefix scan, scatter and block map inside a single GPU command buffer added 89% at 512 tokens by eradicating CPU synchronization inside every MoE layer. Moving from 16-row to 32-row tiles with 4 simdgroups added 13.2% at 2K; a register-resident Gated DeltaNet scan added 5.6%. Expert GEMMs are roughly 90% of prefill time. Long prompts run in bounded chunks so momentary activations don’t compete with weights and cache for reminiscence.

Decode: reduce bytes moved per token

Batch-1 decode has nearly no weight reuse, so bandwidth units the ceiling. One recorded step launched 795 kernels forming 555 sequential phases; Lily data actual dependencies in a concurrent Metal move so unbiased kernels overlap. The chosen token is written straight into the subsequent step’s GPU-resident enter slot, eradicating a per-token CPU spherical journey, and 4 kernel chains are fused to maintain intermediates in registers.

Coalesced cache reads lifted key bandwidth from 33.8 to 47.9 GB/s and worth bandwidth from 42.0 to 61.8 GB/s. GQA packing, 4 question heads sharing one threadgroup so every KV row masses as soon as, improved decode 23.8% at 32K. A fixed-block consideration format at 32K and above improved decode 7.7% at 32K, 27.4% at 64K, and 40.2% at 128K.

Results

On one 40-core, 128 GB M5 Max at batch 1, loading an identical 4-bit checkpoint bytes in opposition to MLX-LM’s quickest direct-generation path throughout ten lengths from 256 to 128K tokens, Lily averaged 4,156 prefill tokens/s versus 3,388 (1.23x) and 170.0 decode tokens/s versus 126.4 (1.35x). At a 4K immediate and 4K context it reached 5,749.9 and 186.6 tokens/s in opposition to 4,737.5 and 140.9, and was sooner at each recorded level: 1.12–1.42x prefill, 1.31–1.37x decode. A teacher-forced test throughout 192 positions put Lily’s perplexity 0.04% greater, with the identical top-ranked token 96.35% of the time.