|

Meet FreeToken: An Edge-Native MoE Serving Engine that Runs 753B GLM-5.2 on a Single Workstation GPU

ByRicardo

Frontier open-weight fashions are transport quicker than the {hardware} assumptions round them. Kimi-K3, GLM-5.2 and DeepSeek-V4-Flash are closing the aptitude hole with proprietary methods, however releasing parameters solely determines who can get hold of a mannequin — not who can afford to run it. Serving them nonetheless assumes datacenter-class GPU clusters, and as agentic workloads push inference demand up, that value lands hardest on particular person builders and small groups. Meanwhile, greater than a hundred million shopper machines already carry discrete GPUs. A group of researchers from UC Berkeley and UT Austin suggest FreeToken. The analysis group argued the lacking piece is just not {hardware} however a serving system: it treats a private machine as a unified, elastic inference platform slightly than a small GPU, and constantly maps computation and mannequin state onto no matter GPU, CPU, reminiscence and interconnect bandwidth the machine really has. The result’s a 35B mannequin at interactive pace on an 8 GB laptop computer GPU, 284B on a gaming desktop, and the 753B GLM-5.2 on a single workstation card.

Is it deployable?

Yes, FreeToken is Apache-2.0 on GitHub, revealed on PyPI as freetoken v0.1.2 (uv pip set up "freetoken[accel]"), and shipped as a one-click desktop app for Windows and Linux at flashml.ai. The CLI targets Linux x86_64 with an NVIDIA GPU on driver r580+ (CUDA 13). ft serve exposes OpenAI- and Anthropic-compatible endpoints on port 1919, and ft launch claude wires up Claude Code, Codex, OpenCode or OpenClaw towards your individual field.

Who it suits: solo builders, startups and SMB engineering groups whose agent token payments already exceed the price of a GPU they personal; enterprises ought to deal with it as an air-gapped or regulated-workload path, not a datacenter alternative. Strongest trade match: healthcare and authorized (knowledge by no means leaves the machine), protection, finance, and IP-heavy R&D. Typical functions: native coding brokers, personal code overview, offline contract evaluation, synthetic-data era, batch evals.

The hole it targets

Mixture-of-Experts makes native frontier inference arithmetically possible. DeepSeek-V4-Flash prompts 6 of 256 routed consultants in every of 43 layers, so solely 13B of its 284B parameters take part in any single token. Sparsity doesn’t shrink the knowledgeable pool, although — at FP4 the complete set is roughly 140 GB, so inactive consultants sit in host reminiscence and enter the execution path on demand.

The analysis group isolates three failure modes in present engines (llama.cpp, KTransformers, Ollama, MoE-Infinity):

  • Prefill destroys sparsity: Thousands of tokens per layer route to almost the entire knowledgeable set, so a prefill go streams the whole pool throughout PCIe — about two seconds on an RTX 5090, 5 on PCIe 4.0 desktops, ten or extra on the x8 hyperlinks frequent in laptops.
  • Static placement misses decode site visitors: llama.cpp assigns MoE tensors at load time; KTransformers pins a “sizzling” subset. Routing shifts each token, so most knowledgeable evaluations fall to the CPU whereas the GPU and the PCIe hyperlink sit idle.
  • Consumer CPUs can’t carry the rest: Dual-channel DDR5 delivers 80–90 GB/s towards the 1–1.8 TB/s an RTX 4090 or 5090 attracts from on-package reminiscence.