Editorial illustration for FreeToken Serves 753B Model on Single GPU, Fixing Existing Engine Flaws
753B Model Runs on Single GPU With FreeToken
A workstation with a single GPU just ran GLM-5.2, a 753-billion-parameter model, at usable speed. That claim comes from a team at UC Berkeley and UT Austin behind a new serving engine called FreeToken, and it cuts against the assumption baked into most inference software: that big open-weight models need datacenter GPU clusters to run. Kimi-K3, GLM-5.2 and DeepSeek-V4-Flash have narrowed the gap with closed models this year, but publishing weights only settles who can download a model, not who can afford the hardware to serve it.
Agentic workloads are pushing inference demand higher still, and that cost falls hardest on solo developers and small teams without cluster budgets. There are already more than a hundred million consumer machines with discrete GPUs sitting under desks and in laptops, mostly idle for this kind of work. FreeToken is built on the premise that the bottleneck isn't hardware at all.
It's the serving layer. The system treats a personal machine, GPU, CPU, memory, interconnect, as one elastic pool rather than isolated resources, and reallocates across all of them continuously.
On an 8 GB RTX 4060 laptop the NVFP4 build serves 35B at 39.3 tok/s — above the 33 tok/s median decode speed measured for Codex in production traces. On a single RTX PRO 6000, GLM-5.2 (753B, 40B active) runs at 14.9 tok/s versus llama.cpp’s 7.3.
Why this matters
FreeToken's real contribution isn't the 753B headline number, it's the diagnosis. Prefill breaking sparsity assumptions is exactly the kind of engineering detail that gets glossed over when everyone's racing to publish benchmark screenshots. If thousands of tokens per layer really do route across nearly the whole expert set, then the entire pitch of "MoE models are cheap because only a fraction activates" quietly falls apart the moment you're doing prefill instead of decode. That's a gap between marketing and memory bandwidth that a lot of teams building on llama.cpp or Ollama forks haven't priced in.
For developers and small teams eyeing GLM-5.2 or Kimi-K3 on a single workstation GPU, this matters more than the parameter count. The gate here isn't whether you can download the weights, it's whether your PCIe bus can survive prefill without becoming the bottleneck. We'd want to see FreeToken's fix benchmarked against KTransformers and MoE-Infinity on real agentic workloads, not just synthetic prompts, before anyone treats "one GPU, one frontier model" as solved.
Related Reading
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
Open Source OpenAI's GPT Transcribe Cuts Error Rate to 3.31%, Improving on GPT-4o
Open Source Study: 6.7% of Deepfake Requests on Hugging Face Targeted Children
From Archives DRAM Shortages Lift Nvidia AI Server Prices by 15 Percent
From Archives Vercel's Free 'Is Agentic' Tool Audits Sites With 100+ Ora Checks
Common Questions Answered
How does FreeToken enable running a 753-billion-parameter model on a single GPU?
FreeToken is a new serving engine developed by researchers at UC Berkeley and UT Austin that optimizes inference for large models on consumer-grade hardware. The engine successfully ran GLM-5.2, a 753-billion-parameter model, at usable speeds on a single RTX PRO 6000 GPU, achieving 14.9 tokens per second compared to llama.cpp's 7.3 tokens per second.
What performance improvements does FreeToken demonstrate on laptop GPUs?
On an 8 GB RTX 4060 laptop, FreeToken's NVFP4 build serves a 35-billion-parameter model at 39.3 tokens per second, which exceeds the 33 tokens per second median decode speed measured for Codex in production traces. This demonstrates that FreeToken can deliver production-grade inference speeds even on modest consumer hardware.
What flaw in existing MoE serving engines does FreeToken address?
FreeToken identifies a critical issue with how prefill operations interact with sparsity assumptions in mixture-of-experts models. The engine discovered that during prefill, tokens route across nearly the entire expert set rather than just a fraction, which breaks the core assumption that MoE models are cost-efficient because only a small portion of experts activate.
Why is FreeToken's engineering diagnosis more significant than its headline performance numbers?
While running a 753-billion-parameter model on a single GPU is impressive, FreeToken's real contribution is exposing how prefill operations contradict the efficiency claims of mixture-of-experts architectures. This engineering insight reveals that the entire pitch of MoE models being cheap becomes questionable during prefill operations, not just decode phases.
What is the difference between what FreeToken enables versus simply downloading open-weight models?
Publishing model weights only determines who can download a model, but FreeToken solves the practical problem of who can actually run these models efficiently. The serving engine bridges the gap between having access to large open-weight models like GLM-5.2, Kimi-K3, and DeepSeek-V4-Flash and being able to serve them at usable speeds on consumer hardware.
Further Reading
- Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution - arXiv
- FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution - AlphaXiv
- A 753 billion parameter model ran on a single workstation GPU - GroundTruth
- Song Han/Zaharia/Stoica System Runs 753B MoE on One Workstation GPU - AI Weekly
- UC Berkeley just open-sourced FreeToken. (2–4x faster local LLM ... - LinkedIn
