FreeToken - An open-source client-side MoE large-model inference system that supports full-capacity operation. - AiBoss

What is FreeToken?

FreeToken is an open-source edge MoE (Model-Based Inference) system for large models, jointly developed by UC Berkeley, MIT, and other institutions. Through technologies such as full-layer double buffering, bandwidth-adaptive hybrid scheduling, agent state reuse, and elastic hot-scaling of GPU memory, the system unifies the CPU, memory, PCIe, and GPU of a personal computer into an elastic computing platform. This enables a single consumer-grade card to run ultra-large MoE models such as Qwen3.6-35B and DeepSeek-V4-Flash 284B at full capacity, making local deployment of cutting-edge AI a reality.

The main functions of FreeToken

End-side full-health deductionSupports full-power operation of large MoE models such as Qwen3.6-35B and DeepSeek-V4-Flash 284B on consumer-grade single cards such as RTX 4060/5090.
Full-layer double-buffered prefetchWhen the GPU computes the current layer, the background prefetches the Expert weights of the next layer via PCIe streaming, completely eliminating I/O wait bubbles.
Bandwidth adaptive schedulingReal-time detection of PCIe bandwidth and CPU computing power; dynamic allocation of missed Expert data to GPU cache or CPU on-site computation, pushing the hardware throughput to its limit.
Agent state reuseLightweight checkpoints are set at special token boundaries, and context editing only prefills from the most recent anchor, avoiding duplicate calculations of the complete history.
Flexible memory thermal expansionWhen background applications preempt video memory, the GPU cache is seamlessly shrunk and the computation is handed over to the CPU, achieving a smooth degradation with zero downtime and no OOM (Out of Memory) errors.
Extremely fast and low latencyFirst token latency is reduced by 42–58%, Agent multi-turn interaction TTFT is reduced by 65–80%, and overall it is 2–4 times faster than Ollama.

FreeToken's technical principles

Full-layer double-buffered prefetch:During the Prompt processing phase, FreeToken achieves complete overlap between computation and data transfer. While the GPU is computing layer l, the Expert weights of layer l+1 have already been prefetched and streamed through the PCIe backend, which basically eliminates I/O waiting bubbles and prevents the Prefill process from being severely blocked by fetching Expert from system memory layer by layer.
Bandwidth adaptive hybrid scheduling:The runtime system detects the actual PCIe bandwidth and instantaneous CPU computing power of the machine in real time, and dynamically calculates the optimal traffic splitting ratio. A portion of the Experts hit the GPU's LRU cache, while the portion that misses is intelligently split according to the real-time bandwidth. Some of the data is transmitted to the GPU via PCIe, while some is directly computed in parallel on the CPU, pushing the hardware throughput to the theoretical limit of this topology.
Agent-oriented intelligent state reuse:For scenarios where Coding Agents or tool invocation models frequently fine-tune the context, FreeToken sets lightweight checkpoints at special token boundaries. When the context is edited, the system only needs to restore from the most recent valid anchor point and incrementally prefill the added suffix, without having to recalculate the entire context, which greatly reduces the redundant calculation overhead in multiple rounds of tool invocation and mind chain iteration.
Flexible memory with dynamic thermal expansion and contraction:In response to the reality that other applications on personal computers often preempt video memory, FreeToken supports dynamic hot resizing of the Expert Cache in the GPU without restarting the Serving service. When available video memory suddenly drops, the cache is seamlessly shrunk and more missed Experts are transferred to the CPU for computation, ensuring that the inference service is smoothly degraded without triggering a CUDA Out of Memory crash.

How to use FreeToken

Desktop App InstallationVisit the FreeToken website https://www.flashml.ai/ to download the Windows or Linux desktop app. After installation, you can start the inference service through the GUI interface.
quick installation via command line: Execute via command line uv pip install "freetoken[accel]" Install the CLI tool with one click.
Model loading and automatic schedulingLoad the supported MoE model weights, and the system will automatically complete CPU-GPU heterogeneous scheduling and bandwidth adaptive configuration.
Hardware configuration recommendationsWhen running large models such as DeepSeek-V4-Flash, it is recommended to have 32GB of video memory and at least 192GB of RAM to ensure stable and smooth performance.
Flexible video memory automatic managementIf a game or rendering process occupies video memory in the background, FreeToken will automatically hot-expand and shrink without manual intervention or service restart.

FreeToken's core advantages

Breaking down hardware barriersConsumer-grade single cards (such as RTX 4060/5090) can run ultra-large MoE models with parameters of 35B–753B at full power without the need for data center clusters.
Leading in reasoning speedIt is 2–4 times faster than Ollama and 1.46 times faster than llama.cpp. A laptop RTX 4060 can run Qwen3.6-35B at a speed of up to 39.3 tok/s.
The first token has extremely low latency.By using full-layer double buffering and bandwidth adaptive scheduling, the TTFT of long Prompt is reduced by 42–58%.
Agent interaction optimizationBased on lightweight checkpoints and state reuse at token boundaries, the TTFT in multi-turn tool call scenarios is reduced by 65-80%.
Flexible memory does not crashWhen background applications preempt video memory, the GPU cache can be hot-expanded or shrunk and the computation can be handed over to the CPU, achieving a smooth degradation with zero downtime and no OOM (Out of Memory) errors.
Full-stack heterogeneous collaborationIt unifies the CPU, system memory, PCIe bus, and GPU into a flexible inference platform, automatically converging to the theoretical maximum throughput of the hardware topology.

FreeToken's project address

Project official website：https://www.flashml.ai/
GitHub repository：https://github.com/FlashML-org/FreeToken
arXiv technical paper：https://arxiv.org/pdf/2608.16157

Comparison of FreeToken and its competitors

Comparison Dimensions	FreeToken	Ollama
position	End-side MoE dedicated full-stack inference system	General Local Large Model Runtime Framework
MoE model support	Natively optimized for full performance, supporting 20+ MoE models.	Some MoE models lack support or cannot be started (such as DSV4-Flash).
Reasoning speed	Faster than Ollama 2–4 times(RTX 4060 achieves 39.3 tok/s in Qwen3.6-35B)	Baseline speed, PCIe bandwidth bottleneck is obvious.
First Token Delay	Reduce TTFT by double buffer prefetch 42–58%	Fetching Experts layer by layer under a long Prompt results in high latency.
Agent multi-turn interaction	State reuse reduces TTFT 65–80%	Context modification necessitates recalculation, resulting in accumulated delays.
Memory flexibility	The system can hot-expand or shrink memory when preempting video memory in the background.0 No downtime, no OOM	Insufficient video memory directly triggers a CUDA OOM crash.
Hardware Collaboration	Unified elastic scheduling of CPU + memory + PCIe + GPU	It mainly relies on the GPU, and the CPU has limited offloading efficiency.
Installation method	GUI App / CLI `uv pip install "freetoken[accel]"`	`ollama run` Command line one-click pull

FreeToken Application Scenarios

Local AI Coding AgentRun programming agents such as Claude Code and OpenClaw locally on laptops or gaming PCs, supporting multi-round tool calls and thought chain iterations, and enabling intelligent code completion and debugging without an internet connection.
End-side ultra-large MoE model inferenceIndividual users can run cutting-edge models such as DeepSeek-V4-Flash (284B) and GLM-5.2 (753B) at full speed on consumer-grade single cards such as RTX 4060/5090, breaking the dependence on data centers.
High-concurrency multi-turn dialogue serviceFor dialogue scenarios such as customer service and Q&A that require frequent context modification, token boundary checkpoints and state reuse can be used to avoid redundant calculations and significantly reduce multi-round response latency.
A flexible AI assistant that can handle both gaming and work.When running 3D rendering, games, or development tools in the background, FreeToken can automatically shrink the GPU cache and transfer the computation to the CPU, ensuring that the AI inference service is uninterrupted and does not crash.
Edge device research and prototype verificationResearchers can quickly deploy and test the latest open-source MoE models on ordinary workstations in the laboratory without applying for server resources, thus lowering the threshold for large-scale model research.