How They Fixed AI's Biggest Local Hardware Problem (with FreeToken)

Free Token Inference Engine Overview
🚀 Free Token is an open-source inference engine designed by researchers from Berkeley and the University of Texas, specifically optimized for running Mixture of Experts (MoE) models on consumer hardware.
🧠 Unlike standard tools that use static layering, it treats model execution as a scheduling problem, utilizing the GPU memory as a dynamic cache for the most frequently used "experts."
⚡ It employs double buffering to hide latency during the prefill phase by streaming model layers over PCIe while the GPU simultaneously processes existing data.

Performance Optimization Strategies
📉 Q-star Policy: When a cache miss occurs, the system dynamically measures bandwidth between your RAM and PCIe connection to decide whether to compute the expert on the CPU or stream it to the GPU, choosing the faster path.
💾 Custom FTW Format: The engine uses a specialized weight format that allows models to load directly from disk into the required memory layout, skipping time-consuming repackaging steps during startup.
🛠️ Live Cache Resizing: Users can adjust the GPU expert cache size on a running server in approximately 1 second without needing a restart.

Comparative Benchmarking & Use Cases
📊 In a test using a 35B parameter model (exceeding 32GB VRAM), Free Token achieved 132 tokens/second, significantly outperforming Ollama, which managed 58 tokens/second by relying on slower CPU-based layer processing.
⚖️ Performance Threshold: Free Token is most effective when models exceed your available VRAM; if a model fits entirely within the GPU memory, Ollama remains faster due to lower engineering overhead.
💻 The tool includes a user-friendly desktop application (currently Windows and Linux only) that profiles your hardware and provides visual feedback on whether a specific model can run on your system.

Key Points & Insights
➡️ Adaptive Scheduling: The primary strength of Free Token is its ability to automatically profile hardware and tailor the streaming strategy to specific machine constraints without manual configuration.
➡️ MoE Specificity: The engine is purpose-built for Mixture of Experts models; it provides no benefit for dense models and introduces unnecessary overhead if the model fits comfortably within local VRAM.
➡️ Efficiency Gains: By utilizing the fact that most requests only trigger a small subset of "hot" experts, the system maintains high performance even when the full model size significantly exceeds the 32GB VRAM of high-end consumer cards like the RTX 5090.

📸 Video summarized with SummaryTube.com on Sep 14, 2026, 03:01 UTC