FreeToken

Overview

FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine designed for running frontier-scale open-weight models on personal and consumer hardware. It treats GPUs, CPUs, host memory, and interconnects as a unified, elastic inference platform.

Key Features

• Fast edge-native MoE runtime with bandwidth-adaptive CPU–GPU co-execution, full-layer double-buffered prefill streaming, global LRU expert caching, graph-compatible execution, and the FTW fast weight format.
• Semantic-aware caching with semantic anchor checkpoints for recurrent state and KV caches, helping avoid redundant context recomputation during agentic context edits.
• Elastic memory management that dynamically re-allocates VRAM between expert caches and KV memory at runtime without engine restarts or weight reloading.
• Broad MoE and ecosystem support across parameter scales and quantization formats including MXFP4, NVFP4, FP8, and BF16.
• Anthropic/OpenAI-compatible APIs for integration with coding and tool-calling agents.
• Runs across consumer laptops, gaming desktops, and workstation GPUs, with native support for NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.

Use Cases

• Running frontier-scale open-weight MoE models locally on gaming PCs and consumer desktops.
• Serving models to coding and tool-calling agents through Anthropic/OpenAI-compatible APIs.
• Using the desktop GUI to run models, chat, and tune the engine.
• Experimenting with local model serving across heterogeneous edge resources such as GPUs, CPUs, and host memory.

Getting Started

• Download the Windows or Linux desktop app from the project website; it sets up the engine and provides a GUI for running models, chatting, and tuning the engine.
• Install the CLI with uv using uv pip install "freetoken[accel]" or with pip.
• Build from source by cloning the repository, creating a uv virtual environment, activating it, and running uv pip install -e ".[accel]".

Deployment & Requirements

• Desktop app is available for Windows or Linux.
• CLI installation is supported with uv, recommended, or pip, and the project can be built from source.
• Native GPU support is provided for NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.
• The engine targets consumer laptops, gaming desktops, and workstation GPUs.

Before You Adopt

• License: Apache-2.0. Review its terms before using, modifying, or distributing the project.