Edge0
Edge0 provides device-native foundation models and an open-source streaming MoE inference framework, running large sparse models such as 145B on a laptop and 35B on a phone with SSD-streamed weights, offline on-device processing, and zero
Visit Edge0-AI/Edge0Overview
Edge0 is an open-source streaming MoE inference framework for running large sparse models on consumer hardware. It provides command-line, local API server, and Edge0 Chat app access, with current 145B and 120B latent-MoE models for a single NVIDIA GPU and a previous 35B release on the native engine.
Key Features
- Runs large sparse mixture-of-experts models on consumer hardware using latent MoE and SSD-streamed expert weights.
- Offers edge0-infer, a GGML-based unified inference engine with a shared C++ API, CLI, OpenAI-compatible local server, and chat app examples for desktop, Android, and iOS.
- Supports on-device inference so sensitive data stays on the hardware without being sent to a cloud model for processing.
- Enables offline operation and low-power, low-memory deployment across phones, laptops, earbuds, glasses, toys, and other devices.
- Includes language and voice models such as Edge0-35B, Edge0-8B, Audio8-ASR-3B, and Audio8-TTS-0.6B.
- Eliminates per-use charges and token costs by running inference on hardware you already ship.
Use Cases
- Smart earbuds: an edge-cloud voice SDK embedded in earbuds for natural commands and live readouts with zero hardware changes.
- Robotics and embodied AI: edge-cloud HRI SDK and robot chips plus edge voice models for natural-language dialogue, command execution, and offline or weak-network voice interaction.
- Smart glasses: edge-cloud coordination keeps frequent commands on-device and complex tasks in the cloud to cut latency and cost.
- Smart cockpit: a fully on-device cockpit for low power and high performance, with voice interaction accuracy of 93% and command response latency under 200 ms, keeping data in the vehicle.
Getting Started
- For 145B and 120B, clone the Edge0 repository, build edge0-strata and edge0-infer using their build guides, and run edge0-infer serve --model /path/to/Edge0-145B-q3k from edge0-infer.
- The serve command prints a local listening address and exposes OpenAI- and Anthropic-compatible endpoints.
- Use edge0-infer chat --model /path/to/Edge0-145B-q3k for an interactive session, or run Edge0 Chat on Windows by choosing a model folder such as Edge0-145B-q3k.
- Pass one model folder, not the folder that contains several models; edge0-infer identifies the model by model_name in model.json.
Deployment & Requirements
- Latest 145B and 120B models run on Linux or Windows with an NVIDIA GPU and CUDA 13; build instructions target x64 or ARM64.
- Platform support depends on model and engine: edge0-35b native engine supports Linux CPU, Windows CPU or Vulkan, macOS Apple Silicon Metal or CPU, iOS Metal, and Android CPU.
- The 8B checkpoint is available, with native-engine support planned.
Before You Adopt
- License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
- Model and platform support varies by engine; check the model-and-engine pairing before deployment.
- The unified edge0-infer engine replaces the earlier Python framework and per-platform engines; older code is kept at a tag.
- The 8B checkpoint is available, but native-engine support is planned rather than current.
- A 16 GiB VRAM plus 20 GiB RAM reading is a test constraint, not a minimum system requirement.
- Apple M2 Ultra Metal measurements do not establish macOS release availability.