Colibri

Overview

Colibri is an open-source inference engine for running large language and other models on hardware you already own. It places model weights across GPU memory, system RAM, and storage, keeping dense parts in memory and streaming routed mixture-of-experts weights from disk when the router requests them. The project includes a dashboard, model catalog, APIs, MCP support, and a CLI.

Key Features

• Hierarchical placement of model weights across GPU memory, system RAM, and storage, with routed experts streamed from disk when needed.
• Mixture-of-experts routing activates only a subset of experts per token, allowing very large models to run on machines where the whole model does not fit in fast memory.
• CPU inference with optional GPU acceleration through CUDA, Vulkan, and Metal across Windows, Linux, and macOS.
• A dashboard for local chat, expert routing, activity, and structured decisions, with real timing and throughput from the engine.
• Compatibility with OpenAI-compatible and Anthropic-compatible APIs, MCP, and a CLI named coli for agent and tool workflows.
• Support for 21 catalogue entries and 13 engines, including language, vision, image-generation, and decision models, plus published GGUF files read as-is.

Use Cases

• Running large open mixture-of-experts language models locally on owned hardware, including models that stream experts from disk.
• Using local models from chat, coding agents, and automation tools through OpenAI-compatible, Anthropic-compatible, MCP, or CLI interfaces.
• Making structured choices with probabilities, such as routing a request, picking an action, or scoring a signal.
• Researching and measuring inference optimizations through the dashboard and reproducible end-to-end benchmarks.

Getting Started

• On Windows, download the project ZIP, unzip it, and run START-HERE.bat.
• On Ubuntu or Debian, install git, python3, and build-essential, clone the Colibri repository, and run ./start-here.sh.
• On macOS with Homebrew, install the Xcode command line tools plus libomp, git, and python, clone the repository, and run ./start-here.sh.
• The setup checks RAM, disk, CPU, and GPUs; recommends a fitting model; builds or downloads the engine; downloads the model with resume; and opens the dashboard.
• On later runs, START-HERE.bat or ./start-here.sh starts Colibri without downloading or building again.

Deployment & Requirements

• At least 8 GB of RAM; 16 GB or more is better.
• 22 GB free disk for the smallest model, plus model-specific download and cache space.
• An internet connection for setup and model download; a graphics card is optional.
• Supported platforms include Windows, Linux, and macOS; RAM, disk, and backend needs depend on the chosen model.
• Setup can use a compiler to build the engine or download a prebuilt engine when no compiler is present.

Before You Adopt

• License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
• Model files carry their own licences, separate from the engine.
• Disk speed is a practical performance factor for large models that stream experts from storage.
• Some supported models require a conversion or preparation step after download before use.