kimi-k3-in-c

Overview

kimi-k3-in-c is a portable C99 inference engine for the Kimi K3 model. It keeps a small working set in memory and streams the dense trunk and routed experts from disk, allowing a 2.78-trillion-parameter model to run on one CPU with measured peak memory of 8.24 GB.

Key Features

• Runs Kimi K3, a 2.78-trillion-parameter mixture-of-experts model, on a single CPU without a GPU.
• Streams model data from disk instead of holding the full 1.56 TB checkpoint in memory, with a measured peak RSS of 8.24 GB.
• Uses portable C99 with no BLAS and no framework, and supports Linux, macOS, and Windows builds.
• Multiplies routed experts directly from packed 4-bit form, which ships at 0.53125 bytes per weight.
• Uses KDA recurrent state and MLA shared latent caching to reduce attention memory.
• Includes test gates and cache simulation tools for verifying engine output and expert cache capacity.

Use Cases

• Running Kimi K3 inference on consumer laptops or desktops with limited RAM.
• Experimenting with large mixture-of-experts inference and disk-streaming techniques on a single CPU.
• Validating an inference engine against a PyTorch reference using the repository test suite.
• Measuring memory use, token speed, and expert cache behavior across RAM budgets.

Getting Started

• Clone the repository, run make -j, and run make test to verify the engine without downloading a checkpoint.
• Run scripts/k3-doctor.sh to check the toolchain, size RAM to a preset, and measure storage.
• Set HF_TOKEN and run scripts/download-model.sh to fetch and verify the 1.56 TB checkpoint.
• Run scripts/pack-trunk.sh to pack the 93 dense layers into a trunk file.
• Run bin/k3 with the model directory, trunk path, tokenizer, preset, prompt, and generation options.

Deployment & Requirements

• Storage: about 1.7 TB free for the 1.56 TB checkpoint plus 109 GB packed trunk, ideally on fast local disk.
• RAM: 8 GB and up; every preset works, and more memory is faster but never changes output.
• OS: Linux x86-64 is the reference; macOS/arm64 and Windows/x86-64 also build and pass every gate.
• CPU: AVX2 and FMA on x86-64, or NEON on arm64; AVX-512 is unnecessary.
• Toolchain: GCC 9 or newer, or Clang 10 or newer, with GNU make or CMake, and Python 3.9 or newer for download, pack, and analysis tools.

Before You Adopt

• License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
• Deployment is gated by storage: the checkpoint and packed trunk require roughly 1.7 TB free, ideally on a fast local disk.
• A partial checkpoint download can produce wrong tokens, so per-shard verification should be treated as a hard stop on failure.