openTPU

Overview

openTPU is an open-source AI accelerator developed by AI. Its monorepo includes the SystemVerilog hardware design, instruction set, bit-exact simulator, kernel language and compiler, host software, and profiler. The design runs modern models on an Inspur YPCB-00338 card with a Xilinx Kintex-7 xc7k480t and two DDR3 channels, and the card produces tokens that match the simulator bit for bit.

Key Features

• One monorepo contains RTL, ISA, simulator, compiler, profiler, and host software.
• Runs ten modern models with real weights on an Inspur YPCB-00338 Kintex-7 PCIe card, matching the simulator token for token.
• Supports int8 and 4-bit weight formats, with an int8 LM head for accuracy.
• Provides otpu-chat, otpu-serve, otpu-smi, otpu-lens, otpu-selftest, and otpu-diag tools.
• Includes an ASIC flow using open-source tools and NanGate45, from SystemVerilog to routed layout.
• Streams experts for mixture-of-experts models larger than the card's 4 GiB memory.

Use Cases

• Learning how an AI accelerator works end to end, from Python matmul kernels to RTL and PCIe card bring-up.
• Running supported language models locally on a Kintex-7 FPGA card or in the ISA simulator.
• Validating device outputs against Hugging Face transformers with quantization-aware golden checks.
• Serving many clients through an OpenAI-style API with contexts kept on the card and host memory or disk.

Getting Started

• Install the Python package with pip install -e . and install pytest, torch, and transformers.
• Run python3 -m pytest -q; RTL tests also need Verilator 5.
• Download a supported checkpoint, for example hf download LiquidAI/LFM2.5-230M --local-dir models/LFM2.5-230M.
• Chat on the simulator with otpu-chat --model lfm2 --backend isa.
• With a card, build the bitstream with make bit in boards/ypcb-00338, load it over JTAG, run sudo otpu-setup, then use otpu-chat --backend board.

Deployment & Requirements

• Everything except the card runs on a laptop.
• RTL tests need Verilator 5.
• Board use requires an Inspur YPCB-00338 card with Xilinx Kintex-7 xc7k480t and two DDR3 channels, plus a Vivado bitstream loaded over JTAG.
• Mixture-of-experts models larger than the card's 4 GiB require streaming experts from host storage.
• Gemma 4 E2B validation can peak at 16 GB and needs a host with about 22 GB free.

Before You Adopt

• License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
• RTL testing has an external tool dependency on Verilator 5.
• Using the FPGA board requires building and loading a bitstream.
• Mixture-of-experts models larger than the card's 4 GiB depend on streaming experts from host storage.
• Gemma 4 E2B validation has substantial host memory requirements.
• The production clock has a narrow timing margin.