nanochat

Overview

nanochat is an experimental harness for training LLMs. It is designed to run on a single GPU node with minimal, hackable code and covers tokenization, pretraining, finetuning, evaluation, and inference. A single depth setting controls model size and automatically derives other hyperparameters so runs remain compute-optimal.

Key Features

• End-to-end LLM pipeline covering tokenization, pretraining, finetuning, evaluation, and inference.
• A single complexity dial, --depth, sets the transformer depth and automatically calculates width, heads, learning-rate adjustments, training horizons, and weight decays.
• The reference speedrun trains a GPT-2 capability model on an 8XH100 GPU node in about 1.5 hours for roughly $48, or closer to $15 on a spot instance.
• A time-to-GPT-2 leaderboard tracks wall-clock training time against the DCLM CORE score, with runs/speedrun.sh as the reference path.
• A CLI chat interface lets users talk to the trained model.
• Precision is managed explicitly through a global COMPUTE_DTYPE and can be overridden with the NANOCHAT_DTYPE environment variable.

Use Cases

• Train and then chat with a GPT-2 capability model on a single 8XH100 GPU node.
• Run short pretraining experiments with a 12-layer GPT-1-sized model to test code changes quickly.
• Study scaling laws and train a miniseries of compute-optimal models by varying depth.
• Run a small CPU or Apple Silicon/MPS example, with a dramatically reduced model and weaker results.

Getting Started

• Install dependencies with uv sync --extra gpu for CUDA or uv sync --extra cpu for CPU-only/MPS, then activate the virtual environment with source .venv/bin/activate.
• On an 8XH100 GPU node, start the reference training pipeline with bash runs/speedrun.sh; the run takes about 1.5 hours.
• After training, talk to the model over the CLI with python -m scripts.chat_cli.
• For development dependencies, use uv sync --extra gpu --group dev.

Deployment & Requirements

• The target setup is a single GPU node; the reference speedrun is designed for an 8XH100 GPU node.
• The code also runs on an 8XA100 GPU node, somewhat slower, and on a single GPU by omitting torchrun, but that takes about eight times longer.
• GPUs with less than 80GB require tuning hyperparameters such as --device-batch-size to avoid OOM or running out of VRAM.
• CPU and Apple Silicon/MPS execution is shown through a small example script, but it does not produce strong results.
• Dependencies are managed with uv, using either the gpu or cpu extra.

Before You Adopt

• License: MIT. Review its terms before using, modifying, or distributing the project.
• GPUs with less than 80GB require hyperparameter tuning to avoid out-of-memory failures.
• CPU or Apple Silicon execution uses a dramatically smaller model and does not produce strong results.
• The original nanochat announcement contains deprecated information and describes an older model with worse results than current master.
• Some alternate PyTorch backends are less tested and may have rough edges.