TrnRT
An open inference engine for large language models on AWS Trainium, serving an OpenAI-compatible chat and completions API from one trn2.48xlarge with NKI megakernels and a prefix cache.
Visit RunanywhereAI/trnrtOverview
TrnRT is an open inference engine for large language models on AWS Trainium. It serves an OpenAI-compatible API from a single trn2.48xlarge and uses NKI megakernels, a prefix cache, and an offline compile pipeline.
Key Features
- NKI megakernels run decode as roughly one fused kernel per layer half, bundling KDA attention, the fused DSA layer, mHC, and the MoE FFN.
- 1M-token context is supported through sparse attention with top 2,048 and linear attention with fixed-size state.
- A prefix cache is built as a radix tree over FP8 KV pages, with a host-memory tier for evicted pages and state checkpoints.
- An OpenAI-compatible API covers chat and completions, streaming, tool calls, and reasoning output.
- Each box runs two engines, each tensor-parallel over 32 logical NeuronCores, behind a session-affine router.
- Offline compile builds every graph on a CPU host; the serving box loads them and compiles nothing by default.
Use Cases
- Serving GLM-5.3-Flash on AWS Trainium2 through an OpenAI-compatible API.
- Long-context inference workloads that need a 1M-token context and prefix caching.
- Replaying coding-agent sessions and other high-throughput inference workloads on a single trn2.48xlarge.
Getting Started
- Clone the repository and build the Docker image with Neuron SDK 2.32 and Python 3.13.
- Build the compile cache on any x86_64 host; the source notes this takes hours.
- Copy the cache and NKI files to the trn2.48xlarge, download GLM-5.3-Flash, and run the serving container on port 8000.
- Call the server with any OpenAI client pointed at the local OpenAI-compatible endpoint.
Deployment & Requirements
- Docker is required.
- Serving requires an AWS Trainium2 trn2.48xlarge instance.
- Compilation can run on any x86_64 host and does not need a NeuronCore.
- The stack targets Python 3.13 and Neuron SDK 2.32 (neuronx-cc 2.27).
Before You Adopt
- License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
- The project is described as early, with one supported model and a truncated quickstart.
- Building the offline compile cache takes hours on an x86_64 host.
- TrnRT targets AWS Trainium2 hardware and is built against Neuron SDK 2.32.