Skip to main content
AdOpenFree logoPromote your productReach more potential users and drive product growth and revenue.Advertise
Favicon of TrnRT

TrnRT

Free Listing

An open inference engine for large language models on AWS Trainium, serving an OpenAI-compatible chat and completions API from one trn2.48xlarge with NKI megakernels and a prefix cache.

Visit RunanywhereAI/trnrt

Overview

TrnRT is an open inference engine for large language models on AWS Trainium. It serves an OpenAI-compatible API from a single trn2.48xlarge and uses NKI megakernels, a prefix cache, and an offline compile pipeline.

Key Features

  • NKI megakernels run decode as roughly one fused kernel per layer half, bundling KDA attention, the fused DSA layer, mHC, and the MoE FFN.
  • 1M-token context is supported through sparse attention with top 2,048 and linear attention with fixed-size state.
  • A prefix cache is built as a radix tree over FP8 KV pages, with a host-memory tier for evicted pages and state checkpoints.
  • An OpenAI-compatible API covers chat and completions, streaming, tool calls, and reasoning output.
  • Each box runs two engines, each tensor-parallel over 32 logical NeuronCores, behind a session-affine router.
  • Offline compile builds every graph on a CPU host; the serving box loads them and compiles nothing by default.

Use Cases

  • Serving GLM-5.3-Flash on AWS Trainium2 through an OpenAI-compatible API.
  • Long-context inference workloads that need a 1M-token context and prefix caching.
  • Replaying coding-agent sessions and other high-throughput inference workloads on a single trn2.48xlarge.

Getting Started

  • Clone the repository and build the Docker image with Neuron SDK 2.32 and Python 3.13.
  • Build the compile cache on any x86_64 host; the source notes this takes hours.
  • Copy the cache and NKI files to the trn2.48xlarge, download GLM-5.3-Flash, and run the serving container on port 8000.
  • Call the server with any OpenAI client pointed at the local OpenAI-compatible endpoint.

Deployment & Requirements

  • Docker is required.
  • Serving requires an AWS Trainium2 trn2.48xlarge instance.
  • Compilation can run on any x86_64 host and does not need a NeuronCore.
  • The stack targets Python 3.13 and Neuron SDK 2.32 (neuronx-cc 2.27).

Before You Adopt

  • License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
  • The project is described as early, with one supported model and a truncated quickstart.
  • Building the offline compile cache takes hours on an x86_64 host.
  • TrnRT targets AWS Trainium2 hardware and is built against Neuron SDK 2.32.

Comments

Sign in to leave a comment.

More like TrnRT

Favicon of DwarfStar

DwarfStar

Free ListingStars: 24K

Local inference engine for DeepSeek, GLM, and Qwen

Inference Servers & APIs

DwarfStar is a small native inference engine for running selected DeepSeek, GLM, and Qwen models on consumer hardware with Metal, CUDA, and ROCm support. It uses the project's own GGUF files and is not a general GGUF runner.

Favicon of DeepInfer

DeepInfer

Free ListingStars: 7.9K

High-performance inference framework for DeepSeek models

Inference Servers & APIs

DeepInfer is a high-performance, large-scale inference framework for DeepSeek models. This open-source release provides offline inference with optimized kernels, parallelism, scheduling, deterministic sampling, and structured outputs.

Favicon of vLLM

vLLM

Free ListingStars: 94.2K

Easy, fast, and cost-efficient LLM serving for everyone.

Inference Servers & APIs

vLLM is a high-throughput, memory-efficient inference and serving engine for LLMs. It aims to make LLM serving easy, fast, and cost-efficient for everyone.