llm-d

Overview

llm-d is an open-source, Kubernetes-native stack for distributed LLM inference. It orchestrates model servers such as vLLM and SGLang to provide production-grade serving across a cluster and supported accelerators.

Key Features

• Intelligent routing with prefix-cache and load-aware balancing, plus experimental predicted-latency scheduling.
• Advanced KV-cache management, including tiered offloading to CPU or disk and global indexing of KV-cache state.
• Serving large models with prefill/decode disaggregation and wide expert parallelism over fast accelerator interconnects.
• Operational features for production, including flow control for multi-tenant serving and SLO-aware autoscaling.
• Batch processing with OpenAI-compatible Batch APIs and asynchronous processing for offline inference.
• Portability across GPUs, TPUs, XPUs, CPUs, and emerging NPUs.

Use Cases

• Deploying an optimized baseline for production LLM serving on Kubernetes.
• Routing requests by predicted latency or precise prefix-cache awareness.
• Scaling large models such as DeepSeek-R1 and GPT-OSS with disaggregation and wide expert parallelism.
• Running large-scale offline inference with OpenAI-compatible Batch APIs.

Getting Started

• Follow the Quickstart Guide to deploy an optimized inference service on Kubernetes.
• Set up the llm-d stack, configure the intelligent router, and validate performance with production-ready benchmarks.
• Most users begin with the Optimized Baseline well-lit path.

Deployment & Requirements

• A Kubernetes cluster.
• Supported accelerators including GPUs, TPUs, XPUs, CPUs, and emerging NPUs.
• Model servers such as vLLM, SGLang, or others as part of the inference stack.

Before You Adopt

• License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
• llm-d is a CNCF Sandbox project, indicating early-stage project maturity within the CNCF.
• Predicted-latency scheduling is described as experimental.
• The batch gateway is labeled experimental in release notes.