SGLang: High-performance LLM and multimodal model serving

SGLang is a high-performance serving framework for large language and multimodal models, designed for fast, scalable inference across diverse models and hardware.

Overview

SGLang is a high-performance serving framework for large language and multimodal models. It provides production-grade inference for large-scale deployments, from a single GPU to distributed clusters, and supports a wide range of open models and hardware platforms.

Key Features

Production-grade inference built for large-scale deployments with reliable, low-latency, high-throughput serving from a single GPU to distributed clusters.
Model and hardware flexibility across a wide range of open models, including LLMs and diffusion models, and diverse hardware platforms.
Advanced optimizations including disaggregated prefill/decode, speculative decoding, parallelisms, a zero-overhead scheduler, and optimized GPU kernels.
OpenAI-compatible API endpoints for querying served models.
Support for NVIDIA GPUs, AMD GPUs, CPU servers, TPU, Ascend NPUs, and XPU.

Use Cases

Large-scale production inference for large language and multimodal models.
Serving open models across single-GPU and distributed-cluster deployments.
Running diffusion models and other supported open models on diverse hardware.
Agentic and RL-rollout workloads that can use session-aware caching.

Getting Started

Install via pip or Docker.
Launch the server with a single command pointing to your model.
Query the API using standard OpenAI-compatible endpoints.

Deployment & Requirements

Supported hardware includes NVIDIA GPUs, AMD GPUs, CPU servers, TPU, Ascend NPUs, and XPU.
Installation options include Python via uv, Python, or Docker.
The installer offers Stable or Nightly builds, CUDA, ROCm, XPU, or CPU platforms, and CUDA 12.9 or CUDA 13.0 versions.

Before You Adopt

License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
Beam search works alongside regular requests but does not yet support mixing with speculative decoding, disaggregation, or DP.
Prefill context parallelism v1 has been removed, and prefill CP is rejected on HIP, NPU, and MUSA until those platforms are ported.
Spec V1 is deprecated, with EAGLE and MTP now running on the unified V2 worker.

SGLang