vLLM

Overview

vLLM is an open-source library and serving engine for LLM inference, originally developed in the Sky Computing Lab at UC Berkeley. It provides state-of-the-art serving throughput and efficient attention key and value memory management with PagedAttention.

Key Features

• State-of-the-art serving throughput with continuous batching, chunked prefill, and prefix caching.
• Efficient attention key and value memory management using PagedAttention.
• Quantization support including FP8, INT8, INT4, GPTQ/AWQ, GGUF, and more.
• OpenAI-compatible API server, plus Anthropic Messages API and gRPC support.
• Tensor, pipeline, data, expert, and context parallelism for distributed inference.
• Supports 200+ model architectures on Hugging Face and diverse hardware platforms.

Use Cases

• Serving open-source LLMs behind an OpenAI-compatible API.
• High-throughput production inference with optimized GPU utilization.
• Running quantized models to reduce inference costs.
• Deploying models across NVIDIA, AMD, Intel, CPU, and other hardware environments.

Getting Started

• Install vLLM with uv: uv pip install vllm
• Optionally run vllm download-kernels for precompiled kernels for faster startup.
• Alternatively build from source for development.

Deployment & Requirements

• Requires Python 3.10+; Python 3.12+ recommended.
• uv is recommended for faster and more reliable installation.
• Supports NVIDIA CUDA, AMD ROCm, Intel XPU, and CPU platforms, among others.

Before You Adopt

• License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
• vLLM requires a Python runtime of at least version 3.10, with Python 3.12+ recommended.