LMCache
LMCache is an open-source KV cache management layer for LLM inference that stores, compresses, searches, and reuses KV cache across GPU, CPU, local storage, and external backends to reduce repeated prefill and improve time-to-first-token.
Visit LMCache/LMCacheOverview
LMCache is a vendor-neutral KV cache management layer for LLM inference. It turns KV cache into reusable AI-native memory that can be stored, compressed, searched, transferred, and observed across GPU memory, CPU memory, local storage, and external backends. LMCache is designed for workloads where long prompts, conversation history, and retrieved content are reused across requests.
Key Features
- Persistent, tiered KV cache offloading and reuse across requests, sessions, and engine instances.
- Engine-independent deployment, including a standalone multiprocess server mode that isolates KV cache management from the inference engine.
- Pluggable storage and transport backends including CPU RAM, local disk, Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS.
- Non-prefix KV reuse with CacheBlend, which can reuse cached KV blocks at any position in the prompt and selectively recompute tokens for quality recovery.
- KV cache transfer for prefill-decode disaggregation over NVLink, RDMA, or TCP through transport layers such as NIXL.
- Production-level KV cache observability metrics for health, performance, cache hits, lifecycle, and usage.
Use Cases
- Long-context agentic workloads and multi-turn conversations where context is repeatedly reused.
- Knowledge-augmented workloads such as RAG that reuse retrieved content across requests.
- Distributed inference deployments that need to move KV cache across workers or between prefill and decode stages.
- Teams that want to switch serving engines or storage backends while reusing stored KV caches.
Getting Started
- Install LMCache with pip: pip install lmcache.
- For setup options and examples, consult the installation, quickstart, recipes, CLI reference, benchmarking guide, production deployment, and FAQ documentation.
Deployment & Requirements
- No architectural changes are required; choose the deployment mode that matches your setup.
Before You Adopt
- License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
- LMCache is an independent open-source project.
- LMCache grew out of systems research at the University of Chicago and continues to evolve as an open-source project.
