DeepInfer

Overview

DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale and supports DeepSeek-V4.1-Flash.

Key Features

• Optimized kernels and model and sampler runtime for efficient inference.
• Parallelism support including Attention DP or balanced CP and MoE EP with EPLB.
• DSpark, confidence-scheduled verification, and SWA bounded replay.
• Asynchronous scheduling that overlaps CPU batch scheduling with device execution.
• Batch-invariant inference and deterministic sampling for repeatable results.
• Structured outputs constrained by regex, JSON schema, or EBNF grammar.

Use Cases

• Running offline inference for DeepSeek-V4.1-Flash.
• Serving DeepSeek models at scale with optimized kernels and parallelism.
• Generating regex-, JSON schema-, or EBNF-constrained structured outputs.
• Deploying inference on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs.

Getting Started

• Create and activate an empty Python 3.12 environment before building DeepInfer.
• Export DEEPINFER_ACCELERATOR=nvidia or ascend as appropriate, then run ./build.sh.
• Run python examples/offline_inference.py.
• The default DeepSeek-V4.1-Flash checkpoint downloads to the Hugging Face cache on first run; set DEEPINFER_HF_MODEL_DIR to run from a local copy instead.

Deployment & Requirements

• Linux with GCC >= 13.0 and Python 3.12.
• NVIDIA SM100 GPUs with CUDA 13, or Huawei Ascend 950 NPUs with CANN 9.2.
• A complete inference service requires a KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas for prefill and decode; these are not included in this release.

Before You Adopt

• License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
• The current release is described as offline inference only for now.
• A complete inference service requires additional components that are not included in this release.