DeepSeek open-sources DeepInfer, the inference engine behind its production service

Chinese AI lab DeepSeek on 20 October 2026 open-sourced DeepInfer, the inference framework that it says powers its large-scale production services. The code is on GitHub under the Apache 2.0 licence.

What is in the release

• For now DeepInfer supports a single model, DeepSeek-V4.1-Flash; the default checkpoint is DeepSeek-V4.1-Flash-1017.
• It runs on Nvidia SM100 data-centre GPUs and Huawei Ascend 950 NPUs.
• Features listed on GitHub include optimised kernels, several kinds of parallelism (including mixture-of-experts expert parallelism), speculative decoding, asynchronous batch scheduling, deterministic sampling and structured outputs.
• The release covers offline inference. The repository says a complete service still needs pieces that are not included: a KV-cache system, prefill/decode disaggregation and load balancing across data-parallel replicas.
• According to The Next Web, DeepInfer was originally built mainly on the open-source engine vLLM, and DeepSeek also released deepseek-recipe, a library matching how its API tokenises text.

Why it matters

Inference — running a trained model to answer requests — is a large part of what AI services cost. Bloomberg, as cited by The Next Web, reported that cloud providers and companies running DeepSeek models on their own servers could use the same techniques to cut costs; such methods are usually kept in-house. DeepSeek’s speed figures are its own and have not yet been independently verified.

The release came a day after the US ordered federal civilian agencies to remove DeepSeek products from their systems.