← Back to blog
GeneralAbout 6 min read

DeepSeek Open-Sourced Its Serving Stack. Read the Missing Pieces First.

Published Nov 7, 2026
DeepSeek Open-Sourced Its Serving Stack. Read the Missing Pieces First.

DeepSeek published DeepInfer this week: the inference framework that runs its own large-scale production traffic, now available under Apache 2.0. It has climbed past 25,000 stars in a month without a launch event, which tells you how starved people are for a look at how a frontier lab actually serves a model.

Before anyone benchmarks it against vLLM on a Sunday afternoon, read the section of the README that lists what is not in the release. It is the most honest paragraph in the repo.

What you get

DeepInfer is an offline inference engine for exactly one model: DeepSeek-V4.1-Flash. Not a family, not a plug-in architecture with a dozen model adapters. One checkpoint, tuned until the kernels stop complaining.

That narrowness is the point. The feature list is a tour of everything that costs money at scale:

  • Optimised kernels — FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, plus custom TileLang kernels.
  • Parallelism — attention data parallelism with balanced context parallelism, MoE expert parallelism with EPLB (expert-level load balancing, the thing that stops one expert from becoming a hotspot).
  • DSpark and confidence-scheduled verification — speculative decoding where the draft length is gated by confidence rather than a fixed number. Draft length defaults to 5 via DEEPINFER_USE_NEXTN_DECODE.
  • SWA bounded replay — instead of replaying an entire prefix, replay only a bounded suffix. The default V4.1 settings keep 140 tokens and require the suffix to cover the sliding window plus the maximum backtracking distance.
  • Asynchronous scheduling — batch scheduling on CPU overlapping device execution, on by default.
  • Batch-invariant inference and deterministic sampling — the same input tokens produce the same output tokens regardless of what else is in the batch.
  • Structured outputs — regex, JSON schema, or EBNF grammar constraints passed through SamplingParams.

That last cluster is the one I'd underline. Almost every serving stack can do structured output. Very few can promise that your output is byte-identical whether you sent one request or a thousand, because most kernels pick different reduction orders depending on batch shape, and different reduction orders produce different floating point results. If you have ever had a regression test fail because the batch got bigger, you know why this matters.

A macro view of a GPU board edge with copper traces and a liquid cooling fitting

What you don't get

Three things, spelled out in the README without much drama:

  • A KV cache system.
  • VE / prefill / decode disaggregation — "most required engine functionality is already implemented," but the deployment plumbing is not shipped.
  • Load balancing across data-parallel replicas, for prefill and decode separately.
  • Anything other than DeepSeek-V4.1-Flash.

Read that list twice. A production inference service is not a fast kernel; it is a fast kernel plus a cache that survives eviction plus a router that knows which replica is cold. DeepInfer gives you the engine, not the car. If you are a research team with eight GPUs and a benchmark to reproduce, this is exactly right. If you are a startup hoping to swap it in for your vLLM deployment next sprint, you are going to spend a quarter rebuilding the parts that were deliberately left out.

Hardware is the real story

Supported hardware: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Two lines, no commentary.

That is a remarkable thing to put in a README. A framework that treats a top-end NVIDIA part and a domestic Chinese accelerator as peer targets is telling you where its production traffic actually runs, or expects to. The build system makes it explicit: export DEEPINFER_ACCELERATOR=nvidia, or ascend instead. CUDA 13 on one path, CANN 9.2 on the other.

For most Western teams this is trivia. For anyone building in China, or building something that has to survive an export-control headline, it is the most consequential line in the repo.

Installing it, honestly

The prerequisites are not gentle: Linux, GCC 13 or newer, Python 3.12, CUDA 13. Then:

export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py

The default checkpoint pulls from Hugging Face on first run; set DEEPINFER_HF_MODEL_DIR if you have it locally. Benchmarks were measured on four SM100 GPUs for prefill and sixty-four for decode, which should calibrate your expectations about what "fast" means here. You will not reproduce those numbers on a 4090, and there is no consumer path at all.

The lineage is stated plainly: originally built on vLLM, with ideas from SGLang and TensorRT-LLM. That's a graceful acknowledgement, and also a useful map. If you know vLLM's scheduler and SGLang's radix cache, you can guess which parts DeepInfer rewrote and which parts it inherited.

Why labs open-source this

Three plausible reasons, and they are not mutually exclusive.

First, credibility. Publishing the serving stack is a way of saying the benchmark numbers came from something real. Anyone can quote tokens per second. Shipping the code that produced them is a different level of claim.

Second, ecosystem. Serving frameworks are where inference engineers live. If your model runs well on the framework everybody already knows, you get adoption for free. If it only runs well on your framework, the framework has to earn its own users.

Third, hiring and standards. A lab that publishes its kernels sets the vocabulary. When the next paper says "batch-invariant," everyone means what DeepInfer means.

What it does not do is hand you a product. The gap between "we open-sourced our engine" and "you can serve this model cheaply" is exactly the gap the README is honest about.

What to do with it this month

If you run inference for a living, clone it, run the offline example on whatever Blackwell or Ascend hardware you can rent, and measure two things: time to first token under a realistic prompt distribution, and whether deterministic sampling actually holds across batch sizes. Those are the two claims that are cheap to verify and expensive to fake.

There is also a narrower, more practical use: offline batch scoring. If you need to run a fixed evaluation suite ten times and compare the outputs, deterministic sampling turns that from a statistical exercise into a diff. Most evaluation harnesses spend half their complexity working around the fact that the same prompt produces different answers at different batch sizes. A serving stack that guarantees otherwise removes a whole category of tooling.

If you don't, the repo is still worth an hour. Not as software, as a spec. It is a frontier lab telling you which problems it considered hard enough to solve in-house: expert load balance, bounded replay for sliding-window attention, confidence-gated speculation, and bit-exact reproducibility. That list is a decent map of where inference costs actually come from, and it will be true long after V4.1-Flash is retired.

Open source, in this case, means "here is the engine, here is the tuning, here is the one model we care about." The rest — cache, router, the unglamorous middle — remains the part you build.

Related articles