DeepSeek V4: How They Crushed the Million-Token Memory Problem

The problem nobody talks about

Everyone celebrates the headline number: one million tokens of context. But there's a cost hiding inside that number that makes most implementations impractical.

The model has to remember every token it has ever seen. That memory — the running record of attention computations across every past token — is called the KV cache. It grows linearly with context length. At a million tokens, it becomes the bottleneck. Not compute. Not model size. Memory.

DeepSeek V4 is the first architecture to solve this in a way that feels like it was designed from the ground up, not bolted on.

(DeepSeek-V4-Pro: 1.6T total parameters, 49B activated, 1M token context. DeepSeek-V4-Flash: 284B total parameters, 13B activated, 1M token context. Both released April 2026.)

Two kinds of memory instead of one

Imagine you're reading a million-word book and someone asks you questions about it. Remembering every single sentence verbatim would break most people's brains. But good readers don't do that — they keep two kinds of memory running at once.

Memory type 1 — the "highlights" memory. Every few pages, you jot down a sticky note. You're compressing, but you're keeping enough detail that you can zoom back in when needed. When someone asks a specific question, you scan the sticky notes and open the right one.

(This is Compressed Sparse Attention, or CSA. Every 4 tokens in the sequence get compressed into 1 KV entry via softmax-gated pooling. When processing a new query, a "lightning indexer" — running in FP4 with ReLU-scored multi-head dot products — selects only the top-k most relevant compressed blocks. A separate sliding-window branch keeps the most recent tokens fully uncompressed for recency.)

Memory type 2 — the "vibe" memory. Every hundred-or-so pages, you compress down to a single impression. Not details — just the feeling, the theme, the shape of that section. You can hold the whole book in your head this way without drowning in specifics.

(This is Heavily Compressed Attention, or HCA. Every 128 tokens compress into 1 KV entry. Because the compression is so aggressive, the resulting sequence is tiny enough that the model can run full dense attention over all of it — no sparse selection needed. Same sliding-window branch handles recency.)

The 61-layer model alternates between these two across its depth. Layers 0–1 run HCA only. Layers 2–60 alternate CSA and HCA. The final MTP block runs sliding-window only.

The result: at a million-token context, V4-Pro needs only 27% of V3.2's single-token inference FLOPs and 10% of its KV cache. Compared to a standard BF16 GQA-8 baseline, the KV cache drops to roughly 2%. That's not a rounding improvement. That's a different class of system.

(The compound savings come from two multiplied factors: compression ratio (4x or 128x depending on layer type) × storage quantization (FP8 for most KV entries, FP4 for the indexer weights, BF16 only for RoPE dimensions which need precision). They multiply, not add.)

The skip connection problem nobody warned you about

There's a second problem that gets less attention: how do you keep signal stable when you're passing information through 61 layers?

Picture passing a ball down a line of 61 kids. Without rules, two things can go wrong. Each kid makes the ball slightly louder — by kid 61, it's a scream. Or each kid makes it slightly quieter — by kid 61, it's silence. Both are catastrophic. The model stops learning.

The standard fix since ResNet has been the residual connection: pass the original ball forward unchanged alongside whatever the kid does with it. That way even if one kid garbles it, the original still arrives. (x → x + f(x). The skip connection ensures gradient flows back directly through the identity path.)

DeepSeek V4 wanted more. They wanted the connection itself to be learnable — different layers, different mixing weights, more expressive information flow. But the moment you make the weights learnable with no constraints, you're back to the screaming kid problem.

Their fix: make the weights always look like a fair redistribution. No amplification allowed.

Here's the simplest way to see it. Imagine you have buckets of water. You're allowed to pour between buckets however you like, but two rules must hold: 1. Every bucket you pour from ends up completely empty — you used all of it. 2. Every bucket you pour into ends up exactly full — no overflow, no shortage.

You can redistribute however you want. But you cannot create water or lose water.

(This is a doubly stochastic matrix: every row sums to 1, every column sums to 1. The Birkhoff polytope is the mathematical "shape" containing all valid doubly stochastic matrices — i.e., all valid redistribution patterns.)

The algorithm that enforces this during training is beautifully simple: take your weight matrix, normalize each row so it sums to 1, then normalize each column so it sums to 1, repeat a handful of times. It converges fast.

(This is the Sinkhorn-Knopp algorithm. Each round of alternating row-then-column normalization moves the matrix closer to the Birkhoff polytope. ~5–10 iterations is enough in practice.)

Why does "no water created or lost" prevent explosion? Because a transformation that only redistributes — never amplifies — can stretch any signal by at most a factor of 1. Not 1.1. Exactly 1. Multiply sixty-one of those matrices together and you never blow up, no matter how deep.

(Spectral norm of a doubly stochastic matrix is bounded to ≤ 1. Spectral norm is the maximum factor by which a matrix can stretch any input vector. Bounding it to 1 eliminates explosion in both the forward pass and the backward gradient pass.)

This is what DeepSeek calls Manifold-Constrained Hyper-Connections (mHC). Learnable skip connections, locked inside a shape where the only legal move is "redistribute, don't amplify," enforced cheaply at every training step via Sinkhorn-Knopp.

The agentic thinking trick that matters most

Here's the one that's easy to miss but is the most practically significant for anyone building agents.

A normal model forgets its reasoning between your messages. You ask it something, it thinks, it answers, the thinking is gone. You ask again, it starts fresh. For a chatbot, that's fine.

For an agent running a multi-step job — calling tools, getting results, planning the next step — that's a serious problem. The model has to re-derive its reasoning from scratch every turn, even when nothing has changed about the overall plan.

V4 keeps its scratchpad alive across user messages — but only when tool calls are present in the conversation. In pure conversation mode, the thinking is still flushed for brevity.

(When tool calls are present, <think> blocks are preserved across user message boundaries and appended to the context as the conversation continues. In standard conversational turns, they're discarded. This is an explicit architectural decision in the post-training protocol, not a prompt engineering trick.)

A real assistant doing a multi-step job shouldn't have to re-figure out what it was doing every time it reports back to you. This is V4 formalizing that at the model level.

The tool call format changed too — away from JSON-in-string (which breaks on nested quotes) to a typed XML schema with an explicit string flag separating string parameters from structured JSON parameters. Less fragile under the kinds of nested-quote situations that regularly break agent pipelines.

How they trained the reasoning at scale

Teaching a model to use tools correctly requires actually running the tools. That means running hundreds of thousands of sandboxed environments simultaneously during RL training — each one a little world where the model tries something and gets graded on the result.

DeepSeek built the infrastructure for this from scratch, in Rust, and called it DSec (DeepSeek Elastic Compute). It runs across four substrate types behind a single Python SDK: plain functions, containers, Firecracker microVMs, and full QEMU virtual machines. You pick the isolation level based on what the task requires; the API doesn't change.

The detail that makes this work at RL scale: if training gets interrupted mid-rollout, the system replays the trajectory safely rather than re-executing tool calls from scratch. That matters because re-running a tool call mid-sequence can corrupt the training signal — the result at step 3 shouldn't change just because training resumed from a checkpoint.

(Preemption-safe trajectory replay via 3FS layered storage. Fast container loading means no startup delay between RL rollout episodes. The system ran at hundreds of thousands of concurrent sandboxes during V4 training.)

What the benchmarks actually say

V4-Pro-Max competitive results at release: -

1. SWE-bench Verified: 80.6% (on par with Opus-4.6-Max at 80.8%)
2. LiveCodeBench: 93.5% - Codeforces Rating: 3206 — among the highest reported
3. 1M-token MRCR (multi-needle retrieval): 83.5 mean MRR, outperforming Gemini-3.1-Pro at 76.3
4. MRCR accuracy at 256K tokens: >0.82; drops to 0.59 at 1M — honest about the degradation curve

The interesting engineering question isn't "can a model process a million tokens." It's "can it do so without the cost making the architecture economically unviable."

V4 answers that question architecturally. The KV cache compression is not a serving trick — it's baked into the attention design. The interleaved thinking is not a prompt template — it's a post-training decision about when reasoning state gets preserved. The DSec infra is not an inference optimization — it's the training system that made the agentic capabilities possible.

The three things most worth understanding in this paper: the dual-path attention mechanism and how compression ratios compound with quantization; the Birkhoff polytope constraint on residual connections and why it buys stability for free; and the interleaved thinking decision and what it means for how you design multi-turn agent turns.

Source: DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (April 2026). Technical report available via huggingface.co/deepseek-ai/DeepSeek-V4-Pro.