DeepSeek-V4

DeepSeek-V4's technical report describes two open-weight mixture-of-experts models: Flash has 284B total parameters with 13B activated; Pro has 1.6T total with 49B activated. Both target million-token contexts.

Two different economies are at work: routing reduces the parameters used per token, while compressed attention addresses long-context costs. Neither number alone tells you how much memory a deployment needs.

Start with the distinction between a library and the books opened for one query. A token uses a subset of the experts, but the serving system must still make the other expert weights available for other tokens. Different tokens can select different experts.

The attention architecture mixes Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). CSA selects from compressed entries; HCA uses a more heavily compressed representation. The report also specifies an additional sliding-window branch. The diagram separates these layer types rather than drawing them as two mandatory sequential steps within every layer.

These are complementary choices. Compression changes what an entry represents; selection changes which entries a query reads. Neither should be confused with expert routing in a feed-forward block.

Using the reported rounded counts, Pro's activated fraction is $49/1600\approx3.06\%$. Flash's is $13/284\approx4.58\%$. These ratios are not latency ratios, nor are they the exact fraction of experts selected: embeddings and other shared components also affect parameter counts.

For a hypothetical compressed store that represents each group of $c$ tokens with one entry, a prefix of $T$ tokens has approximately $\lceil T/c\rceil$ entries. Selecting $k$ of those limits the number read by that branch. The cost of constructing and selecting entries, their dimensions, and a local window must be counted separately.

The following is a toy storage ledger, not DeepSeek's learned compressor or indexer. Its chosen compression ratio and top-k are illustrative.

1from math import ceil
2
3def ledger(tokens, compression=8, selected=16, window=128):
4    entries = ceil(tokens / compression)
5    return {
6        "compressed_entries": entries,
7        "selected_entries": min(entries, selected),
8        "local_tokens": min(tokens, window),
9    }
10
11assert ledger(1024) == {
12    "compressed_entries": 128,
13    "selected_entries": 16,
14    "local_tokens": 128,
15}
16assert ledger(2048)["compressed_entries"] == 256
17assert ledger(2048)["selected_entries"] == 16

The selected count can stay fixed while the store grows. That is the point of this example: a bounded read is not the same as bounded storage. It does not model how incomplete compression groups or attention sinks are handled.

Consider a serving incident where resident expert weights exceed device memory even though the active parameter count seems small. Loading only one token's selected experts does not automatically solve the problem: the next token may need different experts, and transfers have a cost. Batching changes the union of experts requested.

Separately, a compressed cache can save space yet lose information relevant to a particular task. Evaluate answer quality under the actual context distribution; a nominal context limit is not a guarantee that every long-range dependency will be used correctly.

A report says an attention implementation reads only a fixed number of compressed entries. A teammate concludes that doubling context cannot increase memory usage. Identify the missing terms.

The compressed store itself may grow, even if the selected subset does not. The local window, temporary buffers and compression state also need their own accounting. To make a latency claim, add indexer work, kernel behavior and memory movement. The toy ledger demonstrates the storage/read distinction; it does not predict the real model's speed.

Sources and scope

DeepSeek-V4 technical report, §2.1 and §2.3 and official model repository. The parameter counts and mechanism names are sourced. The ledger, ratios and debugging scenarios are teaching examples, not a benchmark reproduction or a full architecture implementation.