Qwen3.8-27B
Trace the 64-layer hybrid: recurrent Gated DeltaNet state alongside full-attention KV caches.
Jump to a section
Sixty-four layers, sixteen growing caches
Qwen3.8-27B is a dense vision-language model, and only 16 of its 64 decoder layers keep a key/value cache that grows with the context. The other 48 carry a recurrent state instead.
"Dense" here describes the feed-forward layers: there is no expert routing, so every token uses every feed-forward weight. It says nothing about sequence mixing. Reading "dense" as "ordinary full attention everywhere" overestimates the cache fourfold, and reading the recurrent layers as "linear attention throughout" drops the cache that remains.
Three recurrent layers, then one attention layer
The official model card lays out the decoder as 16 repetitions of one four-block group: three Gated DeltaNet blocks followed by one gated-attention block, each paired with a feed-forward network. Every fourth block is full attention.
The two mixers keep different kinds of history. A Gated DeltaNet block summarises earlier tokens in a state that it updates as each token arrives, and the state does not grow with the context. An attention block keeps a key and a value for every earlier position, so a later query can address any of them directly. A D block is not attention with a smaller window; it is a different mechanism.
The hybrid does not replace every growing cache with a fixed-size state. The A blocks keep one, which is why the decoder as a whole cannot run in constant memory as the context grows.
The full-attention blocks have 24 query heads, 4 KV heads and a head dimension of 256. Query heads share KV heads in groups of six, which is grouped-query attention, and it is the KV head count that sets the cache size.
Sizing the attention cache
For the full-attention layers alone, the unquantized key and value tensors take
bytes, where is the batch size, the number of cached tokens and the bytes per element. The 2 counts the key and the value.
With 16 attention layers, 4 KV heads, a head dimension of 256 and two-byte elements, each token costs bytes, or 64 KiB. At batch 1 and 32,768 tokens that is exactly 2 GiB. For comparison, Llama 3 8B, which uses full attention in all 32 layers, needs 128 KiB per token: twice as much for a model about a third the size (see the KV cache).
This is a derived tensor size, not measured GPU memory. It leaves out the recurrent state, the weights, activations, allocator overhead and any implementation-specific cache layout.
1layers = ["delta", "delta", "delta", "attention"] * 16
2assert len(layers) == 64
3assert layers.count("attention") == 16
4
5def attention_cache_bytes(tokens, batch=1, element_bytes=2):
6 return layers.count("attention") * batch * tokens * 2 * 4 * 256 * element_bytes
7
8assert attention_cache_bytes(1) == 64 * 1024
9assert attention_cache_bytes(32768) == 2 * 1024**3
10assert attention_cache_bytes(65536) == 2 * attention_cache_bytes(32768)
11print(attention_cache_bytes(32768) / 1024**3, "GiB for full-attention K/V only")The code is architecture accounting. It does not run Qwen, simulate Gated DeltaNet or claim any throughput gain. Check the exact checkpoint configuration against the model card before adapting it to a deployment.
Why decoding still slows down with context
Suppose decoding gets slower as the context grows, even though three quarters of the layers are recurrent. That is what this architecture predicts. Each of the 16 attention layers reads its entire cached prefix for every new token, so per-token attention work grows linearly with context in those layers, and other runtime costs grow too.
To find what dominates, measure prefill and decoding separately, record the batch size and context length, and account for both kinds of memory. A single aggregate tokens-per-second number cannot tell you which component is responsible.
Where cache estimates go wrong
Query heads instead of KV heads. A calculator that reports 12 GiB for this model's full-attention cache at batch 1, 32,768 tokens and two-byte elements is off by exactly six, the ratio of 24 query heads to 4 KV heads. That is the likely substitution, but a ratio is a hypothesis, not proof: confirm it in the code.
Every layer as attention. Counting all 64 layers with the attention shape gives 8 GiB at 32,768 tokens, four times the real figure. Check that a calculator counts only the 16 full-attention layers and does not silently include other buffers.
No attention at all. Calling the whole model linear attention and dropping the cache gives zero, which underestimates it just as badly in the other direction.
Sources and further reading
- Official Qwen3.8-27B model card, Model Overview, reviewed October 14, 2026. This is a model-card breakdown; there is no separate Qwen3.8 technical report behind it. The layout and head counts are sourced; the storage calculation and failure cases are authored examples, not measured inference results.
- Yang et al., Gated Delta Networks (2024): the recurrent layer type used in three of every four blocks.
- Ainslie et al., GQA (2023): grouped key and value heads.