Qwen3.8-27B

Qwen3.8-27B is a dense vision-language model with a hybrid decoder. “Dense” describes the absence of expert routing in its feed-forward layers; it does not mean that every sequence-mixing layer uses ordinary full attention.

The official model card specifies 64 decoder layers arranged as 16 repetitions of three Gated DeltaNet blocks followed by one gated-attention block, each paired with a feed-forward network. The breakdown below reads that architecture, rather than treating the model's benchmark scores as an explanation.

The two mixer types keep different kinds of information. A recurrent state summarizes earlier tokens through updates. A full-attention cache retains per-position keys and values that a later query can address. A hybrid combines them; it does not replace every growing cache with a fixed-size state.

Read the diagram in order. Every fourth block is full attention. The D blocks are not just “attention with a smaller window”; they use a different mechanism. Conversely, the A blocks prevent you from assuming constant memory for the complete decoder as context grows.

The model card gives the full-attention blocks 24 query heads, 4 KV heads and a head dimension of 256. Query heads share KV heads in groups of six.

For the full-attention subset alone, estimate the unquantized K/V tensor storage as

Here $B$ is batch size, $T$ cached tokens, and $s$ bytes per element. Using 16 full-attention layers, batch 1, 32,768 tokens, 4 KV heads, dimension 256 and two-byte elements gives 2 GiB. This is a derived tensor-size calculation, not measured GPU memory. It excludes recurrent state, weights, activations, allocation overhead and implementation-specific cache layouts.

Using 24 query heads in place of 4 KV heads overestimates this component by six times. Calling the entire model “linear attention” and dropping this component underestimates it instead.

1layers = ["delta", "delta", "delta", "attention"] * 16
2assert len(layers) == 64
3assert layers.count("attention") == 16
4
5def attention_cache_bytes(tokens, batch=1, element_bytes=2):
6    return layers.count("attention") * batch * tokens * 2 * 4 * 256 * element_bytes
7
8assert attention_cache_bytes(32768) == 2 * 1024**3
9assert attention_cache_bytes(65536) == 2 * attention_cache_bytes(32768)
10print(attention_cache_bytes(32768) / 1024**3, "GiB for full-attention K/V only")

This is architecture accounting. It does not run Qwen, simulate Gated DeltaNet, or claim a throughput improvement. Compare the exact checkpoint configuration with the model card before adapting it to a deployment.

Suppose decoding gets slower as context length increases, even though most layers are recurrent. That observation is compatible with this architecture. The full-attention layers still read a growing prefix, and other runtime costs can matter too.

To diagnose it, measure prefill and decoding separately, record batch and context length, and account for both memory types. A single aggregate “tokens per second” number cannot tell you which component dominates.

A cache calculator reports 12 GiB for this model's full-attention K/V at batch 1 and 32,768 tokens with two-byte elements. Which likely substitution caused that sixfold error?

Reference: it probably used the 24 query heads instead of 4 KV heads. That diagnosis is a hypothesis to verify in the code, not proof from the ratio alone. Then check that it counted only the 16 full-attention layers and did not silently include other buffers.

Source and scope

Official Qwen3.8-27B model card, Model Overview, reviewed October 14, 2026. This is a model-card breakdown; we are not presenting a separate Qwen3.8 technical report. The layout and head counts are sourced, while the storage calculation and failure case are authored examples, not measured inference results.