Qwen3.5-0.8B

Qwen3.5-0.8B is a small vision-language model whose text stack mixes recurrent and full attention. Its official configuration lists 24 layers: three Gated DeltaNet layers followed by one gated full-attention layer, repeated six times. Its feed-forward layers are dense; do not apply the Qwen family's general MoE description to this checkpoint.

The repeated pattern gives the model two kinds of history. Recurrent layers update a fixed-size state. Full-attention layers keep token-indexed keys and values. The diagram shows the actual layer order from the checkpoint configuration, not measured activations.

This is a useful example of architectural efficiency at small scale. Read it alongside linear attention and KV caching to understand which storage grows with the prefix.

The full-attention configuration has two KV heads of dimension 256. For a batch of one, six full-attention layers and two-byte cache elements:

$\text{KV bytes} = 2\times6\times T\times2\times256\times2 = 12{,}288T$

At 4,096 tokens that is 48 MiB for those K/V tensors. Recurrent states, convolution buffers, vision features, model weights, padding and allocator overhead are excluded. This is a tensor-size calculation, not a measured serving footprint.

layer_types = ['linear', 'linear', 'linear', 'full'] * 6
assert len(layer_types) == 24
assert layer_types.count('linear') == 18
assert layer_types.count('full') == 6
cache_bytes = 2 * layer_types.count('full') * 4096 * 2 * 256 * 2
assert cache_bytes / 2**20 == 48
print(cache_bytes / 2**20, 'MiB for full-attention K/V only')

The count is checkpoint-specific. A different family member may use different depth, head counts, dimensions or feed-forward design. Read the configuration rather than deriving them from the model name.

Does three-quarters linear attention mean three-quarters less inference memory? No. Parameters, recurrent state, full-attention caches and runtime allocations have different sizes. Ratios of layers are not ratios of total bytes or latency.

Can FlashAttention still be relevant? Yes. A hybrid's full-attention layers can use an exact optimized kernel; the linear layers need kernels for their own recurrence or chunkwise computation.

Double the prefix length while holding the model, batch and cache dtype fixed. The calculated full-attention K/V storage doubles. The fixed-size recurrent matrices do not acquire one new matrix per token.

Sources: official Qwen3.5-0.8B model card, checkpoint configuration. Reviewed October 2026. No inference or benchmark reproduction is claimed.