Linear Attention
Understand recurrent memory, gated delta updates, and why modern hybrids retain some full-attention layers.
Jump to a section
A memory that does not grow with the prefix
A softmax attention layer keeps a key and a value for every token it has seen. For one head of width 64 that is 128 numbers per token, so at 4,096 tokens the head holds 524,288 cached numbers, and the next query reads all of them. Both the storage and the work per new token grow with the prefix.
Linear attention replaces that history with a fixed-size state. Each token writes into a matrix, 4,096 numbers, and each query reads from it. Past 32 tokens the state is already smaller than the cache it replaces, and at 4,096 tokens the cache is 128 times larger.
This changes the mechanism, not just its execution. FlashAttention computes exact softmax attention more efficiently; linear attention computes a different function. "Linear" refers to cost growing linearly in sequence length with the feature widths fixed. It does not mean the state is a single vector, that the model is linear in its inputs, or that every implementation is faster at every context length.
Dropping the softmax lets the product reassociate
Softmax attention computes , and the softmax forces the score matrix to exist before is applied. Replace the exponential with a product of positive feature maps, , and the bracket can move: . The right-hand side never forms an matrix. is , whatever is.
Under a causal mask the same idea becomes a recurrence. Keep a running sum of key-value outer products and a running sum of keys:
Replace the growing history with a recurrent state.
A fixed-size matrix summarizes key–value associations. Each new token updates that state and reads it with its query. Gated and delta-rule variants control what is forgotten or overwritten.
- Dense score entries / head
- 16,777,216
- One 64 × 64 recurrent matrix
- 4,096
Element counts with fixed head width 64. Excludes model weights, other intermediates, batching and layers; not peak GPU memory or a speed benchmark.
This changes the attention mechanism. Fixed-size memory can lose distinctions between past tokens. A hybrid still retains KV caches in its full-attention layers. Yang et al., 2025
Katharopoulos et al. used and showed that a causal Transformer written this way is an RNN, with as its hidden state. They reported autoregressive generation up to 4,000 times faster than softmax attention on very long sequences. The state is fixed in size for fixed key-feature and value widths, but the function is not the same as softmax attention. Later variants, Gated DeltaNet among them, change how the state is written to narrow the quality gap.
Writing to a matrix memory
The state is easiest to reason about as an associative memory. A key says where to write, a value says what to store there, and a query reads back whatever is stored along its direction. In the plain sum above, every write is added on top of what is already there. Nothing is ever forgotten or replaced, so associations pile up and interfere.
The delta rule fixes the replacement half. Before writing, it reads what the memory currently returns for this key, and writes only the difference. With and unit-norm keys, and with a decay gate added as in Gated DeltaNet:
The gate decays old memory; the write strength controls how much of the correction lands. With and , reading the key straight back gives : the new value replaces the old one exactly.
This form is unnormalized and stores the matrix the other way round from the kernel formulation, instead of . Do not mix the orientation or normalization of the two.
In code
A two-dimensional delta-rule memory with scalar values and no decay. It is not a complete Gated DeltaNet layer, but it shows replacement and interference exactly.
1state = [0., 0.]
2def write(key, value, beta=1.):
3 prediction = sum(s*k for s,k in zip(state, key))
4 for i in range(2):
5 state[i] += beta * (value - prediction) * key[i]
6write([1., 0.], 2.)
7write([0., 1.], 3.)
8write([1., 0.], 5.)
9assert state == [5., 3.]
10print(state) # Replaces the first association; preserves the orthogonal one.
11
12read = lambda key: sum(s*k for s,k in zip(state, key))
13state[:] = [0., 0.]
14write([1., 0.], 2.)
15write([0.6, 0.8], 3.) # unit norm, but not orthogonal to the first key
16assert abs(read([0.6, 0.8]) - 3.) < 1e-12
17assert abs(read([1., 0.]) - 3.08) < 1e-12 # the first association was disturbedThe first half writes 2 under one key, 3 under an orthogonal key, then 5 under the first key again. The third write replaces the 2 and leaves the 3 alone. Write the second value under [1., 0.] instead and the two cannot both survive, because both writes address the same direction.
The second half uses keys that overlap. Each write is still exact for its own key, but writing the second value moved the memory along the first key's direction, so the first key now reads back 3.08 instead of 2. A -dimensional memory has only orthogonal directions, and once there are more associations than directions, some interference is unavoidable.
Hybrids keep a few full-attention layers
A fixed state cannot give exact recall of an arbitrary earlier token, and that is exactly what full attention is good at. Hybrid models use recurrent layers for most of the depth and keep some full-attention layers for direct retrieval.
Qwen3.5-0.8B has 24 text layers: 18 Gated DeltaNet layers and 6 gated full-attention layers, in a repeating pattern of three recurrent layers then one attention layer. At 4,096 tokens the six attention layers hold 48 MiB of KV cache; with all 24 as full attention of the same shape it would be four times that.
A hybrid's memory is therefore not constant in context length. The recurrent layers stay fixed, but the full-attention layers still grow a KV cache with every token. Counting layers does not tell you the quality or latency either; those need measuring.
Where the fixed state falls short
Recall. Gates and targeted updates control what gets overwritten. They do not give a finite matrix unlimited exact recall. Retrieving one specific earlier token is what a full-attention layer does directly and what a compressed state can fail at, which is the reason hybrids exist.
Training. The recurrence defines what the layer computes, not how it must be trained. Chunkwise algorithms process blocks of tokens in parallel and pass the state between blocks, and efficient kernels still matter even when the cost is linear in sequence length.
Short contexts. Linear in sequence length describes how cost grows, not the speed at any given length. When the context is short the quadratic term is small too, and a well-optimized softmax kernel may still be faster, so compare at the lengths you will actually serve.
Sources and further reading
- Katharopoulos et al., Transformers are RNNs (2020): kernelized attention, the recurrent form and the elu + 1 feature map.
- Yang et al., Gated Delta Networks (2024): combining a decay gate with the delta rule.
- Official Gated DeltaNet implementation: reference code. These sources describe different members of the family; the normalized kernel rule is not a full Qwen implementation.