Linear Attention

Linear attention avoids an ever-growing pairwise attention computation by summarizing history in a recurrent state. This changes the mechanism. FlashAttention instead reorganizes the computation of exact softmax attention.

“Linear” describes scaling in sequence length with feature widths fixed. It does not mean the state is a single vector, the model is linear in its inputs, or every implementation is faster at every context length.

Think of a matrix as an associative memory. A key describes where to write; a value describes what to store; a query reads from that state. Newer designs add forgetting gates and targeted replacement rules. A compact state is useful for streaming and long contexts, but several stored associations can interfere with each other.

A hybrid model keeps some full-attention layers for explicit token retrieval and uses recurrent layers elsewhere. The Qwen3.5-0.8B breakdown shows one concrete pattern. A hybrid's total memory is not constant in context length because its full-attention layers still keep caches.

A normalized kernel formulation uses positive feature maps $\phi$:

$S_t = S_{t-1} + \phi(k_t)v_t^\top, \qquad z_t = z_{t-1} + \phi(k_t)$ $o_t = \frac{\phi(q_t)^\top S_t}{\phi(q_t)^\top z_t}$

This has a fixed-size state for fixed key-feature and value widths. It is not generally the same function as softmax attention.

A gated delta rule uses a different, unnormalized memory convention. With $S\in\mathbb{R}^{d_v\times d_k}$ and normalized keys, one form is:

$\widetilde S_t = \alpha_t S_{t-1}$ $S_t = \widetilde S_t + \beta_t(v_t-\widetilde S_t k_t)k_t^\top, \qquad o_t=S_tq_t$

The gate decays old memory. The delta term writes the difference between the incoming value and the value currently associated with this key. Do not mix the orientation or normalization of these two formulations.

This scalar-value, two-key example isolates the delta update with no decay. It is not a complete Gated DeltaNet layer.

1state = [0., 0.]
2def write(key, value, beta=1.):
3    prediction = sum(s*k for s,k in zip(state, key))
4    for i in range(2):
5        state[i] += beta * (value - prediction) * key[i]
6write([1., 0.], 2.)
7write([0., 1.], 3.)
8write([1., 0.], 5.)
9assert state == [5., 3.]
10print(state)  # Replaces the first association; preserves the orthogonal one.

Why use a hybrid? Recurrent state limits per-layer history storage, while occasional full-attention layers retain direct access to past token representations. Layer counts alone do not establish quality or latency.

Where does compression fail? Non-orthogonal keys can interfere. Targeted updates help control overwriting but do not give a finite matrix unlimited exact recall.

Must training be sequential? The recurrence defines semantics. Chunkwise algorithms can parallelize training; efficient kernels remain important even when the mechanism is linear in sequence length.

In the code, replace the second key with [1., 0.]. Can both values remain independently retrievable? No: both writes address the same direction. The second overwrites the first when the write strength is one.

Sources: Transformers are RNNs, Gated Delta Networks, official implementation. These sources describe different members of the family; the normalized kernel rule is not a full Qwen implementation.