Skip to content
Research visuals

The architecture behind the attention.

The full encoder–decoder Transformer, with the structure of Figure 1 from Attention Is All You Need. Select any block to follow its role.

The Transformer · Vaswani et al., 2017Six encoder layers and six decoder layers. Encoder output supplies keys and values to each decoder cross-attention sublayer. Residual connections use post-normalization. Inputs and shifted outputs receive positional encoding.Encoder × 6Decoder × 6Input embeddingOutput embedding+Positionencoding+PositionencodingMulti-HeadAttentionAdd & NormFeed ForwardAdd & NormMasked Multi-HeadAttentionAdd & NormMulti-HeadAttentionAdd & NormFeed ForwardAdd & NormLinearSoftmaxOutput probabilitiesSource tokensTarget tokens · shifted rightKVQStructure: Vaswani et al., Figure 1 (2017) · Diagram: LiminalML
The Transformer · 2017

Represent tokens and positions

Source tokens and shifted target tokens enter separate embedding paths. The original model scales embeddings by √d_model and adds sinusoidal positions before either stack.

x = √512 · Embedding(token) + PE(position)

Encoder self-attention · all source positions

QKᵀ / √d_kmask · softmaxweighted sum of V
Mask structure only; colors are not measured attention weights.

Original base model: d_model = 512, 8 heads, d_k = d_v = 64. Post-norm; dropout is inside the residual branch. This map shows structure, not live model activations.

Explore positional encodingVaswani et al. · Figure 1

Source-grounded structure, not a running language model. The original architecture uses post-normalization; the numerical lab in Study shows a separately labelled pre-norm variant.

Read the complete explanation and original figure

The architecture keeps evolving.

Follow the trade-off from exact GPU kernels to recurrent memory and sparse retrieval.

Same attention. Less memory traffic.

Tiles of Q, K and V move through fast on-chip memory. An online softmax combines their contributions without storing the full score or probability matrix in high-bandwidth memory.

Dense score entries / head
16,777,216
One 64 × 64 score tile
4,096

Element counts with fixed head width 64. Excludes model weights, other intermediates, batching and layers; not peak GPU memory or a speed benchmark.

KeysOne tileQueries × keysonline softmaxcombine values
Every score still contributes. Only the tile and running row statistics need temporary storage.

Exact dense softmax attention up to floating-point rounding. Arithmetic remains quadratic in sequence length; this is not a linear-attention architecture. Dao et al., 2022

Qwen3.5-0.8BA small dense model with a 3:1 hybrid attention pattern.DeepSeek V4.1-FlashCompressed sparse attention and shared encoder memory.Inside linear attentionState updates, delta rules, and what fixed memory can forget.