The Transformer · 2017

Represent tokens and positions

Source tokens and shifted target tokens enter separate embedding paths. The original model scales embeddings by √d_model and adds sinusoidal positions before either stack.

x = √512 · Embedding(token) + PE(position)

Vaswani et al. · Figure 1

Give complex ideas a form.

Interactive models, figures from your data, and technical diagrams.

Open Studio

Free with LiminalML attribution · No account needed

Find a form for your next idea.

Technical visuals to make your own. For papers, presentations, and the explanations in between.

Browse the gallery

View preset

From a mechanism to your next explanation.

Turn an ML mechanism into an interactive explanation for your next article or presentation.

LoRA constrains the update, not the base model.

That’s the point this article needs to make. Keep the frozen weight W, then show the correction BAx occupying a lower-dimensional space.

ΔW = (α/r)BA · rank(ΔW) ≤ r

For a 4096 × 4096 matrix at rank 8, the adapter has 65,536 trainable parameters. The 16,777,216 base weights remain.

LoRA paper · Hu et al.

Preparing the 3D view…

The same story, in your presentation.

Edit this explanation

16:9 slide · Keynote / PowerPoint

LoRA constrains the update, not the base model.

Preparing the 3D view…

Change rank from one to three.

Slide layout preview. Studio Publish exports this layout as PowerPoint, with rendered figures and full technical context in speaker notes. Free PNG downloads contain the figure itself.

The architecture keeps evolving.

Follow the trade-off from exact GPU kernels to recurrent memory and sparse retrieval.

Same attention. Less memory traffic.

Tiles of Q, K and V move through fast on-chip memory. An online softmax combines their contributions without storing the full score or probability matrix in high-bandwidth memory.

Context length 4,096 tokens

Dense score entries / head: 16,777,216
One 64 × 64 score tile: 4,096

Element counts with fixed head width 64. Excludes model weights, other intermediates, batching and layers; not peak GPU memory or a speed benchmark.

Every score still contributes. Only the tile and running row statistics need temporary storage.

Exact dense softmax attention up to floating-point rounding. Arithmetic remains quadratic in sequence length; this is not a linear-attention architecture. Dao et al., 2022

Qwen3.5-0.8BA small dense model with a 3:1 hybrid attention pattern.DeepSeek V4.1-FlashCompressed sparse attention and shared encoder memory.Inside linear attentionState updates, delta rules, and what fixed memory can forget.

Local drafts · Share snapshots · Download editable projects

Free downloads & export formats

The architecture keeps evolving.

Follow the trade-off from exact GPU kernels to recurrent memory and sparse retrieval.

Same attention. Less memory traffic.

Tiles of Q, K and V move through fast on-chip memory. An online softmax combines their contributions without storing the full score or probability matrix in high-bandwidth memory.

Context length 4,096 tokens

Dense score entries / head: 16,777,216
One 64 × 64 score tile: 4,096

Element counts with fixed head width 64. Excludes model weights, other intermediates, batching and layers; not peak GPU memory or a speed benchmark.

Every score still contributes. Only the tile and running row statistics need temporary storage.

Exact dense softmax attention up to floating-point rounding. Arithmetic remains quadratic in sequence length; this is not a linear-attention architecture. Dao et al., 2022

Represent tokens and positions

Give complex ideas a form.

Find a form for your next idea.

LoRA update space

Rotary position encoding

KV-cache memory

PCA projection

From a mechanism to your next explanation.

LoRA constrains the update, not the base model.

The same story, in your presentation.

LoRA constrains the update, not the base model.

The architecture keeps evolving.

Same attention. Less memory traffic.

Represent tokens and positions

Give complex ideas a form.

Find a form for your next idea.

LoRA update space

Rotary position encoding

KV-cache memory

PCA projection

From a mechanism to your next explanation.

LoRA constrains the update, not the base model.

The same story, in your presentation.

LoRA constrains the update, not the base model.

The architecture keeps evolving.

Same attention. Less memory traffic.