Understand the systems you work with.

ML and software engineering, from intuition to implementation.

Open Study

2 free sessions · No account needed

Find the gap. Work through it.

Find the gaps in your ML understanding. Work through them, then explain the idea with or without your notes.

A small detail changes the implementation.

“T5’s 32 buckets mean 8 exact distances and 24 logarithmic ones.”

That misses a branch in the code. The encoder separates direction first; the decoder doesn’t. Follow the allocation and the difference becomes visible.

Authored sample session. This is a misconception to investigate, not a learner result.

T5 · 32 relative-position buckets

Attention mode

Exact distanceLogarithmic distance

Key − query position -17

Offset -17 selects bucket 10.

16 slots per direction; 8 exact and 8 logarithmic in each half. Zero uses bucket 0; bucket 16 is unused. Max distance: 128. Colors show allocation, not learned bias values.

Reference implementation

Know what the evidence says.

A clear explanation is useful progress. An implementation that passes checks adds evidence. Applying the idea unaided to a new failure adds something different.

Read the full example and runnable code

Reference implementation: Checks not run yet
Support used on the new problem: No hint or reference opened
Your explanation: Not observed yet
Unaided use a week later: Not observed here

A mastery label needs this context. This demo records only the actions above; it does not establish durable learning or general competence.

Follow an idea into the details.

Worked examples that connect the picture, the calculation, and the implementation.

How current models make attention cheaper

Separate efficient kernels, recurrent state, sparse selection and hybrid architectures.

From tokens to a prediction

Follow the computation, then inspect the operations that make it work.

Change a model. Understand the constraint.

Connect gradients, low-rank updates, and the geometry of projection.

Read the architecture, not just the benchmark.

Recent open-weight models, from a small on-device design to a trillion-parameter MoE. Work through the mechanism, a calculation, and a failure case. Sources reviewed October 2026.

DDDA

One four-layer pattern, repeated 6 times. D: Gated DeltaNet + FFN. A: gated full attention + FFN.

0.8B · dense hybrid