T5 Relative-Position Bias

Relative-position bias adds a learned scalar to an attention score using the separation between a query and a key. T5 shares a bias value among offsets assigned to the same bucket. This is a learned preference in score space, not an extra token embedding or a rotary transformation.

The often-repeated “8 exact plus 24 logarithmic” description of T5's 32 buckets misses the encoder/decoder distinction. Direction is allocated before distance. The visual below executes the mapping; its colors are not trained bias values.

Use the convention offset = key position − query position. A negative offset looks backward; a positive one looks forward.

For bidirectional attention, the 32 slots split into two directional halves. Each half allocates 8 slots to exact distances and 8 to logarithmic distances. For causal attention there is no split: 16 slots are exact and 16 logarithmic. With this implementation zero selects bucket 0 and bidirectional bucket 16 is never selected. A separate mask blocks future keys in the decoder.

Try −17, then +17. Encoder buckets are 10 and 26. In the causal mapping −17 selects 16, while +17 clamps to 0. That last number does not grant permission to attend to a future token.

Let $S$ be the slots available after the direction split: 16 for bidirectional mode, 32 for causal mode. Set $E=S/2$ and $M=128$. For a nonnegative distance $d$, the within-half index is

In bidirectional mode use the absolute offset, adding 16 for a positive offset. In causal mode use the negated offset clamped at zero. Large distances saturate at the last slot. Saturation starts at the lower boundary of that final logarithmic bin, not necessarily at exactly 128.

A learned table maps the resulting index to one bias per attention head. Distances sharing an index share that positional contribution; their content-based scores can still differ.

Download the complete executable example. Run python3 t5_buckets.py; it needs only the Python standard library. The script checks encoder and decoder outputs, plus the cache-position example below.

1from math import floor, log
2
3def bucket(offset, bidirectional=True):
4    slots = 16 if bidirectional else 32
5    exact = slots // 2
6    distance = abs(offset) if bidirectional else max(0, -offset)
7    direction = 16 if bidirectional and offset > 0 else 0
8    if distance < exact:
9        return direction + distance
10    growth = floor(log(distance / exact) / log(128 / exact) * exact)
11    return direction + min(slots - 1, exact + growth)
12
13assert [bucket(-17), bucket(17)] == [10, 26]
14assert [bucket(-17, False), bucket(17, False)] == [16, 0]

This is a scalar teaching implementation for the default configuration, not a replacement for a vectorized attention kernel. It guards the logarithm by branching before evaluating it at zero.

Imagine a cached decoder at absolute query position 128 reading key position 124. The correct offset is −4, yielding causal bucket 4. If the query index is accidentally reset to zero, the computed offset becomes +124 and clamps to bucket 0. The code may remain numerically valid while encoding the wrong relationship.

A useful test suite therefore covers self-position, both signs, exact/log boundaries, saturation and cached absolute positions. Check masking independently: a passing bucket test says nothing about whether future attention scores were excluded.

Change the number of buckets to 64 while keeping the maximum distance at 128. Before writing code, determine how many exact-distance slots each bidirectional half receives. Then explain why the causal case differs.

Reference: each bidirectional half has 32 slots, of which 16 are exact. The causal case keeps all 64 together, of which 32 are exact. The logarithmic scale must change with those values; changing just an array's length is insufficient.

Source

Hugging Face's T5 implementation, specifically _relative_position_bucket and compute_bias, reviewed October 14, 2026. The browser visual, Python script and debugging cases are authored teaching examples.