LLMs
Tokenization & BPE
Follow a pair of vocabulary merges and distinguish learning merge rules from applying a tokenizer to new text.
Jump to a section
Why this matters
Byte-pair encoding builds a subword vocabulary by repeatedly merging frequent adjacent symbols in a training corpus. At inference, the tokenizer applies the learned merge rules to turn new text into token IDs.
Separate learning the vocabulary from tokenizing a string with that vocabulary. A merge that looks useful inside one word may have a different priority in the full corpus. Byte-level variants start with bytes, which helps represent unfamiliar text.
Build the mental model
Training: look at a big corpus as sequences of characters. Find the most frequent adjacent pair (say t + h → th), merge it into a new symbol, and add it to the vocabulary. Repeat. Frequent sequences like ing, tion, or whole common words get absorbed into single tokens; rare words stay split into subwords.
Follow two vocabulary merges
Read each box as the current token sequence. Merge e + r first, then l + o, using an already learned merge order.
The visual loads as you reach this section.
Read the connections
l o w e r -> l o w er : merge e + r l o w er -> lo w er : merge l + o
Encoding new text just replays the learned merges in the order they were learned. The result: "tokenization" might be one token, while "antidisestablishmentarianism" becomes several subwords, but nothing is ever unrepresentable.
Work through the math
BPE is an algorithm rather than a closed-form equation. Training to a target vocabulary size :
- Initialize the vocabulary with all base symbols (characters, or the 256 bytes for byte-level BPE).
- Count the frequency of every adjacent symbol pair across the corpus.
- Merge the most frequent pair ; add to the vocabulary and record the merge rule.
- Repeat steps 2–3 until (e.g. 50,257 for GPT-2).
Encoding a new string: greedily apply the learned merge rules in learned order until no more apply. The number of tokens for a piece of text is what counts against a model's context window and per-token API pricing, so tokenization efficiency directly affects cost and how much you can fit in context.
Key takeaway: BPE = greedy frequency-based merging from characters up to a target vocab size. Subword wins because it's the only granularity with no out-of-vocabulary tokens and reasonable sequence lengths, and token count, not word count, is what you pay for.
| Word-level | Character-level | Subword (BPE) | |
|---|---|---|---|
| Vocabulary size | Huge (100k+) | Tiny (256 bytes) | Tunable (32k–256k) |
| Out-of-vocabulary tokens | Common | None | None |
| Sequence length | Short | Very long | Moderate |
| Cost driver per request | Low token count | High token count | Balanced |
Read the implementation
1from collections import Counter
2
3def get_pairs(tokens: list[str]) -> Counter:
4 return Counter(zip(tokens, tokens[1:]))
5
6def train_bpe(corpus: list[str], num_merges: int) -> list[tuple[str, str]]:
7 tokens = list(" ".join(corpus)) # char-level start
8 merges: list[tuple[str, str]] = []
9 for _ in range(num_merges):
10 pairs = get_pairs(tokens)
11 if not pairs:
12 break
13 best = pairs.most_common(1)[0][0] # most frequent adjacent pair
14 merges.append(best)
15 merged, i = [], 0
16 while i < len(tokens): # apply the merge in one pass
17 if i < len(tokens) - 1 and (tokens[i], tokens[i + 1]) == best:
18 merged.append(tokens[i] + tokens[i + 1])
19 i += 2
20 else:
21 merged.append(tokens[i])
22 i += 1
23 tokens = merged
24 return merges
25
26merges = train_bpe(["lower", "lowest", "newer"], num_merges=5)
27print(merges) # e.g. [('e','r'), ('l','o'), ('lo','w'), ...]Questions and trade-offs
- Conceptual: Why use subword tokenization instead of words or characters? (Words explode the vocabulary and hit OOV; characters make sequences too long. Subwords balance vocabulary size, sequence length, and never produce OOV.)
- Implementation: What does one BPE training step do? (Count adjacent symbol-pair frequencies and merge the most frequent pair into a new vocabulary symbol.)
- Applied: Why does token count, not word count, matter for context limits and cost? (The model and pricing operate on tokens; a rare word may be several tokens, so token count determines how much fits in context and what you pay.)
- Systems-level: What is byte-level BPE and what problem does it solve? (BPE over raw bytes instead of characters: it can represent any Unicode text, including emoji and rare scripts, with no unknown tokens.)
- Failure modes: How does BPE vs. WordPiece differ? (Both are subword; BPE merges by raw pair frequency, WordPiece merges by which pair most increases corpus likelihood under a language model.)
Check your understanding
Without looking: describe the BPE training loop in three steps, explain why subword beats word- and character-level, and say why token count drives cost and context. Check against Stage 3.
Oliver Perrin
Machine Learning Engineer · Founder, LiminalML
LiminalML brings together concept explanations, mathematical examples, and working visuals. Find more writing by Oliver on the LiminalML Substack.
Keep exploring the idea.
Ask a follow-up in a guided session, work through the self-check, or open the visual in Studio to build on it.
Related concepts
LLMs
Embeddings
Compare vectors by direction and magnitude, and connect embedding geometry to retrieval and representation learning.
Deep Learning
Transformer Architecture
Follow attention, feed-forward layers, normalization, and residual connections through a Transformer block.
LLMs
Temperature, Top-k & Top-p Sampling
Watch temperature reshape probabilities and distinguish that transformation from top-k, top-p, and sampling.