Gemma 4 E2B
Separate per-layer embeddings, stored weights and attention patterns in a small multimodal model.
Jump to a section
Five billion parameters, labelled 2B
Gemma 4 E2B is a lesson in reading a model's size label. The technical report describes it as 2.3B effective parameters out of 5B in total, and gets there with per-layer embeddings. "Effective" does not make the 2.7B difference disappear from the checkpoint. Those weights still have to be stored, loaded or made reachable, and a runtime has to account for them.
The Gemma 4 family spans dense and mixture-of-experts models. This breakdown stays with E2B, because several family-level optimisations do not apply to it.
What a lookup costs compared with a layer
A token can bring information into the network in two ways: through an embedding lookup, or through the matrix multiplications of the decoder layers. They cost very differently. A matrix multiplication uses every weight in the matrix for every token. A lookup reads one row: in a table of rows, each token touches of it.
Per-layer embeddings put that cheap path at several depths, adding token-dependent information to layers beyond the first. A large table can therefore hold many parameters while adding little arithmetic per token, which is why total parameter count is a poor guide to per-token work in this design. Keep two questions apart: what work is done for one token, and which tables must be stored or made accessible.
The schematic shows the per-layer embedding path and the layer pattern described next. It leaves out normalisation, feed-forward sublayers, multimodal encoders and cache sharing.
Four local layers for every global one
The report gives E2B a 4:1 pattern of local to global attention. A local layer lets each query attend only to a window of nearby positions. A global layer can connect any two positions in the context.
For a causal layer at token index , a local window of width exposes at most
keys, including the current token. Full causal attention exposes . This counts allowed query–key relationships. It is not a count of floating-point operations, and it says nothing about what information can reach the token indirectly through a stack of layers.
Take and an illustrative , a teaching value rather than Gemma's configured window. A local layer exposes 128 keys and a global layer 1,000. One repeat of the pattern, four local layers and one global, exposes relationships for that query, against 5,000 for five global layers. As grows the local share stays fixed at 512, so the pattern's total falls toward one fifth of the all-global count.
The larger Gemma 4 models also use a key-as-value optimisation. The report states that E2B and E4B do not, which is one reason a family diagram is not enough to implement a particular checkpoint.
Counting visible keys in code
The function below returns the key positions a query may attend to, with or without a window, and checks the boundary cases from the teaching example.
1def visible_keys(query, window=None):
2 start = 0 if window is None else max(0, query - window + 1)
3 return range(start, query + 1)
4
5assert list(visible_keys(2, 128)) == [0, 1, 2]
6assert len(visible_keys(999, 128)) == 128
7assert len(visible_keys(999)) == 1000
8local = 4 * len(visible_keys(999, 128))
9global_block = len(visible_keys(999))
10assert local + global_block == 1512
11print(local + global_block) # 1512, not a latency estimateEarly positions are the edge case: at query 2 the window is not yet full, so a local layer sees the same three keys a global one would. A real implementation also needs the checkpoint's exact layer types, window sizes, cache-sharing rules and positional encoding. Treat this as a mask-boundary exercise, not inference code.
Where memory estimates go wrong
Blaming the label. A runtime loads the checkpoint, then runs out of memory on a long request. The effective-parameter label is not a diagnosis. Separate stored weights, temporary activations, attention caches, batch size, allocator overhead and any offloading, and find which one grew.
Multiplying by every layer. A teammate estimates the cache as sequence length times the number of decoder layers. A local layer only ever needs keys and values for its window, so its cache can stop growing at positions while a global layer's keeps growing. Check which layers are local or global, whether cache tensors are shared, the head dimensions, the precision and the runtime's allocation strategy against the checkpoint's configuration. The layer-pattern schematic alone cannot give you allocated bytes.
Reading local as blind. A local layer limits its direct range, but information still travels through other layers, including the global ones. It is wrong to say the model can never use evidence from outside one local window.
Sources and further reading
- Gemma 4 technical report, §2 and Tables 1 to 3: parameter labels, per-layer embeddings, the 4:1 local/global pattern and the key-as-value distinction. The window of 128, the code and the scenarios are LiminalML teaching examples; we have not run the model or reproduced its benchmarks.