Gemma 4 E2B

Gemma 4 E2B is a useful case study in reading a model's size label carefully. The technical report describes 2.3B effective parameters out of 5B total, using per-layer embeddings. “Effective” does not mean all other weights disappear from the checkpoint.

The family spans dense and mixture-of-experts designs. This article concentrates on E2B rather than transferring every family-level optimization to the smallest model.

A token can provide information through an embedding lookup as well as through the dense transformations in a decoder. Per-layer embeddings add token-dependent information at multiple layers. Separate the work performed for a particular token from the tables that must be stored or made accessible.

The report gives E2B a 4:1 local-to-global attention pattern. Local blocks limit direct attention to a nearby window; global blocks can connect distant positions. The diagram isolates those ideas. It omits normalization, feed-forward sublayers, multimodal encoders and cache-sharing details.

The larger models' key-as-value optimization is explicitly not used in E2B or E4B. A family diagram is not enough to implement a particular checkpoint.

For a hypothetical local causal attention block at token index $t$, a window of width $w$ exposes at most

keys, including the current token. A full causal attention block exposes $t+1$ keys. This counts allowed query-key relationships, not floating-point operations or the information reachable through a stack of layers.

At $t=999$ with an illustrative $w=128$, that is 128 direct keys locally versus 1,000 globally. Four local blocks followed by one global block expose $4(128)+1000=1512$ relationships for that query across the five blocks. Five full-attention blocks expose 5,000. This toy count deliberately uses a teaching window, not Gemma's configured window, and excludes projections, cache sharing and kernels.

This executable Python example checks the boundary conditions in that teaching window:

1def visible_keys(query, window=None):
2    start = 0 if window is None else max(0, query - window + 1)
3    return range(start, query + 1)
4
5assert list(visible_keys(2, 128)) == [0, 1, 2]
6assert len(visible_keys(999, 128)) == 128
7assert len(visible_keys(999)) == 1000
8local = 4 * len(visible_keys(999, 128))
9global_block = len(visible_keys(999))
10print(local + global_block)  # 1512, not a latency estimate

A real implementation also needs the checkpoint's exact layer types, window sizes, cache-sharing rules and positional encoding. Treat this as a mask-boundary exercise, not inference code.

Suppose a runtime loads the checkpoint successfully but runs out of memory during a long request. Blaming the effective parameter label is not a diagnosis. Separate stored weights, temporary activations, attention caches, batch size, allocator overhead and any offloading.

A local block reduces its direct attention range, but information can travel indirectly through other layers. It is inaccurate to say the whole model can never use evidence outside one local window.

A teammate estimates every layer's cache by multiplying sequence length by the number of decoder layers. What would you inspect before accepting that number?

Look for which layers are local or global, whether cache tensors are shared, head dimensions, precision and the runtime's allocation strategy. Check the actual checkpoint configuration against the report. The layer-pattern diagram alone cannot determine allocated bytes.

Source and scope

Gemma 4 technical report, §2 and Tables 1–3. Parameter labels, architecture distinctions and the local/global ratio come from the report. The window calculation, code and debugging scenarios above are LiminalML teaching examples. We have not run the model or reproduced its benchmarks.