EmbeddingGemma 2

EmbeddingGemma 2 is an open embedding model that maps text, code, images, audio, video and interleaved mixtures of them to one 768-dimensional unit vector. A query in one modality can retrieve a target in another because both land in the same space.

Earlier designs forced a choice. CLIP-style models add a separate tower per modality, so they compare an image with a caption but cannot read an image and a caption together as one input. Embedding models adapted from multimodal LLMs can read mixed input, but they usually have several billion parameters. EmbeddingGemma 2 keeps mixed input at 744M parameters and loads only the encoders a task needs.

The paper's headline result is a new size–quality frontier on four benchmark suites. Each star below sits above the previous frontier at its parameter count.

On a Google Pixel 11 Pro, the int4 per-channel checkpoint peaks at 191 MB for text, 447 MB with vision and 567 MB with every encoder, including runtime compilation overhead.

Think of the model as one reader with optional eyes and ears. A vision encoder turns an image, or each sampled video frame, into a short sequence of "soft tokens". An audio encoder does the same for sound at 25 tokens per second. Text goes through the ordinary embedding table. Linear adapters project every soft token to the backbone width of 512, and the pieces are concatenated in their original order between delimiter tokens.

The backbone is a Gemma 4 decoder with its causal mask switched to bidirectional attention. A generator must not see future tokens; an embedding model should, because the vector summarizes the whole input. After 24 layers the model mean-pools every position, projects 512 to 768 dimensions and rescales the result to unit length. Similarity is then a dot product.

Because encoders only produce input tokens, they are modular. Text-only search can skip both encoders and run 271M parameters. Use the selector to see which pieces each configuration loads. The audio encoder is frozen and bit-identical to Gemma 4's; the vision encoder is trained, which matters later.

The embedding. For token embeddings $\mathbf T_{\text{in}}\in\mathbb R^{L\times 512}$ and the 24-layer bidirectional transformer $\mathcal M$,

where $f$ is a linear map. Every input, whatever its length or modality, becomes one point on the unit sphere.

How long can one input be? Images are pooled to at most $N_{\text{vis}}$ soft tokens (280 by default; 70 to 1,120 at inference). Audio gives 25 per second. Video is sampled at 1 frame per second and capped at 32 frames, at 140 tokens per frame by default. Each image, frame or audio segment adds two delimiters. Training sequences reach 8,192 positions. A 120-second clip with its soundtrack therefore needs at most $32(140+2)+120(25+2)+2=7{,}786$ positions, but 32 frames at the image budget alone need $32(280+2)=9{,}024$. Change the inputs and budgets below.

The contrastive loss. Each example has a query $\mathbf q_i$, a positive target $\mathbf p_i^+$ and hard negatives $\mathbf p_i^-$. With cosine similarity $s$ and temperature $\tau$,

$Z^{HN}_i$ sums over the example's own hard negatives. $Z^{IB}_i$ sums over the other examples' positive targets. The same-tower term $Z^{STN}_i$ also pushes query away from query and target away from target, which keeps queries and documents from settling into two separate regions. A mask $\mathbb 1_{\text{TN}}(i,j)$ removes example $j$ when its query or target duplicates example $i$'s, so a repeated target is never penalized as a negative.

Three supporting losses. A spread-out regularizer penalizes $(\mathbf q_i^\top\mathbf q_j)^2$ and $(\mathbf p_i^{+\top}\mathbf p_j^+)^2$ for random pairs. Independent uniform points on a $d$-dimensional sphere have mean dot product 0 and mean squared dot product $1/d$, so the penalty pushes the space toward that isotropic baseline. Distillation matches the student's vectors to truncated, normalized Gemini Embedding 2 vectors, $\lVert\mathbf u_i-\tilde{\mathbf u}_i\rVert_2^2$. For unit vectors that equals $2-2\cos(\mathbf u_i,\tilde{\mathbf u}_i)$: matching a teacher is maximizing cosine similarity with it. Matryoshka Representation Learning applies each loss to the first 768, 512, 256 and 128 coordinates.

This executable Python example implements Equation 2 for one hard negative per example and checks the duplicate mask. The temperature is our illustrative choice; the paper does not report $\tau$.

1import math
2
3def cos(a, b):
4    dot = sum(x * y for x, y in zip(a, b))
5    return dot / math.sqrt(sum(x * x for x in a) * sum(y * y for y in b))
6
7def contrastive_loss(q, p, hard, tau=0.2, mask=True):
8    """Equation 2 with one hard negative per example."""
9    B, total = len(q), 0.0
10    e = lambda a, b: math.exp(cos(a, b) / tau)
11    for i in range(B):
12        if mask:   # 1_TN drops j when its query or target duplicates example i
13            others = [j for j in range(B) if q[j] != q[i] and p[j] != p[i]]
14        else:
15            others = [j for j in range(B) if j != i]
16        z_p = e(q[i], p[i])
17        z_hn = e(q[i], hard[i])
18        z_ib = sum(e(q[i], p[j]) for j in others)
19        z_stn = sum(e(q[i], q[j]) + e(p[i], p[j]) for j in others)
20        total -= math.log(z_p / (z_p + z_hn + z_ib + z_stn))
21    return total / B
22
23q = [[1.0, 0.2], [0.1, 1.0], [0.9, 0.4]]
24p = [[0.9, 0.1], [0.1, 0.9], [0.9, 0.1]]   # example 3 repeats example 1's target
25hard = [[0.2, 1.0], [1.0, 0.3], [0.3, 1.0]]
26
27masked, unmasked = contrastive_loss(q, p, hard), contrastive_loss(q, p, hard, mask=False)
28assert masked < unmasked   # without 1_TN, a duplicate target is punished as a negative
29print(round(masked, 3), round(unmasked, 3))
30
31# Section 2.2 upper bounds: soft tokens plus two delimiters per image, frame or audio segment
32video_120s = min(120, 32) * (140 + 2) + 120 * (25 + 2) + 2   # frames, soundtrack, BOS/EOS
33assert video_120s == 7786 and video_120s <= 8192
34assert 32 * (280 + 2) > 8192   # 32 frames at the image budget alone exceed the training length

A production loss also needs batched matrix operations, multiple hard negatives, MRL sub-losses and the listwise Plackett–Luce variant for ranked targets.

Per-layer embeddings without the table. Gemma 3n and Gemma 4 give each layer an extra input built from a projection of the token embeddings plus a dedicated lookup table indexed by token ID. Image and audio soft tokens have no token ID, so that table cannot help them, and it costs memory. EmbeddingGemma 2 keeps only the projection.

The recipe separates what the model sees from what it optimizes. Text comes first, then every modality under pure distillation, then a contrastive finetune that drops the teacher. The final checkpoint averages the end of finetuning with a much earlier checkpoint from the same run.

Why stop distilling? The teacher's space was not shaped for a 744M student or for 128-dimensional prefixes. When the paper continued distillation for roughly the same compute, MMEB-v2 stayed flat at 52.28; switching to the contrastive recipe reached 55.46. Freezing the Gemma 4 vision encoder instead of training it cost 4.1 points.

How many vision tokens? More tokens help visual documents, where small text matters, well past the training budget. Images flatten at 280. Video gets worse at 560 per frame, which the authors attribute to diluted attention and mean pooling over very long sequences.

How short can the vector be? Every suite keeps at least 95% of its score at 256 dimensions, a third of the storage. At 128, MMEB-v2 and spoken-query retrieval drop sharply. Native audio also beats transcribing speech first: even 256-dimensional native embeddings outscore full-width embeddings of ASR text. The paper attributes the gap to transcription errors from accents, homophones and audio artifacts, which native audio avoids.

Two more cautions. On English text the model regresses slightly against EmbeddingGemma 1 (MTEB English 68.46 versus 69.67) while improving on multilingual text and code. And its speech representations are dominated by meaning: zero-shot clustering by speaker gender scores 0.40, yet a linear probe recovers gender at 66.40. The information is present, but not the main axis of similarity.

A team stores 768-dimensional vectors for scanned invoices embedded at 280 vision tokens. To cut cost they propose truncating to 128 dimensions and embedding at 70 tokens. What do the paper's results predict?

Both changes hurt documents most. In the budget ablation, visual-document NDCG@5 fell from 63.38 at 280 tokens to 51.61 at 70, while 560 tokens raised it to 67.62. Truncating to 128 dimensions cut the MMEB-v2 visual-document average from 67.8 to 49.5. Truncating to 256 is the safer saving: 62.6 on visual documents for a third of the storage. Re-embed queries and documents with the same dimensions, compare them with cosine similarity, and measure on your own documents, because these are benchmark averages.

Source and scope

EmbeddingGemma 2: Unified and Efficient Multimodal Representations, arXiv:2610.37364, §2–4, Tables 1–11 and Figures 1–2. Architecture, objectives, recipe and all scores come from the paper; ablations use intermediate recipes. Figure 2 was digitized from the paper's vector graphic. Sequence-length totals, byte counts and the code are LiminalML derivations and teaching examples. We have not run the model or reproduced its benchmarks.