Mistral Large 4

Mistral Large 4 is Mistral AI's largest model, nicknamed "Le Chonk". Its open weights were released on 29 October 2026, after an API preview earlier that month. It reads text and images, writes text, accepts 1,048,576 tokens of context, and reasons at a chosen effort: low, medium or high. Mistral says it was trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in its own European datacenters. A technical report has been announced but not yet published. This breakdown is therefore built from the checkpoint itself: its configuration, the shapes and some learned values in its weights, and the Transformers implementation.

The model is sparse in two ways. In 71 of its 72 layers, each token passes through 4 of 128 routed experts plus one shared expert. A token therefore uses 49.8B of the 1,051B stored parameters, or 52.3B counting embeddings and the output head. Separately, each attention layer reads at most 2,048 earlier positions, chosen by a small indexer, from a compressed cache. On top of both, layers reuse each other's work: some attend over another layer's cache, and others reuse another layer's choice of positions.

Before it attends, an attention layer has two jobs: decide what to keep about each past position, and decide which positions to look at.

What to keep. Multi-head Latent Attention (MLA, from DeepSeek-V2) compresses each position into a 512-value latent, plus a 64-value rotary key shared by all 64 heads. Each head rebuilds its own keys and values from the latent through a learned up-projection. The cache therefore holds 576 values per position per layer rather than 64 heads × 256.

Where to look. DeepSeek Sparse Attention (DSA, introduced with DeepSeek-V3.2) adds an indexer: a cheap scorer with 32 small heads that rates every earlier position and keeps the best 2,048. All 64 attention heads then attend only to those positions.

Mistral builds on that recipe and changes several things:

• Queries come from a direct projection, without low-rank compression.
• Every fourth layer has no rotary position encoding at all (NoPE).
• From layer 39 onward, every fourth layer reuses layer 35's latent instead of computing its own.
• 28 layers reuse the selection made by the most recent layer that ran its indexer.
• Each head rescales its query by a learned amount that depends on position.
• The rotary frequencies get learned multipliers.

After attention, a router sends the token to 4 experts.

The three kinds of reuse are independent of each other. Every layer that shares a cache is a NoPE layer, and so is its source, layer 35. The implementation keeps separate shared slots for rotated and unrotated caches, so a layer never reads a cache built with a different positional scheme. Reuse saves memory and compute, but a reusing layer cannot choose for itself. Mistral has not published ablations, so the quality cost is unknown. The map below shows all 72 layers; select one to see what it computes and what it borrows.

Cache. A layer with its own latent stores $\mathbf c_t\in\mathbb R^{512}$ for each position $t$. A RoPE layer also stores a rotary key $\mathbf k^R_t\in\mathbb R^{64}$. Across the model, 54 RoPE layers and 9 NoPE layers own a latent:

Without sharing, the nine reusing layers would add 4,608 values. Full per-head keys and values would cost $72\times64\times(128+128)=1{,}179{,}648$ values per position, 33 times more. At BF16 and $2^{20}$ positions, the latent cache takes $35{,}712\times2\times2^{20}\approx74.9$ GB.

Sparse attention. For the query at position $t$, the indexer scores every earlier position $s$. It keeps the top 2,048, and attention normalizes over those alone:

The indexer's cost still grows with $t$, but each scored position costs only 32 × 128 multiply-adds, against 64 × (512 + 64 + 512) for each position a head actually attends to. Across the model, one new token costs about $4.94\times10^6$ multiply-adds per attended position and $1.80\times10^5$ per scored position. Attention is capped at 2,048 positions, so the indexer becomes the larger cost beyond $4.94\times10^6\times2048/1.80\times10^5\approx56{,}000$ tokens. At $2^{20}$ tokens the sparse layer still does 26 times less work than dense MLA.

Query scaling. Each head $h$ multiplies its query by a learned function of position before the usual dot product:

Why this helps: suppose one key beats $n-1$ competitors by a logit gap $\Delta$. Its attention weight is $1/\big(1+(n-1)e^{-\Delta}\big)$, which shrinks as $n$ grows. If the logits are multiplied by $c\ln n$, the competitors' total becomes $(n-1)\,n^{-c\Delta}$, which stays bounded when $c\Delta\ge1$. This is the idea of Scalable-Softmax, and the configuration names it ssmax. Here $s_h$ and $b_h$ are learned, so a head can also choose to flatten with distance.

Positions. RoPE rotates dimension pair $i$ at angular frequency $\omega_i=10000^{-2i/64}$. YaRN is configured with an original window of 512 and a factor of 2,048 ($512\times2048=2^{20}$). It keeps pairs that complete at least 32 turns within 512 positions, divides by 2,048 the pairs that complete less than one, and blends linearly in between. LeRoPE then multiplies each frequency by a learned $e^{\alpha_i}$:

This executable Python example rebuilds the layer schedule from the configuration rules and checks the cache and work arithmetic. It also implements the router and the YaRN frequencies.

1import math
2
3LAYERS = 72
4rope = [(l + 1) % 4 != 0 for l in range(LAYERS)]                        # no_rope_layers
5kv_source = [35 if l >= 39 and (l + 1) % 4 == 0 else l for l in range(LAYERS)]
6indexer = "F" * 17 + "SSFF" * 13 + "SSF"                                 # F runs, S reuses
7assert len(indexer) == LAYERS and indexer.count("F") == 44
8
9# Cache per position: a 512-value latent, plus a 64-value rotary key in RoPE layers.
10latent = sum(512 + (64 if rope[l] else 0) for l in range(LAYERS) if kv_source[l] == l)
11full_kv = LAYERS * 64 * (128 + 128)
12assert latent == 35_712 and all(not rope[l] for l in range(LAYERS) if kv_source[l] != l)
13print(latent, "cached values per position,", round(full_kv / latent), "x fewer than full K/V")
14
15# Multiply-adds per new token: attend to at most 2,048 positions, but score all of them.
16attend = sum(64 * (512 + (64 if rope[l] else 0) + 512) for l in range(LAYERS))
17score = indexer.count("F") * 32 * 128
18def work(t):
19    return attend * t, attend * min(t, 2048) + score * t
20print("indexer outweighs attention beyond", round(attend * 2048 / score), "tokens")
21dense, sparse = work(2 ** 20)
22print(f"at 1M tokens: {dense / sparse:.0f}x fewer multiply-adds than dense MLA")
23
24# Router: softmax over 128 experts, keep the top 4, renormalize their weights.
25def route(logits, k=4):
26    top = max(logits)
27    p = [math.exp(x - top) for x in logits]
28    chosen = sorted(range(len(p)), key=p.__getitem__, reverse=True)[:k]
29    return {e: p[e] / sum(p[i] for i in chosen) for e in chosen}
30weights = route([math.sin(7 * e) for e in range(128)])
31assert len(weights) == 4 and abs(sum(weights.values()) - 1) < 1e-12
32
33# YaRN wavelengths: original window 512, factor 2,048, beta 32 and 1.
34def yarn_wavelengths(dims=64, theta=1e4, window=512, factor=2048, fast=32, slow=1):
35    corr = lambda r: dims * math.log(window / (r * 2 * math.pi)) / (2 * math.log(theta))
36    lo, hi = math.floor(corr(fast)), math.ceil(corr(slow))
37    waves = []
38    for i in range(dims // 2):
39        w = theta ** (-2 * i / dims)
40        keep = 1 - min(max((i - lo) / (hi - lo), 0), 1)
41        waves.append(2 * math.pi / (w * keep + w / factor * (1 - keep)))
42    return waves
43waves = yarn_wavelengths()
44assert waves[3] < 16 and waves[16] > 2 ** 20
45print(f"frequency 16: {waves[16]:,.0f} positions per turn, {waves[16] / math.exp(0.9336):,.0f} after LeRoPE")

The weights explain the hardware bill. Every expert must be resident, even though each token reads only four of them. The checkpoint stores FP8 weights in 128 × 128 blocks with one scale each, and keeps the vision tower, output head, latent projections and parts of the indexer in BF16. The final logits are soft-capped at $30\tanh(z/30)$.

When does sparse attention pay? Not for short prompts. Below 2,048 tokens every position is attended anyway, and the indexer adds about 4% to the attention work. The gain grows with context: 3.5× at 8K tokens, 19× at 128K and 26× at 1M. Reusing selections in 28 layers removes 39% of the indexer's work, and that matters because beyond about 56K tokens the indexer is the dominant cost.

What does the drafter buy? The checkpoint ships a DSpark draft head for speculative decoding. The drafter proposes three tokens, and the full model verifies them in one pass, so the output is the same as without drafting. Generation speeds up when the drafts are accepted.

How good is it? Mistral's own comparisons place it at or near the top of open-weight models on agentic coding. It uses far fewer tokens at high effort, refuses harmful cyber requests more often, and resists more attacks on Lakera's B3 benchmark. It trails the best listed open models on the Artificial Analysis Cyber and Intelligence indices, on finance (Finch) and on web browsing (BrowseComp). Reasoning effort is the main cost lever. Medium effort scores 13 points lower on the coding index but uses 32% fewer tokens.

What does it take to serve? The checkpoint is 1.06 TB. Eight H200s hold 1,128 GB, which leaves about 67 GB for cache and activations. That is enough for Hugging Face's 8 × H200 recipe of 131K context and 16 concurrent requests. An NVFP4 checkpoint targets Blackwell GPUs. Hugging Face reports that a 2-bit GGUF runs on one 512 GB Mac at about 15 tokens per second. The licence allows commercial use, but offering the model as a hosted service requires a separate licence from Mistral.

You want to serve a single 1M-token session on eight 192 GB GPUs, 1,536 GB in total. Is memory the limit? Which part of the per-token cost grows with the context? What would you test before trusting answers that depend on text 900K tokens back?

Memory is not the limit. The weights take 1,060.5 GB, leaving about 475 GB. At 1M tokens the BF16 latent cache needs 74.9 GB and the FP8 indexer keys 5.9 GB, so the cache would fit several times over. The real ceiling is the positional range the model was configured for: 1,048,576 positions. Per new token, attention reads a fixed 2,048 positions per layer. The cost that grows is the indexer's scoring, which at 1M is 95% of attention-related work. Prefill is heavier still. Scoring a 1M-token prompt takes about $1.8\times10^5\times(2^{20})^2/2\approx10^{17}$ multiply-adds, roughly twice the work of all the weight matrices over the same prompt.

For quality, a fact 900K tokens back helps only if it wins a place in the 2,048 positions chosen in the layers that need it. In 28 layers that choice is inherited from an earlier layer. Mistral has published no long-context results, so measure retrieval and multi-step use of distant facts at your own lengths before relying on them.

Source and scope

Sources:

• Mistral Large 4 model card, with its config.json and params.json.
• Tensor shapes from all 54 safetensors headers, and the BF16 values of lerope.alpha and ssmax_scaling, read from the checkpoint on 29 October 2026.
• The Transformers implementation and model documentation.
• vLLM's DSpark speculator.
• Mistral's release post. Its chart data was read from the page.
• Hugging Face's release post.

Mistral's technical report was not available when this was written. Parameter totals, cache sizes, multiply-add counts, wavelengths, the code and the scenario are LiminalML derivations. Benchmarks are as reported by Mistral and Artificial Analysis. We have not run the model.