LFM3
Read Liquid AI's 24-layer tri-hybrid layer by layer, derive its 1.6 GB cache at 128K, and see how six models branch from one base.
Jump to a section
Six models on one tri-hybrid backbone
LFM3-1.7B holds 128K tokens of context in about 1.6 GB of 16-bit cache, because only 6 of its 24 layers keep every token. The other 18 keep a cache that stops growing after a few hundred positions or fewer.
LFM3 is Liquid AI's third generation of open-weight foundation models, released on 27 October 2026. It is six models of roughly 2B parameters on one shared backbone: a text model, vision, audio and omni models, and two decision models that answer typed questions in a single forward pass instead of generating text.
Liquid AI reports that the text model fits in about 1 GB at 4-bit precision, handles 256K tokens of context, and beats Qwen3.5-9B on the model card's unweighted benchmark average, 49.37 against 47.24. Those are the vendor's numbers. This breakdown explains the design that makes the memory claims add up.
One base, six models
22T pre-training tokens; the last 3.5T are curated mid-training that extends context to 256K. Width is proportional to tokens.
- 1SFT1.75T tokens of chat, reasoning, tool calls and agent traces
- 2Mixed RLMath, code, SQL and structured outputs, with a length penalty
- 3On-policy distillationSix RL specialists distilled into one model
- 4Agentic RLTraining inside real agent harnesses
Text post-training of LFM3-1.7B, in order.
Agents, tool calls, structured output
Adds a 164M vision encoder (8 gated 2D conv + 6 attention layers) and a 25M projector; each 2×2 block of 16×16 patches becomes one token
Causal audio encoder plus the LiquidCodec tokenizer for streaming speech
Vision encoder, causal audio encoder and LiquidCodec on one backbone
One forward pass, no generated tokens
One forward pass, no transcript or speech
d1-2B · one call“I was charged twice and my parcel never arrived. Please refund one charge.”
| Name | Type | Question | Answer |
|---|---|---|---|
| refund | noul | Is the customer asking for a refund? | P(yes) = 1.0 |
| team | choice | Which team should handle this? billing · shipping · technical | billing |
| topics | choice, multiple | Which topics does the message raise? billing · delivery · login | billing, delivery |
| urgency | score | How urgent is this ticket? low · medium · high | medium |
Training stages from Liquid AI’s release post; models, sizes and inputs from each Hugging Face card. The d1 example is the output printed in the d1-2B card’s quick start. A decision model reads the state and every named question in one forward pass and returns typed answers with probabilities instead of generating text. The card advises calibrating any threshold on your own data.
d1-2B model card · Quick startThree kinds of memory in one stack
The backbone is a tri-hybrid of 13 gated short convolutions, 5 sliding-window attention layers and 6 global attention layers. Each type keeps a different kind of memory.
A short convolution mixes the current token with the six before it, through gates computed from the input. It is cheap, and between decoding steps its state is a fixed seven positions per channel. Sliding-window attention can look up any of the last 256 tokens by content. Global attention can retrieve anything in the context, but its key/value cache grows with every token.
The order comes from the checkpoint's configuration. After two convolutions at the input, every other layer is attention: global at layer 2 and every fourth layer after it, sliding halfway between, with convolutions filling the gaps. Select a layer to see what it reads.
Twenty-four layers, three kinds of memory
- C · Gated short convolution × 13
- S · Sliding-window attention × 5
- G · Global attention × 6
Layer order from the checkpoint’s config.json. Bars show how many positions one layer reads directly when computing token number 4,096, on a log scale. Stacked layers pass information further: after several convolution layers, a token’s state reflects more than 7 positions, so “reads directly” is not the same as “can be influenced by”.
LFM3-1.7B config.json · layer_typesLFM2 already mixed convolutions with attention. According to the release post, a hardware-in-the-loop architecture search tested replacing some of those convolutions with other blocks, and sliding-window attention gave the best quality–speed trade-off. It adds mid-range lookups without adding a cache that grows with context.
Two smaller choices also matter. Each attention head's output is multiplied by a sigmoid gate computed from the layer input, so a head can switch itself off; Liquid AI reports that this suppresses activation outliers that hurt quantization. And every block uses sandwich norm: its input and its output are both normalised before the residual add.
Where 1.6 GB at 128K comes from
Every attention layer stores one key and one value vector for each of 4 key/value heads of 128 dimensions (16 query heads share them). At 16 bits that is
Only the 6 global layers keep every token: bytes per token, or GB at 128K tokens. The release post reports 1.6 GB. The 5 sliding layers stop growing at 256 tokens, at MB, and the 13 convolutions hold at most MB.
That is the saving the sliding layers buy. Had all 11 attention layers been global, the 128K cache would be 2.95 GB, nearly twice as large.
Only six layers keep growing
- LFM3-1.7B as configured
- Same 11 attention layers, all global
| Cache at 131,072 tokens | Grows with tokens? | Bytes |
|---|---|---|
| 6 global layers · K and V | Yes | 1.61 GB |
| 5 sliding layers · last 256 tokens | Only to 256 | 2.62 MB |
| 13 convolution layers · last 7 positions | No | ≤ 373 KB |
| Total as configured | 1.61 GB |
Our arithmetic from config.json: batch 1, 16-bit elements, 4 key/value heads of 128 dimensions, so 12,288 bytes per token across the global layers. At 128K that is 1.61 GB, matching the 1.6 GB in Liquid AI’s release post. Weights, activations, padding and allocator overhead are excluded. The dashed line is a counterfactual for comparison, not another checkpoint.
LFM3 release post · KV cache at 128KHalf of each global head ignores position
Rotary embeddings (RoPE) rotate each two-dimensional pair of a head by angle at position , with . Because rotations compose, a query at and a key at interact through , which depends only on their distance.
LFM3's global layers keep the 32 highest frequencies of that progression and set the other 32 to . For those pairs is the identity, so half of each global head compares queries and keys by content alone, at any distance. Sliding layers rotate all 64 pairs. The configuration also gives the two layer types different bases: for global layers and for sliding ones.
Global heads rotate only half their dimensions
- Sliding layers · θ = 10⁴, all 64 pairs
- Global layers · θ = 10⁶, 32 pairs rotate
Each point is one of a head’s 64 two-dimensional pairs, plotted at the number of tokens it takes to rotate once (2π/ω). On global layers the code keeps the first 32 frequencies of the full 128-dim progression and zeroes the rest, so those 64 dimensions score query–key matches without any positional rotation. On sliding layers, every pair rotates, though the slowest barely turn within a 256-token window.
modeling_lfm3.py · compute_rope_parametersThe short convolution in code
The gated short convolution is , where , and all come from one input projection. The example below checks the layer counts and cache arithmetic above, then shows that a seven-value buffer reproduces the full causal convolution. That equivalence is why the convolution's state stays constant while decoding.
1layers = ["conv", "conv", "full", "conv", "sliding", "conv", "full", "conv", "sliding", "conv", "full", "conv",
2 "sliding", "conv", "full", "conv", "sliding", "conv", "full", "conv", "sliding", "conv", "full", "conv"]
3assert (layers.count("conv"), layers.count("sliding"), layers.count("full")) == (13, 5, 6)
4
5kv_heads, head_dim, bytes_per = 4, 128, 2
6per_token = 2 * kv_heads * head_dim * bytes_per # K and V, one attention layer
7global_cache = layers.count("full") * per_token * 131_072
8sliding_cache = layers.count("sliding") * per_token * 256
9assert global_cache == 1_610_612_736 # 1.61 GB at 128K tokens
10assert sliding_cache == 2_621_440 # fixed once 256 tokens are cached
11
12def short_conv(xs, B, C, w):
13 """y_t = C_t * sum_j w_j * (B * x)_{t-j}, one channel, causal width len(w)."""
14 gated = [b * x for b, x in zip(B, xs)]
15 return [C[t] * sum(w[j] * gated[t - j] for j in range(len(w)) if t - j >= 0) for t in range(len(xs))]
16
17def short_conv_streaming(xs, B, C, w):
18 window, ys = [0.0] * len(w), []
19 for x, b, c in zip(xs, B, C):
20 window = [b * x] + window[:-1] # keep only the last 7 gated inputs
21 ys.append(c * sum(wj * v for wj, v in zip(w, window)))
22 return ys
23
24xs = [0.5, -1.0, 2.0, 0.25, 1.5, -0.5, 1.0, 3.0, -2.0, 0.75]
25B = [1.0, 0.5, 0.8, 1.2, 0.3, 1.0, 0.9, 0.6, 1.1, 0.7]
26C = [0.9, 1.0, 0.4, 0.7, 1.3, 0.8, 1.0, 0.5, 0.6, 1.2]
27w = [0.4, 0.2, 0.1, 0.1, 0.1, 0.05, 0.05] # width 7
28full, streamed = short_conv(xs, B, C, w), short_conv_streaming(xs, B, C, w)
29assert all(abs(a - b) < 1e-12 for a, b in zip(full, streamed))
30print(global_cache / 1e9, "GB of global K/V at 128K; conv state stays", len(w), "values per channel")In the real layer the weights are learned per channel across 2,048 channels, and an output projection follows. The figure shows both operators and where the 1.73B parameters sit: most of them are in the feed-forward layers, not the mixers.
Inside one layer
- SwiGLU feed-forward, all 24 layers1132.5M · 65.5%
- Token embedding (tied with output)262.1M · 15.2%
- Short-convolution operators, 13 layers218.3M · 12.6%
- Attention operators, 11 layers115.7M · 6.7%
- RMSNorm weights0.2M · 0.0%
From modeling_lfm3.py. Sandwich norm means each operator’s output is normalized again before it joins the residual stream. Parameter shares are our count from config.json; the 1.729B total matches the model card’s 1.73B. The feed-forward layers, not the mixers, hold most of the weights.
LFM3-1.7B modeling_lfm3.pyWhere 1.7B is enough, and where it is not
On the card's table, LFM3-1.7B scores above Qwen3.5-9B on every instruction-following row and on 7 of 11 agent and tool-use rows. It trails Qwen3.5-9B on knowledge (GPQA Diamond 63.37 against 80.40), agentic coding (SWE-bench Verified 43.40 against 51.20) and long-context retrieval (RULER 128K 65.56 against 87.11). The card itself recommends retrieval or search for knowledge-heavy tasks and advises against agentic coding and complex multi-step reasoning.
Where 1.7B parameters are enough
- LFM3-1.7B
- Qwen3.5-9B
LFM3-1.7B is higher on 14 of 29 rows. Unweighted average of every row: 49.37 versus 47.24.
Scores reported on the LFM3-1.7B model card, all served with vLLM in BF16; Liquid AI ran these evaluations, not us. AA-Omniscience is a net score that goes negative when wrong answers outnumber right ones, which is why the axis starts at −60. Averaging such different metrics is the card’s summary, not a measure of general ability.
LFM3-1.7B model card · BenchmarksThe decision models give up text altogether. A d1 model never writes, so there is no chain of thought or free-form explanation. In exchange, one pass returns every named answer with probabilities. The d1-2B card reports 5.7 ms per decision on an NVIDIA H100, warns that answers get less reliable past about 64K tokens of state, and recommends calibrating any threshold on your own data.
What a small cache does not make cheap
Long context is not free. Prefill still runs every layer over every token, and each global layer reads its whole cache for every decoded token. The architecture bounds the memory of 18 of 24 layers; it does not make the other 6 cheap.
The RULER gap has no tested cause. It is tempting to blame the six global layers for the long-context deficit, but the card does not test that, and training data and post-training also differ between the models.
Serving memory is more than the cache. Suppose you plan 64 concurrent sessions, each holding 32,768 tokens with a 16-bit cache. The global layers hold GB per session, and sliding and convolution state add about 3 MB, so 64 sessions need about 26 GB of cache. That excludes the 1.73B weights (about 3.5 GB in BF16), activations and temporary buffers, allocator and page granularity, and any cache quantization or prefix sharing your server applies. Measure with your serving stack before sizing hardware.
Sources and further reading
- LFM3-1.7B model card, with its config.json and modeling_lfm3.py: layer order, shapes and RoPE behaviour, read on 28 October 2026. Our parameter count from the config matches the card's 1.73B. Cache and parameter arithmetic, the code and the scenarios are LiminalML derivations; benchmarks are Liquid AI's, and we have not run the models.
- Liquid AI, LFM3 release post: the architecture search, cache figures and training stages.
- d1-2B model card, along with the LFM3-VL-2B, LFM3-Audio-2B and LFM3-Omni-2B model cards.
- Su et al., RoFormer (2021): rotary position embeddings.