LFM3

LFM3 is Liquid AI's third generation of open-weight foundation models, released on 27 October 2026. It is six models of roughly 2B parameters that share one backbone: a text model, vision, audio and omni models, and two decision models that answer typed questions in a single forward pass instead of generating text.

The backbone is a tri-hybrid. LFM3-1.7B has 24 layers: 13 gated short convolutions, 5 sliding-window attention layers and 6 global attention layers. Liquid AI reports that it fits in about 1 GB at 4-bit precision, handles 256K tokens of context, and beats Qwen3.5-9B on the card's unweighted benchmark average (49.37 versus 47.24). Those are the vendor's numbers; this article explains the design that makes the memory claims add up.

Each layer type keeps a different kind of memory. A short convolution mixes the current token with the six before it, through gates computed from the input. It is cheap, and its state between decoding steps is a fixed seven positions per channel. Sliding-window attention can look up any of the last 256 tokens by content. Global attention can retrieve anything in the context, but its key/value cache grows with every token.

LFM2 already mixed convolutions with attention. According to the release post, a hardware-in-the-loop architecture search tested replacing some convolutions with other blocks, and sliding-window attention gave the best quality–speed trade-off. The model adds mid-range lookups without adding cache that grows with context.

Two smaller choices matter too. Each attention head's output is multiplied by a sigmoid gate computed from the layer input, so a head can switch itself off; Liquid AI reports that this suppresses activation outliers that hurt quantization. And every block uses sandwich norm: its input and its output are both normalized before the residual add. Select a layer to see what it reads.

Cache per token. Every attention layer stores one key and one value vector for each of 4 key/value heads of 128 dimensions. At 16 bits that is

Only the 6 global layers keep every token: $6\times2{,}048=12{,}288$ bytes per token, or $12{,}288\times131{,}072\approx1.61$ GB at 128K tokens. The release post reports 1.6 GB. The 5 sliding layers stop growing at 256 tokens, $5\times2{,}048\times256\approx2.6$ MB, and the 13 convolutions hold at most $13\times2{,}048\times7\times2\approx0.37$ MB. Had all 11 attention layers been global, the 128K cache would be 2.95 GB.

Partial rotary embeddings. RoPE rotates each two-dimensional pair $k$ of a head by angle $m\,\omega_k$ at position $m$, with $\omega_k=\theta^{-2k/128}$. Because rotations compose, a query at $m$ and a key at $n$ interact through $R((n-m)\omega_k)$, which depends only on their distance. LFM3's global layers keep the 32 highest frequencies of that progression and set the other 32 to $\omega_k=0$. For those pairs $R(0)$ is the identity, so half of each global head compares queries and keys by content alone, at any distance. Sliding layers rotate all 64 pairs.

The gated short convolution is $y_t=C_t\odot\sum_{j=0}^{6}w_j\odot(B\odot x)_{t-j}$, where $B$, $C$ and $x$ come from one input projection. This executable example checks the layer counts and cache arithmetic above, then shows that a seven-value buffer reproduces the full causal convolution, which is why the convolution state stays constant while decoding.

1layers = ["conv", "conv", "full", "conv", "sliding", "conv", "full", "conv", "sliding", "conv", "full", "conv",
2          "sliding", "conv", "full", "conv", "sliding", "conv", "full", "conv", "sliding", "conv", "full", "conv"]
3assert (layers.count("conv"), layers.count("sliding"), layers.count("full")) == (13, 5, 6)
4
5kv_heads, head_dim, bytes_per = 4, 128, 2
6per_token = 2 * kv_heads * head_dim * bytes_per          # K and V, one attention layer
7global_cache = layers.count("full") * per_token * 131_072
8sliding_cache = layers.count("sliding") * per_token * 256
9assert global_cache == 1_610_612_736                      # 1.61 GB at 128K tokens
10assert sliding_cache == 2_621_440                         # fixed once 256 tokens are cached
11
12def short_conv(xs, B, C, w):
13    """y_t = C_t * sum_j w_j * (B * x)_{t-j}, one channel, causal width len(w)."""
14    gated = [b * x for b, x in zip(B, xs)]
15    return [C[t] * sum(w[j] * gated[t - j] for j in range(len(w)) if t - j >= 0) for t in range(len(xs))]
16
17def short_conv_streaming(xs, B, C, w):
18    window, ys = [0.0] * len(w), []
19    for x, b, c in zip(xs, B, C):
20        window = [b * x] + window[:-1]                       # keep only the last 7 gated inputs
21        ys.append(c * sum(wj * v for wj, v in zip(w, window)))
22    return ys
23
24xs = [0.5, -1.0, 2.0, 0.25, 1.5, -0.5, 1.0, 3.0, -2.0, 0.75]
25B = [1.0, 0.5, 0.8, 1.2, 0.3, 1.0, 0.9, 0.6, 1.1, 0.7]
26C = [0.9, 1.0, 0.4, 0.7, 1.3, 0.8, 1.0, 0.5, 0.6, 1.2]
27w = [0.4, 0.2, 0.1, 0.1, 0.1, 0.05, 0.05]                 # width 7
28full, streamed = short_conv(xs, B, C, w), short_conv_streaming(xs, B, C, w)
29assert all(abs(a - b) < 1e-12 for a, b in zip(full, streamed))
30print(global_cache / 1e9, "GB of global K/V at 128K; conv state stays", len(w), "values per channel")

In the real layer the weights are learned per channel across 2,048 channels, and an output projection follows. The figure shows both operators and where the 1.73B parameters sit.

Where is 1.7B enough? On the card's table, LFM3-1.7B scores above Qwen3.5-9B on every instruction-following row and on 7 of 11 agent and tool-use rows. It trails Qwen3.5-9B on knowledge (GPQA Diamond 63.37 versus 80.40), agentic coding (SWE-bench Verified 43.40 versus 51.20) and long-context retrieval (RULER 128K 65.56 versus 87.11). The card itself recommends retrieval or search for knowledge-heavy tasks and advises against agentic coding and complex multi-step reasoning.

Does a small cache make long context free? No. Prefill still runs every layer over every token, and each global layer's attention reads the whole cache when decoding. The architecture bounds the memory of 18 of 24 layers; it does not make the other 6 cheap. Do not read the RULER gap as caused by the six global layers either: the card does not test that, and training data and post-training also differ.

What does a decision model give up? d1 models never write text, so there is no chain of thought or free-form explanation. In exchange, one pass returns every named answer with probabilities. The d1-2B card reports 5.7 ms per decision on an NVIDIA H100, warns that answers get less reliable past about 64K tokens of state, and recommends calibrating any threshold on your own data.

You plan to serve 64 concurrent LFM3-1.7B sessions, each holding 32,768 tokens of context, with a 16-bit cache. Roughly how much cache memory do you need, and what is missing from the estimate?

The global layers hold $12{,}288\times32{,}768\approx0.40$ GB per session; sliding and convolution state add about 3 MB. Across 64 sessions that is about 25.9 GB. The estimate excludes the 1.73B weights (about 3.5 GB in BF16), activations and temporary buffers, allocator and page granularity, and any cache quantization or prefix sharing your server applies. Measure with your serving stack before sizing hardware.

Source and scope

LFM3-1.7B model card, its config.json and modeling_lfm3.py; the LFM3 release post; and the d1-2B, LFM3-VL-2B, LFM3-Audio-2B and LFM3-Omni-2B model cards, read on 28 October 2026. Layer order, shapes and RoPE behavior come from the checkpoint files. Cache and parameter arithmetic, the code and the scenarios are LiminalML derivations; our parameter count from the config matches the card's 1.73B. Benchmarks are Liquid AI's. We have not run the models.