Nemotron 3.5 Transcribe

Nemotron 3.5 Transcribe is NVIDIA's open-weights speech LLM, released with a technical report on 28 October 2026. It is a speech-augmented language model: an audio encoder turns sound into embeddings, and those embeddings are placed inside the prompt of a decoder-only LLM, which writes the answer as text.

The point of using an LLM decoder is that transcription becomes promptable. One checkpoint produces verbatim or written-form transcripts, uses supplied vocabulary, labels who spoke, tags laughter and breaths, translates, and identifies the language, in 40 languages and for recordings up to an hour. The instruction alone selects the task. The model has 34B parameters, of which about 4.3B are active per token.

On the public section of the Open ASR Leaderboard its mean word error rate is 4.71%, the best among open-weights models when the report was written. Some commercial systems are still more accurate.

The encoder has two branches that run side by side on 30-second chunks of audio. A trainable ASR encoder, a 32-layer Transformer, turns 128-bin log-mel features into one 1,280-dimensional frame every 80 ms. A frozen diarizer estimates, for the same frames, which of eight speaker slots is active. The diarizer's answer is folded into the ASR frames as a fixed code, so each frame reaches the LLM already labelled with who was speaking. The LLM learns to turn those codes into <spk:0> to <spk:7> tags without any change to its architecture or sequence length.

A single linear layer maps 1,280 to 2,688 dimensions, the LLM width, and the frames replace an <|audio|> placeholder in the prompt: 12.5 positions for every second of audio. The decoder is a hybrid of Mamba-2 state-space layers, mixture-of-experts feed-forward layers and only six attention layers, which keeps memory small for hour-long prompts.

Two extra parts sit around the decoder. A shared multi-token prediction layer learns to guess the next two tokens after the one being generated, and serves as a draft model for speculative decoding. After generation, a 50M-parameter CTC adapter aligns the finished transcript with the ASR frames to produce word timestamps; it never changes the text. Choose a task to see the prompt change while the model stays the same.

Speaker fusion. Let $\mathbf h_t\in\mathbb R^{1280}$ be the ASR frame at time $t$ and $p_{t,s}$ the diarizer's activity for slot $s\in\{0,\dots,7\}$. The released implementation computes

where $K\in\mathbb R^{8\times1280}$ is fixed: $K_{s,2i}=\sin(s\,\omega_i)$ and $K_{s,2i+1}=\cos(s\,\omega_i)$ with $\omega_i=10000^{-2i/1280}$. These are the sinusoidal position codes of the original Transformer, indexed by speaker slot instead of position. Since $\sin a\sin b+\cos a\cos b=\cos(a-b)$,

so the similarity of two slot codes depends only on how far apart their indices are. The LayerNorm over eight slots makes one active speaker, two overlapping speakers and silence produce different weightings; silence produces exactly zero.

Length. At 80 ms per frame, one hour is $3600/0.08=45{,}000$ positions, or 120 chunks of 375. Attention inside each chunk costs $120\times375^2\approx1.7\times10^7$ query–key pairs, against $45{,}000^2\approx2.0\times10^9$ for one window over the whole hour: 120 times fewer. In the decoder, each attention layer stores keys and values for 2 heads of 128 dimensions, so the six attention layers hold $6\times2\times2\times128\times2=6{,}144$ bytes per position at BF16, about 276 MB for an hour of audio. The 23 Mamba-2 layers keep a fixed state of about 48 MB whatever the length.

This executable Python example builds the speaker kernel and fusion exactly as described, then checks the length arithmetic. LayerNorm is shown without its learned scale and shift, which are not published.

1import math
2
3def kernel(speakers=8, width=1280):
4    """Fixed sinusoidal speaker kernel: sin at even columns, cos at odd, row = speaker slot."""
5    return [[math.cos(s * math.exp(-(j - j % 2) * math.log(10000) / width)) if j % 2
6             else math.sin(s * math.exp(-(j - j % 2) * math.log(10000) / width))
7             for j in range(width)] for s in range(speakers)]
8
9def layer_norm(x, eps=1e-5):
10    mean = sum(x) / len(x)
11    var = sum((v - mean) ** 2 for v in x) / len(x)
12    return [(v - mean) / math.sqrt(var + eps) for v in x]
13
14def infusion(probs, K, threshold=0.5):
15    weights = layer_norm([1.0 if p > threshold else 0.0 for p in probs])
16    return [sum(w * K[s][j] for s, w in enumerate(weights)) for j in range(len(K[0]))]
17
18K = kernel()
19dot = lambda a, b: sum(x * y for x, y in zip(a, b))
20assert abs(dot(K[0], K[2]) - dot(K[3], K[5])) < 1e-6   # similarity depends on slot difference only
21assert abs(dot(K[1], K[1]) - 640) < 1e-6                 # every row has the same length
22
23assert max(abs(v) for v in infusion([0.1] * 8, K)) == 0  # silence adds nothing
24solo = infusion([0.9, 0.2, 0, 0, 0, 0, 0, 0], K)
25overlap = infusion([0.9, 0.8, 0, 0, 0, 0, 0, 0], K)
26assert max(abs(a - b) for a, b in zip(solo, overlap)) > 1  # overlap is a different code
27
28frames = 3600 * 1000 // 80                 # one hour of 80 ms encoder frames
29assert frames == 45_000 == 120 * 375       # 120 chunks of 30 s
30print(frames ** 2 // (120 * 375 ** 2), "x fewer encoder attention pairs with 30 s chunks")
31kv_bytes = 6 * 2 * 2 * 128 * 2 * frames    # attention layers, K and V, KV heads, head dim, BF16
32print(kv_bytes / 1e6, "MB of audio K/V in the six attention layers")

Training has no separate alignment stage with a frozen LLM: every part except the diarizer trains together from the first step. Each example gets a randomly chosen task instruction, so the same recording can teach verbatim transcription one time and contextual transcription the next. The multi-token prediction loss is added with weight 0.1. A short reinforcement-learning stage then rewards $1-\text{WER}$ directly.

What does speculative decoding buy? The MTP layer drafts tokens that the full model verifies in one pass, so accepted drafts skip sequential decoding steps. On a single 30-second request, MTP cuts latency 1.4–1.5× and the separately trained DFlash2 drafter 2.0–2.5×. Raising concurrency trades latency for throughput: with NVFP4 weights and DFlash2, one GPU transcribes about 2,238 seconds of audio per second at 256 concurrent requests. Drafts are accepted less often on long recordings, so the gains narrow at 10 minutes.

What did optimizing for transcription cost? The RL reward only scored transcripts, and audio reasoning on MMAR fell from 67.01 to 59.18. The report states that preserving the LLM's text-only abilities was not a goal: treat this as a transcription model, not a general assistant.

Where is it weaker? Granite-Speech is more accurate on meeting audio (AMI), and Qwen3-ASR and Granite-Speech on degraded audio. On full-length long-form recordings it leads the open models with 10.98% WER, but generation loops hit 7 of 451 recordings; without the released loop guardrail the mean would be 12.87%. The checkpoint is 67.5 GB in BF16, about 35 GB in FP8 and about 21 GB in NVFP4.

You need speaker-attributed transcripts of 50-minute meetings with up to ten participants. What does the design let you expect, and what should you test first?

Fifty minutes is within the one-hour recordings seen in training, and the 30-second chunking does not cap length. Speaker labels are the risk. The diarizer and the output both have eight speaker slots, <spk:0> to <spk:7>, so ten participants cannot all receive distinct tags. Slot indices are also local to each 30-second chunk, leaving the LLM to keep labels consistent across chunks, and meetings are where other open models beat it on plain WER. Measure concatenated minimum-permutation WER (cpWER) on your own recordings before relying on the labels.

Source and scope

Nemotron 3.5 Transcribe Technical Report, §2–4, Tables 1, 5–7 and 18, Figures 1–4 and 8; the model card and config.json; and the speaker fusion in vLLM's implementation, read on 28 October 2026. Figure 8 was digitized from the report's vector graphic. Cache, state and attention-pair arithmetic, the code and the scenario are LiminalML derivations. Benchmarks are NVIDIA's. We have not run the model.