Diffusion Models

A diffusion model learns to undo a controlled corruption process. Training examples start from real data and add noise at known levels. A network receives the corrupted input and a description of its noise level, then learns a useful prediction such as the added noise, the clean signal, or a velocity parameterization.

Generation is a different procedure: start from noise and use the trained model inside a sequence of reverse updates. Knowing how noise was added does not give us the noise in a new generated sample. Learning that conditional structure is the difficult part.

Think about one noisy observation rather than a finished image. At a high signal level, the observation retains much of the original structure. At a low signal level, many clean signals could plausibly have produced it. The model must use patterns learned from the training distribution, and possibly conditioning such as text, to make a useful prediction.

The worked example below deliberately keeps the original signal and noise available. It lets you verify the algebra and then introduce an error into the noise estimate. It does not train a network or claim to generate images.

Notice three things as you move the controls:

• Keeping the noise draw fixed isolates the effect of the signal level.
• A perfect noise estimate recovers the original signal wherever the signal coefficient is nonzero.
• The same prediction error is amplified more strongly when very little signal remains.

For the variance-preserving forward process, a noisy sample can be drawn directly from the clean sample:

Here $\bar\alpha_t = \prod_{s=1}^{t}(1-\beta_s)$ is a cumulative signal-retention coefficient. The slider controls that coefficient directly; it does not implement a particular timestep schedule.

Given a noise estimate $\hat\epsilon$, rearranging gives an estimate of the clean signal:

If $\hat\epsilon=\epsilon+\delta$, then

At $\bar\alpha=0.5$, the amplification factor is 1. At $\bar\alpha=0.02$, it is 7. This identity explains why a small error in the predicted noise can cause a much larger error in a direct clean-signal estimate at high noise. A complete reverse sampler uses additional schedule-dependent coefficients and, depending on the method, stochastic noise; repeatedly applying this identity alone is not DDPM sampling.

The following dependency-free example checks one corruption and reconstruction. It uses known noise so the expected answer is inspectable.

1from math import sqrt
2
3
4def corrupt(clean, noise, alpha_bar):
5    return [sqrt(alpha_bar) * x + sqrt(1 - alpha_bar) * e
6            for x, e in zip(clean, noise)]
7
8
9def estimate_clean(noisy, predicted_noise, alpha_bar):
10    if not 0 < alpha_bar <= 1:
11        raise ValueError("alpha_bar must be in (0, 1]")
12    return [(z - sqrt(1 - alpha_bar) * e) / sqrt(alpha_bar)
13            for z, e in zip(noisy, predicted_noise)]
14
15
16clean, noise = [1.0, -1.0], [0.25, -0.5]
17noisy = corrupt(clean, noise, 0.5)
18recovered = estimate_clean(noisy, noise, 0.5)
19assert all(abs(a - b) < 1e-12 for a, b in zip(clean, recovered))

A noise-prediction training loop instead samples real examples and timesteps, creates noisy observations, runs a neural network on those observations and timesteps, and minimizes a prediction loss. The known noise is the training target. It must not be secretly supplied to the model as an input.

Pixel space or latent space? Latent diffusion runs the generative process in a learned compressed representation. An encoder and decoder connect that representation to images. The smaller state can reduce computation, but the autoencoder introduces its own reconstruction limits.

More sampling steps or a better sampler? Step count alone does not specify quality or cost. The model, noise schedule, numerical method, guidance, and hardware all matter. Compare matched settings rather than assuming a universal quality curve.

Why can guidance change the image? Conditioning changes the predicted direction. Stronger guidance can improve agreement with the condition while also changing diversity and introducing artifacts. Our algebraic example contains no conditioning or guidance network.

Is an x₀ estimate the final generated image? Not generally. It can be an intermediate quantity inside a sampler. The assumptions of the sampler determine how it is used.

At $\bar\alpha=0.2$, a scalar noise-prediction error is $+0.1$. What is the resulting error in the direct clean-signal estimate?

The amplification is $\sqrt{0.8/0.2}=2$, and the sign is negative: the clean estimate is off by $-0.2$. Explain why this calculation does not prove that a trained denoiser can recover a unique original image from arbitrary noise.

Sources and further reading

• Ho et al., Denoising Diffusion Probabilistic Models: forward process, training objective, and reverse sampler.
• Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models: diffusion in a learned latent representation.