Diffusion models generate data by learning to reverse a gradual noising process. Training corrupts real samples with Gaussian noise over many steps and teaches a network to predict that noise; generation starts from pure noise and denoises step by step. Since roughly 2021 they have been the leading approach for image, video and audio synthesis, powering Stable Diffusion, DALL-E 2/3, Midjourney and Imagen.
The forward process adds a little Gaussian noise at each of T steps (classically T = 1000) following a variance schedule β1…βT. With αt = 1 − βt and ᾱt = ∏ αs, the noisy sample at any step has a closed form, so training never needs to simulate the chain.
The reverse process is learned. DDPM (Ho et al., 2020) showed that a simple objective works well: sample t and ε, form xt, and minimize ||ε − εθ(xt, t)||2. This is equivalent (up to weighting) to denoising score matching: the network estimates the gradient of the log density, ∇x log pt(x).
import torch, torch.nn as nn, torch.nn.functional as F
T = 1000
betas = torch.linspace(1e-4, 0.02, T)
alphas = 1 - betas
abar = torch.cumprod(alphas, 0)
class TinyEps(nn.Module): # stand-in for a U-Net; works on 2-D toy data
def __init__(self, d=2, h=128):
super().__init__()
self.net = nn.Sequential(nn.Linear(d + 1, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, d))
def forward(self, x, t):
return self.net(torch.cat([x, (t.float() / T).unsqueeze(1)], 1))
model = TinyEps()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
def data(n): # two-moons-like toy distribution
a = torch.rand(n) * 3.1416
s = torch.randint(0, 2, (n,)).float()
return torch.stack([torch.cos(a) + s, torch.sin(a) * (1 - 2 * s) + 0.5 * s], 1)
for step in range(5000): # training: predict the added noise
x0 = data(256)
t = torch.randint(0, T, (256,))
eps = torch.randn_like(x0)
xt = abar[t].sqrt().unsqueeze(1) * x0 + (1 - abar[t]).sqrt().unsqueeze(1) * eps
loss = F.mse_loss(model(xt, t), eps)
opt.zero_grad(); loss.backward(); opt.step()
@torch.no_grad()
def sample(n=500): # ancestral DDPM sampling
x = torch.randn(n, 2)
for t in reversed(range(T)):
tt = torch.full((n,), t)
eps = model(x, tt)
x = (x - betas[t] / (1 - abar[t]).sqrt() * eps) / alphas[t].sqrt()
if t > 0:
x += betas[t].sqrt() * torch.randn_like(x)
return xFor images, replace TinyEps with a U-Net with sinusoidal time embeddings, or use Hugging Face diffusers (UNet2DModel + DDPMScheduler).
Running diffusion on full-resolution pixels is expensive. Latent diffusion (Rombach et al., 2022) first trains a VAE that compresses images about 8× per side, then runs diffusion in that latent space. Text conditioning enters via cross-attention layers inside the denoiser that attend to text-encoder embeddings. Newer systems replace the U-Net with a Diffusion Transformer (DiT / MMDiT) operating on latent patches.
Classifier-free guidance trains one model with the condition randomly dropped (e.g. 10% of the time), then at sampling extrapolates from the unconditional prediction toward the conditional one. It is the main knob trading prompt fidelity for diversity, and it doubles the per-step cost.
Samplers determine how many steps are needed. DDIM makes the reverse process deterministic and allows 20–50 steps; DPM-Solver and similar ODE solvers do well at 10–25 steps. Distillation methods (progressive distillation, consistency models, adversarial diffusion distillation) cut this to 1–4 steps. Flow matching / rectified flow trains straighter trajectories between noise and data and is used in several recent models.
| Diffusion | GAN | Autoregressive | |
|---|---|---|---|
| Training stability | Stable (regression loss) | Fragile min-max | Stable (cross-entropy) |
| Mode coverage | Good | Prone to collapse | Good |
| Sampling cost | Many network evaluations (fewer with distillation) | One pass | One pass per token |
| Likelihood | Variational bound available | None | Exact |
The closed-form forward process is xt = √āt x0 + √(1 − āt) ε. With the original DDPM linear schedule (β rising from 10−4 to 0.02 over T = 1000 steps), the signal and noise coefficients work out to:
| Step t | āt | Signal √āt | Noise √(1−āt) | SNR ā/(1−ā) |
|---|---|---|---|---|
| 1 | 0.9999 | 1.00 | 0.01 | ~10,000 |
| 100 | 0.897 | 0.95 | 0.32 | 8.7 |
| 250 | 0.524 | 0.72 | 0.69 | 1.1 |
| 500 | 0.079 | 0.28 | 0.96 | 0.085 |
| 750 | 0.0034 | 0.06 | 1.00 | 0.0034 |
| 1000 | 0.00004 | 0.006 | 1.00 | 0.00004 |
Two things stand out. By step 500 the image is already mostly noise, so the second half of the schedule spends many steps in a regime where little structure remains — one reason the cosine schedule (Nichol and Dhariwal, 2021) keeps more signal for longer. And the final āT is small but not zero, so a little of the training image leaks into the "pure noise" endpoint; at inference the sampler starts from exactly pure noise, a mismatch that later work (Lin et al., 2024) corrected by rescaling schedules to zero terminal SNR.
Thinking in signal-to-noise ratio, rather than step index, makes schedules comparable: what the network learns at each step is how to denoise at a given SNR. Higher resolutions need more noise to destroy the same amount of structure, because neighbouring pixels are so correlated, which is why high-resolution models shift their schedules toward lower SNR.
With classifier-free guidance, every step runs the denoiser twice — once with the prompt and once with an empty prompt (or a negative prompt) — and extrapolates: ε̂ = εu + w(εc − εu). The sampler then converts that noise estimate into a slightly cleaner latent. In practice the two passes are batched together, so the cost is one forward pass at double batch size. After the final step, the VAE decoder maps the latent back to pixels.
Typical settings for Stable Diffusion-class models: 20–50 steps with a DPM-Solver++ or Euler-style sampler, and guidance around 5–9. Distilled models are designed for 1–4 steps and often for little or no guidance, so applying the usual settings to them produces poor images; check each model's card.
For a product, the most useful evaluation is usually a fixed prompt set drawn from real usage, rendered with fixed seeds after each change, and reviewed side by side.
The same machinery appears wherever a conditional distribution over continuous data is needed: inpainting and outpainting (condition on the unmasked region), super-resolution (condition on a low-resolution image), image editing and control (ControlNet adds edge maps, depth or pose as conditions), video generation (denoising spatio-temporal latents), audio and speech synthesis, and scientific uses such as protein structure generation and weather ensembles. In robotics, diffusion policies generate action sequences conditioned on observations. Practical image work with these models is covered in Stable Diffusion Web UI.