A Generative Adversarial Network (GAN) trains two neural networks against each other: a generator that turns random noise into samples, and a discriminator that tries to tell those samples apart from real data. Introduced by Goodfellow et al. in 2014, GANs were the dominant approach to high-fidelity image synthesis until diffusion models overtook them around 2021–2022. They remain relevant wherever single-step, low-latency generation matters.
This page covers the minimax objective, the training loop, why training is unstable, the stabilizing tricks that matter in practice, a runnable PyTorch example, and the major variants: DCGAN, StyleGAN, and CycleGAN.
The generator G maps a latent vector z (usually 64–512 dimensions of Gaussian noise) to a sample in data space, such as a 64×64 RGB image. The discriminator D is a binary classifier that outputs the probability that its input came from the real dataset. Neither network is given an explicit density; the generator learns an implicit distribution pg purely from the feedback D provides.
Training alternates: one (or k) gradient steps on D using a mini-batch of real and fake samples, then one step on G where the gradient is backpropagated through a frozen D into G's weights. G improves only insofar as D's decision surface provides a useful slope.
The original value function is a two-player zero-sum game:
minG maxD V(D,G) = Ex~pdata[log D(x)] + Ez~pz[log(1 − D(G(z)))]
For a fixed G, the optimal discriminator is D*(x) = pdata(x) / (pdata(x) + pg(x)). Substituting it back shows that G is minimizing 2·JSD(pdata || pg) − log 4, the Jensen-Shannon divergence. The global optimum is pg = pdata, where D outputs 0.5 everywhere.
In practice G uses the non-saturating loss maxG E[log D(G(z))] instead of minimizing log(1 − D(G(z))). Both have the same fixed point, but the non-saturating form gives strong gradients early in training when D easily rejects fakes.
Other objectives worth knowing: WGAN replaces JSD with the Earth-Mover (Wasserstein-1) distance, using a 1-Lipschitz critic (weight clipping or a gradient penalty, WGAN-GP); LSGAN uses least-squares loss; hinge loss is common in large conditional GANs such as BigGAN.
Gradient descent on a min-max game is not guaranteed to converge; simultaneous updates can circle around the equilibrium. The three classic symptoms:
Stabilizers that hold up in practice: spectral normalization on D, R1 gradient penalty on real data, two time-scale update rule (TTUR: higher LR for D), Adam with β1 = 0 or 0.5, exponential moving average (EMA) of G's weights for evaluation, and differentiable data augmentation (DiffAugment, ADA) when data is limited.
import torch, torch.nn as nn
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
device = "cuda" if torch.cuda.is_available() else "cpu"
Z = 64
tfm = transforms.Compose([transforms.ToTensor(), transforms.Normalize([0.5], [0.5])])
loader = DataLoader(datasets.MNIST("data", train=True, download=True, transform=tfm),
batch_size=128, shuffle=True, drop_last=True)
G = nn.Sequential(
nn.Linear(Z, 256), nn.LeakyReLU(0.2),
nn.Linear(256, 512), nn.LeakyReLU(0.2),
nn.Linear(512, 784), nn.Tanh()).to(device)
D = nn.Sequential(
nn.Linear(784, 512), nn.LeakyReLU(0.2),
nn.Linear(512, 256), nn.LeakyReLU(0.2),
nn.Linear(256, 1)).to(device) # logits; BCEWithLogits is numerically stable
bce = nn.BCEWithLogitsLoss()
opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
for epoch in range(20):
for real, _ in loader:
real = real.view(real.size(0), -1).to(device)
b = real.size(0)
ones, zeros = torch.ones(b, 1, device=device), torch.zeros(b, 1, device=device)
# 1) Discriminator step: real -> 1, fake -> 0
fake = G(torch.randn(b, Z, device=device))
loss_d = bce(D(real), ones) + bce(D(fake.detach()), zeros)
opt_d.zero_grad(); loss_d.backward(); opt_d.step()
# 2) Generator step (non-saturating): make D call fakes real
loss_g = bce(D(fake), ones)
opt_g.zero_grad(); loss_g.backward(); opt_g.step()
print(f"epoch {epoch} D {loss_d.item():.3f} G {loss_g.item():.3f}")
with torch.no_grad():
samples = G(torch.randn(16, Z, device=device)).view(-1, 1, 28, 28)Note fake.detach() in the D step so G's graph is not updated twice, and that the G step reuses the same fake tensor with gradients still attached.
| Variant | Key idea | Typical use |
|---|---|---|
| DCGAN | All-convolutional G and D with BatchNorm and strided convs | Baseline image GAN, teaching |
| Conditional GAN | Feed class label or other conditioning to G and D | Class-controlled generation |
| WGAN / WGAN-GP | Wasserstein loss with Lipschitz critic | More stable gradients |
| Pix2Pix | Paired image-to-image with U-Net G and PatchGAN D | Paired translation tasks |
| CycleGAN | Unpaired translation via cycle-consistency | Style / domain transfer |
| StyleGAN | Mapping network + style modulation per layer | Photorealistic faces, latent editing |
| BigGAN | Large-batch class-conditional GAN, truncation trick | ImageNet-scale synthesis |
Strengths: one forward pass per sample (fast inference), sharp outputs, a compact and often well-structured latent space that supports interpolation and editing.
Weaknesses: no likelihood to evaluate, unstable and hyperparameter-sensitive training, poorer mode coverage than likelihood-based models, harder to scale to open-ended text-to-image tasks.
Compared with VAEs, GANs give sharper samples but no encoder or ELBO. Compared with diffusion models, GANs are much faster at sampling but cover the distribution less completely; modern distillation methods for diffusion often reintroduce an adversarial loss to get few-step sampling. Evaluation typically uses FID (Fréchet Inception Distance) and precision/recall metrics rather than the training losses.
BCEWithLogitsLoss, which causes NaNs when D saturates.