Autoregressive generative models produce data one element at a time, each conditioned on everything generated so far. GPT-style large language models are the best-known example: a decoder-only transformer trained to predict the next token. The same recipe has been applied to images (PixelCNN, Image GPT, DALL-E 1, Parti), audio (WaveNet) and code.
Any joint distribution can be written exactly as a product of conditionals: p(x1, …, xT) = ∏t p(xt | x<t). An autoregressive model parameterizes each conditional with a neural network and trains by maximum likelihood, i.e. minimizing the cross-entropy between predicted and actual next tokens. Unlike GANs or diffusion, the likelihood is exact, which makes perplexity (exp of mean loss) a direct quality metric.
Text is tokenized (usually BPE or unigram subwords) and embedded. Each block applies causally masked multi-head self-attention, then an MLP, with residual connections and normalization (pre-norm with RMSNorm is common now). Positions come from learned embeddings or rotary position embeddings (RoPE). Because the mask hides future tokens, a single forward pass yields a loss at every position, so training is fully parallel even though generation is sequential.
import torch, torch.nn as nn, torch.nn.functional as F
class Block(nn.Module):
def __init__(self, d, h):
super().__init__()
self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
self.attn = nn.MultiheadAttention(d, h, batch_first=True)
self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
def forward(self, x):
T = x.size(1)
mask = torch.triu(torch.ones(T, T, dtype=torch.bool, device=x.device), 1) # True = blocked
a = self.ln1(x)
x = x + self.attn(a, a, a, attn_mask=mask, need_weights=False)[0]
return x + self.mlp(self.ln2(x))
class TinyGPT(nn.Module):
def __init__(self, vocab, d=128, h=4, n=4, ctx=128):
super().__init__()
self.ctx = ctx
self.tok, self.pos = nn.Embedding(vocab, d), nn.Embedding(ctx, d)
self.blocks = nn.Sequential(*[Block(d, h) for _ in range(n)])
self.ln, self.head = nn.LayerNorm(d), nn.Linear(d, vocab, bias=False)
def forward(self, idx):
pos = torch.arange(idx.size(1), device=idx.device)
return self.head(self.ln(self.blocks(self.tok(idx) + self.pos(pos))))
text = open("input.txt").read() # any plain-text corpus
chars = sorted(set(text)); stoi = {c: i for i, c in enumerate(chars)}
data = torch.tensor([stoi[c] for c in text])
model = TinyGPT(len(chars)); opt = torch.optim.AdamW(model.parameters(), lr=3e-4)
for step in range(3000):
ix = torch.randint(len(data) - 129, (32,))
x = torch.stack([data[i:i + 128] for i in ix]); y = torch.stack([data[i + 1:i + 129] for i in ix])
loss = F.cross_entropy(model(x).view(-1, len(chars)), y.view(-1))
opt.zero_grad(); loss.backward(); opt.step()
@torch.no_grad()
def generate(idx, n, temperature=0.8, top_k=20):
for _ in range(n):
logits = model(idx[:, -model.ctx:])[:, -1] / temperature
v, _ = torch.topk(logits, top_k); logits[logits < v[:, [-1]]] = -float("inf")
idx = torch.cat([idx, torch.multinomial(F.softmax(logits, -1), 1)], 1)
return "".join(chars[i] for i in idx[0].tolist())
print(generate(torch.zeros(1, 1, dtype=torch.long), 300))Generation cost grows with output length: each token needs a forward pass. The KV cache stores keys and values from previous positions so each step only computes the new token, turning quadratic recomputation into linear work per step, at the cost of memory proportional to context length. Speculative decoding uses a small draft model to propose several tokens that the large model verifies in one pass.
Take the four-token sequence "the cat sat down". Training feeds "the cat sat" and asks the model to predict "cat", "sat" and "down" at positions 1, 2 and 3 in a single forward pass. Suppose the model assigns probability 0.20 to "cat" after "the", 0.50 to "sat" after "the cat", and 0.25 to "down" after "the cat sat". The per-token losses are the negative logs: −ln 0.20 ≈ 1.61, −ln 0.50 ≈ 0.69, −ln 0.25 ≈ 1.39. The mean cross-entropy is about 1.23 nats, and perplexity is e1.23 ≈ 3.4. Intuitively, the model is on average as uncertain as if it were choosing uniformly among about 3.4 tokens.
Perplexity is only comparable between models that share a tokenizer and evaluation text: a tokenizer that splits words into more pieces produces more, easier predictions and a lower per-token perplexity for the same underlying quality. Compare in bits per byte or bits per character when tokenizers differ.
The same factorization explains the training-inference gap. During training every prefix comes from real data (teacher forcing). At inference the prefix comes from the model's own samples, so a single unlikely choice early on shifts every later conditional — the "exposure bias" mentioned above.
The model only outputs a distribution; the decoding rule decides what text you actually see, and it matters as much as the weights for perceived quality.
| Method | How it works | Good for | Failure mode |
|---|---|---|---|
| Greedy | Always take the most likely token | Short deterministic outputs, classification-style prompts | Repetition loops, bland text |
| Beam search | Keep the b best partial sequences by total log-probability | Translation, summarization | Favours short, generic outputs in open-ended generation |
| Temperature | Divide logits by T before softmax | Controlling diversity | High T produces incoherence |
| Top-k | Sample only among the k most likely tokens | Cutting the long tail | Fixed k is too narrow for flat distributions and too wide for peaked ones |
| Top-p (nucleus) | Sample from the smallest set whose probability sums to p | General chat and creative writing | Still can drift on long generations |
Common starting points are temperature 0–0.3 for extraction, code and tool calls, and temperature around 0.7 with top-p 0.9–0.95 for open-ended writing. Repetition and frequency penalties lower the logits of tokens already generated. For outputs that must parse, constrained decoding masks any token that would violate a grammar or JSON schema — see Function Calling and Structured Output.
| Autoregressive | Diffusion | GAN | VAE | |
|---|---|---|---|---|
| Likelihood | Exact | Lower bound | None | Lower bound (ELBO) |
| Sampling | One step per token | Tens of denoising steps | One forward pass | One forward pass |
| Natural data | Discrete sequences (text, code, tokens) | Continuous signals (images, audio) | Images | Latent representations |
| Main weakness | Sequential latency, no revision | Many steps | Unstable training, mode collapse | Blurry samples |
The families also combine. DALL-E 1 and Parti first compress images into discrete tokens with a VQ-VAE or ViT-VQGAN and then model those tokens autoregressively; latent diffusion uses a VAE for compression and diffusion for generation. Audio models such as WaveNet were autoregressive over raw samples, which is accurate but slow, and later systems moved to codec tokens to shorten the sequence.