A decoder-only model is a stack of Transformer blocks with a causal mask, trained to predict the next token. That one objective, scaled over large text corpora and then refined with instruction and preference tuning, produces the general-purpose large language models behind GPT, Claude, LLaMA, Mistral and most other current assistants.
This page covers the objective, the generation loop and KV cache, inference phases, decoding strategies, how base models become assistants, and the Mixture-of-Experts variant used to scale parameters.
Given tokens x1..xT, the model factorizes P(x) = ∏t P(xt | x<t) and minimizes the average negative log-likelihood. The causal mask means one forward pass computes the loss for every position in parallel during training. There is no separate encoder: prompt and response share one token stream, so "conditioning on input" is simply having it earlier in the sequence.
Typical modern block details: pre-norm with RMSNorm, rotary position embeddings (RoPE), SwiGLU feed-forward layers, grouped-query attention, and a tied or untied output projection to the vocabulary.
Generation is a loop: run the model, take the logits at the last position, choose a token, append it, repeat. Without caching, step t would recompute attention for the whole prefix. The KV cache stores each layer's keys and values for past tokens, so each new step only computes Q, K, V for one token and attends over the cached keys.
Inference has two very different phases. Prefill processes the prompt in parallel and is limited by compute. Decode generates one token at a time; each step must stream the model weights and the KV cache from memory, so it is limited by memory bandwidth. This is why batching many requests together, quantizing weights, and managing KV memory efficiently (vLLM and quantization) dominate LLM serving cost.
# pip install transformers torch
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
name = "gpt2" # small open decoder-only model
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name).eval()
ids = tok("The Transformer architecture", return_tensors="pt").input_ids
past = None
torch.manual_seed(0)
with torch.no_grad():
for _ in range(30):
out = model(input_ids=ids if past is None else ids[:, -1:],
past_key_values=past, use_cache=True)
past = out.past_key_values # KV cache: no recompute of the prefix
logits = out.logits[:, -1, :] / 0.8 # temperature
probs = torch.softmax(logits, dim=-1)
# nucleus (top-p) sampling
sp, si = probs.sort(descending=True)
keep = sp.cumsum(-1) - sp < 0.9
sp = sp * keep; sp = sp / sp.sum()
nxt = si.gather(-1, torch.multinomial(sp, 1))
ids = torch.cat([ids, nxt], dim=-1)
if nxt.item() == tok.eos_token_id:
break
print(tok.decode(ids[0]))Decoding knobs: temperature rescales logits (lower is more deterministic); top-k keeps the k most likely tokens; top-p keeps the smallest set whose probability mass exceeds p. Greedy decoding (temperature 0) is good for extraction and code where one answer is right; sampling suits open-ended text. In production, use model.generate() or a serving engine rather than a Python loop.
A pretrained base model continues text; it does not follow instructions reliably. Supervised fine-tuning on demonstrations teaches the dialogue format and task following. Preference optimization — RLHF with a reward model and PPO, direct preference optimization (DPO), or constitution/AI-feedback methods — shapes helpfulness and harmlessness. Tool use and agents (see Agents and MCP) are built on top by training and prompting the model to emit structured calls.
A useful rule of thumb: each Transformer block has about 12 d2 parameters (4 d2 for the attention projections and 8 d2 for an MLP with a 4× hidden size), so a model with L layers and width d has roughly 12 L d2 parameters plus the embedding matrix. For the LLaMA-2-7B shape (L = 32, d = 4096), 12 × 32 × 40962 ≈ 6.4 billion, and the input and output embeddings over a 32,000-token vocabulary add about 0.26 billion, which matches the reported 6.7 billion closely. In 16-bit precision the weights alone occupy about 13.5 GB.
The KV cache per token is 2 (keys and values) × layers × KV heads × head dimension × bytes. For the same model with 32 KV heads of size 128 in fp16: 2 × 32 × 32 × 128 × 2 = 524,288 bytes, or 0.5 MB per token. A single 4,096-token sequence therefore holds 2 GB of cache, and 16 such sequences need 32 GB, more than the weights. This is why grouped-query attention matters: with 8 KV heads instead of 32, the cache shrinks fourfold. Paged KV management, cache quantization, and prefix sharing for common system prompts exist for the same reason; see vLLM and quantization.
The model never sees characters, only token ids from a subword tokenizer (byte-pair encoding or SentencePiece). Token counts drive cost, latency and context limits, and they vary by language and content: code and non-English text often take more tokens per word. Positional information comes from rotary embeddings, which rotate query and key vectors by an angle proportional to position so attention scores depend on relative distance. Extending context beyond the training length is done by rescaling those rotations (position interpolation and related methods) plus some continued training on long sequences. A long advertised context does not guarantee uniform use of it; evaluate retrieval of facts placed at different positions for your use case.
Training compute is approximately 6 × N × D floating-point operations for N parameters and D training tokens (forward plus backward pass). Scaling-law studies (Kaplan et al., 2020; Hoffmann et al., 2022) found that loss falls predictably as a power law in N, D and compute, and the Chinchilla analysis suggested roughly 20 tokens per parameter for compute-optimal training. Many recent open models train far beyond that ratio because a smaller model trained longer is cheaper to serve, and inference cost over the model's lifetime often outweighs training cost.
Base models are compared on held-out perplexity and on benchmark suites covering knowledge, reasoning, math and code. Assistants need more: instruction-following, refusal behavior, factual accuracy, and tool use. Public benchmarks suffer from contamination when test items leak into training data, so production teams build private evaluation sets from real tasks, score them with exact-match checks, unit tests (for code), or calibrated LLM-as-judge rubrics, and re-run them on every model or prompt change. See LLMOps and frontier model comparison.
Decoder-only models handle chat, drafting, summarization, code generation, extraction into JSON, classification by instruction, and agents that call tools. Choose by the hardest task you need, latency and cost targets, context length, data-handling requirements (hosted API versus self-hosted open weights), and license. For narrow, high-volume tasks, a small fine-tuned model or an encoder-only classifier is often cheaper and more predictable than a large general model.
Further reading. Radford et al. (2018, 2019), GPT and GPT-2 technical reports; Brown et al. (2020), "Language Models are Few-Shot Learners", arXiv:2005.14165; Kaplan et al. (2020), "Scaling Laws for Neural Language Models", arXiv:2001.08361; Hoffmann et al. (2022), "Training Compute-Optimal Large Language Models"; Touvron et al. (2023), "Llama 2: Open Foundation and Fine-Tuned Chat Models"; Su et al. (2021), "RoFormer" (rotary embeddings); Ainslie et al. (2023), "GQA"; Kwon et al. (2023), "Efficient Memory Management for Large Language Model Serving with PagedAttention".