The Transformer (Vaswani et al., 2017, "Attention Is All You Need") replaced recurrence with self-attention: every token computes a weighted mix of every other token's representation in one parallel step. Stacking that mixing step with a per-token MLP, residual connections and layer normalization gives a block that scales cleanly with data and compute, and it now underlies nearly every large language, speech and vision model.
This section breaks the architecture into its mechanism and its three main families. Start with Self-Attention, then pick the family that matches your task.
Scaled dot-product attention, Q/K/V projections, multi-head attention, masks, and the O(n²) cost.
Bidirectional encoders trained with masked language modelling; the basis of embedding and classification models.
Causal language models: next-token prediction, KV cache, sampling, and how LLMs are built on them.
Sequence-to-sequence models with cross-attention for translation, summarization and speech.
Images as sequences of patches: patch embedding, the class token, and how ViT compares with CNNs.
Sparse feed-forward layers with a learned router: more parameters at roughly constant compute per token.
Each layer applies multi-head self-attention, then a two-layer feed-forward network (often with a GELU or SwiGLU activation and a 4x hidden expansion), each wrapped in a residual connection. The original paper placed LayerNorm after the residual add (post-norm); most current models normalize before each sub-layer (pre-norm, often RMSNorm), which trains more stably at depth.
Self-attention is permutation-invariant, so position must be injected: fixed sinusoids, learned absolute embeddings (BERT, GPT-2), or relative schemes such as rotary embeddings (RoPE) and ALiBi, which generalize better to longer sequences.
Encoder-only models let every token attend in both directions and are trained to reconstruct masked tokens, which produces strong contextual representations — ideal for embeddings, classification and extraction. Decoder-only models apply a causal mask and learn next-token prediction; with scale and instruction tuning they become general-purpose LLMs. Encoder-decoder models read an input with an encoder and generate an output with a decoder that cross-attends to it, the natural fit for translation and summarization.