The encoder-decoder is the original Transformer layout: an encoder builds a bidirectional representation of the input, and an autoregressive decoder generates the output while cross-attending to that representation. It is the natural fit when the output is a transformation of a well-defined input — translation, summarization, speech recognition, structured extraction.
T5 and BART are the two reference model families. This page covers the data flow, their pretraining objectives, a fine-tuning example, and how they compare with decoder-only LLMs.
Each decoder layer has three sub-layers: causal self-attention over the tokens generated so far, cross-attention whose queries come from the decoder and whose keys/values come from the final encoder output, and an MLP. Because the encoder sees the full input bidirectionally, the input is encoded once and reused for every decoding step; only the decoder runs autoregressively.
Training uses teacher forcing: the target sequence shifted right by one is the decoder input, and cross-entropy is computed on every target position in parallel.
T5 (Text-to-Text Transfer Transformer) corrupts random spans of the input, replaces each with a sentinel token, and trains the decoder to output only the missing spans — a short, efficient target. Its relative-position biases and shared vocabulary made it a standard backbone; Flan-T5 adds instruction tuning over many tasks.
BART combines a bidirectional encoder with a GPT-style decoder and is trained to reconstruct the original text from several corruptions (token masking, deletion, span infilling, sentence permutation). Reconstructing full text makes it particularly strong at generation tasks such as abstractive summarization; mBART extends it to many languages.
T5 casts every task — classification included — as "input string to output string", with a task prefix. One loss, one decoding procedure, one model for many tasks. The same idea, scaled up and expressed as natural-language instructions, is what instruction-tuned LLMs do.
# pip install transformers sentencepiece torch
import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
name = "google/flan-t5-base"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSeq2SeqLM.from_pretrained(name)
# --- inference ---
inp = tok(["summarize: The new bridge opened after four years of construction "
"and is expected to cut commute times across the river."],
return_tensors="pt")
out = model.generate(**inp, max_new_tokens=32, num_beams=4)
print(tok.decode(out[0], skip_special_tokens=True))
# --- one fine-tuning step (labels -> decoder input shifted internally) ---
batch = tok(["translate English to German: Good morning"], return_tensors="pt")
labels = tok(["Guten Morgen"], return_tensors="pt").input_ids
labels[labels == tok.pad_token_id] = -100 # ignore padding in the loss
opt = torch.optim.AdamW(model.parameters(), lr=3e-4)
loss = model(**batch, labels=labels).loss
loss.backward(); opt.step(); opt.zero_grad()
print(float(loss))For real training use Seq2SeqTrainer with DataCollatorForSeq2Seq, which pads labels with -100 and handles predict_with_generate for ROUGE/BLEU evaluation.
| Aspect | Encoder-decoder | Decoder-only |
|---|---|---|
| Input encoding | Bidirectional, computed once | Causal, input and output share one stream |
| Best at | Fixed input-to-output transforms; fine-tuned task models | Open-ended generation, few-shot prompting, chat |
| Parameters for a given quality on seq2seq tasks | Often smaller when fine-tuned | Usually larger, but no task-specific training |
| Ecosystem | Strong for MT, ASR, summarization | Dominant for frontier LLMs and tooling |
Most frontier LLMs are decoder-only because one stream scales simply and supports arbitrary prompting. Encoder-decoders remain a good choice for focused, high-volume tasks where a fine-tuned base- or large-size model is cheaper than prompting an LLM.
truncation settings.-100 for padding or the model learns to emit pads.no_repeat_ngram_size or a repetition penalty for long outputs.Let the encoder output be a matrix H of shape (source length S × model width d). In each decoder layer, the cross-attention sub-layer computes queries from the decoder's current hidden states, Q = XdecWQ, and keys and values from the encoder output, K = HWK, V = HWV. The output is softmax(QKT / √dk)V, split across heads as in self-attention. Each generated position therefore looks over the whole source and pulls in what it needs; in translation, the attention weights often roughly align target words with their source words.
The cost has three parts. The encoder pays O(S2) once. Decoder self-attention pays O(T2) over T output tokens. Cross-attention pays O(S · T). Because K and V for cross-attention depend only on the encoder output, they are computed once and cached for every decoding step, alongside the usual self-attention KV cache. A long input with a short output (summarizing a 2,000-token article into 100 tokens) is exactly the case where this layout is efficient.
Worked shape example. With t5-base (d = 768, 12 heads of 64), a 512-token source and a 64-token target, the encoder output is 512 × 768. At decoding step t, the cross-attention score matrix per head is 1 × 512: one query against all source keys. Over the full target during training it is 64 × 512 per head, computed in parallel thanks to teacher forcing.
Generation quality depends heavily on the decoding procedure. Greedy decoding takes the most probable token at each step; it is fast but can lock into repetitive or suboptimal sequences. Beam search keeps the B most probable partial outputs and is the usual choice for translation and summarization, where there is a narrow set of correct answers. Raw beam search favors short outputs because every token multiplies in a probability below one, so a length penalty normalizes scores by length. Sampling with top-k, nucleus (top-p) and temperature gives diverse outputs for creative or data-augmentation tasks. Constraints such as no_repeat_ngram_size, minimum and maximum lengths, and forced beginning tokens (used by mBART and Whisper to select a target language or task) are applied on top. All are exposed through model.generate() in Hugging Face Transformers.
| Setting | Guidance |
|---|---|
| Model size | Start with a base-size checkpoint (t5-base, bart-base, flan-t5-base) to get a working pipeline, then scale up if evaluation shows headroom. |
| Learning rate | Around 1e-4 to 3e-4 with Adafactor is common for T5; around 1e-5 to 5e-5 with AdamW for BART. Use warmup and validate on a held-out split. |
| Max lengths | Set source and target lengths from the token-length distribution of your data (for example the 95th percentile) so few examples are truncated. |
| Task prefix | Keep T5-style prefixes ("summarize: ") consistent between training and inference. |
| Precision | bf16 is generally safe for T5; original T5 checkpoints are known to overflow in fp16. |
| Parameter-efficient tuning | LoRA adapters on attention projections cut memory substantially for large checkpoints. |
Automatic metrics compare generated text with references: BLEU and chrF for translation, ROUGE for summarization, word error rate for speech recognition, exact match and F1 for extraction. They are cheap and good for tracking regressions, but they reward surface overlap, so a correct paraphrase can score poorly. Complement them with human review of a sample, factual-consistency checks for summaries (does every claim appear in the source?), and, increasingly, LLM-as-judge rubrics that are themselves validated against human ratings. Always decode the validation set with the same generation settings you will use in production.
Further reading. Sutskever, Vinyals and Le (2014), "Sequence to Sequence Learning with Neural Networks"; Bahdanau, Cho and Bengio (2014), "Neural Machine Translation by Jointly Learning to Align and Translate"; Vaswani et al. (2017), "Attention Is All You Need", arXiv:1706.03762; Raffel et al. (2020), "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", arXiv:1910.10683; Lewis et al. (2020), "BART", arXiv:1910.13461; Chung et al. (2022), "Scaling Instruction-Finetuned Language Models" (Flan-T5); Radford et al. (2022), "Robust Speech Recognition via Large-Scale Weak Supervision" (Whisper).