Machine translation maps a sentence in a source language to a sentence with the same meaning in a target language. Modern systems are neural: an encoder reads the source, a decoder generates the target one subword at a time, and cross-attention learns which source tokens matter for each output token. It is the task the original Transformer paper was built for, and it remains the cleanest example of the encoder-decoder pattern.
This page covers the architecture, decoding, evaluation metrics, a runnable Hugging Face example, and when to use a dedicated NMT model versus a general-purpose LLM. It sits under Natural Language Processing.
A neural MT model learns the conditional distribution P(y | x) = ∏t P(yt | y<t, x). The encoder turns the tokenized source x into contextual vectors. The decoder is autoregressive: at step t it attends to the target prefix it has already produced (masked self-attention) and to the encoder outputs (cross-attention), then emits a probability distribution over the target vocabulary.
Training uses parallel corpora (sentence pairs) and teacher forcing: the gold prefix is fed in and the loss is cross-entropy on the next token, usually with label smoothing. Source and target share a subword vocabulary (BPE or SentencePiece unigram) so rare words, names and compounds decompose into known pieces rather than becoming <unk>.
Cross-attention weights act as a soft alignment between source and target positions. They replace the explicit word-alignment models (IBM Models 1–5) of statistical MT, and they are what lets the model reorder words, e.g. German verb-final clauses into English SVO order.
Rule-based systems encoded linguistic knowledge by hand and were brittle outside their design domain. Statistical MT learned phrase tables and reordering models from parallel data and scored candidates with an n-gram language model. Neural MT replaced this pipeline with one differentiable model: first RNN seq2seq with attention, then the Transformer, which parallelizes training across the whole sequence and handles long-range dependencies better.
Multilingual models (one network, many language pairs, a target-language tag on the input) let low-resource pairs borrow from high-resource ones, and enable zero-shot directions never seen as direct pairs in training.
Greedy decoding takes the arg-max token at each step; it is fast but locks in early mistakes. Beam search keeps the k highest-scoring partial hypotheses at every step and is the standard for MT (typical k 4–6). Because summed log-probabilities penalize long outputs, scores are divided by a length penalty such as ((5 + |y|) / 6)α.
Sampling (top-k, nucleus) adds diversity and is useful for back-translation data augmentation, but for production translation you usually want the deterministic, high-likelihood output that beam search gives. Very large beams can actually hurt quality (the "beam search curse") because the model's highest-probability outputs are often too short or generic.
sacrebleu with its signature.Surface metrics reward matching one reference; a perfectly good paraphrase can score low. Use them to compare systems on the same test set, not as absolute quality numbers.
# pip install transformers sentencepiece torch sacrebleu
from transformers import MarianMTModel, MarianTokenizer
import sacrebleu
name = "Helsinki-NLP/opus-mt-de-en"
tok = MarianTokenizer.from_pretrained(name)
model = MarianMTModel.from_pretrained(name).eval()
src = ["Das Haus ist wunderbar.", "Ich habe heute keine Zeit."]
batch = tok(src, return_tensors="pt", padding=True)
out = model.generate(**batch, num_beams=4, max_new_tokens=64)
hyp = tok.batch_decode(out, skip_special_tokens=True)
print(hyp)
refs = [["The house is wonderful.", "I don't have time today."]]
print(sacrebleu.corpus_bleu(hyp, refs).score) # corpus BLEU
print(sacrebleu.corpus_chrf(hyp, refs).score) # chrF, more robust for morphologyMarianMT models are compact single-pair Transformer encoder-decoders trained on OPUS data and run comfortably on CPU. Swap the model name for other pairs (opus-mt-en-fr, opus-mt-en-es, ...) or use a multilingual model such as NLLB or mBART-50, which take explicit source/target language codes.
| Concern | Dedicated NMT (Marian, NLLB) | General LLM (prompted) |
|---|---|---|
| Cost / latency | Low; small models, batchable, CPU-viable | Higher per token |
| Determinism | Beam search is reproducible | Varies with sampling and model versions |
| Document context & style | Usually sentence-level | Handles tone, glossary, and cross-sentence context via the prompt |
| Low-resource languages | Strong with targeted multilingual models | Uneven |
| Failure mode | Literal, awkward phrasing | Fluent but may add, drop, or "improve" content |
A common production pattern is NMT for bulk traffic plus an LLM post-editing or quality-estimation pass for high-value content.