Encoder-only models keep just the encoder stack of the Transformer. With no causal mask, every token's representation is conditioned on the whole input in both directions, which makes them excellent at understanding tasks — classification, named-entity recognition, extractive QA — and the standard backbone for text embedding models used in semantic search and RAG.
This page covers BERT and RoBERTa pretraining, how token vectors become a sentence embedding, the bi-encoder / cross-encoder split, and a guide to the embedding models documented on this site: MiniLM (384 dims), multi-qa-mpnet (768 dims) and OpenAI ada-002 (1536 dims).
BERT (2018) is pretrained with masked language modelling: about 15% of input tokens are selected, mostly replaced by [MASK] (sometimes by a random token or left unchanged), and the model predicts the originals from both-side context. BERT also used next-sentence prediction (NSP) on sentence pairs. Base: 12 layers, 768 hidden, about 110M parameters; Large: 24 layers, 1024 hidden.
RoBERTa kept the architecture but changed the recipe: much more data, longer training, larger batches, dynamic masking each epoch, byte-level BPE, and no NSP. It showed BERT was undertrained. Later variants include DistilBERT and MiniLM (distillation for speed), ALBERT (parameter sharing), DeBERTa (disentangled position attention), ELECTRA (replaced-token detection) and MPNet (permuted plus masked prediction).
An encoder outputs one vector per token. For a single sentence vector, you pool: the [CLS] vector, the mean of token vectors (masking padding), or a max. Raw BERT pooled vectors are poor similarity features; Sentence-BERT fixed this by fine-tuning siamese encoders on sentence pairs so that cosine similarity reflects meaning. Modern embedding models train with contrastive losses (e.g. multiple-negatives ranking: each query's paired passage is the positive, all others in the batch are negatives) on very large pair datasets.
A bi-encoder embeds query and documents independently, so documents can be embedded once and searched with an approximate nearest-neighbour index — the retrieval stage of RAG. A cross-encoder feeds the pair through the encoder together so attention flows between them; it is more accurate but must run once per pair, so it is used to re-rank a shortlist.
# pip install sentence-transformers
from sentence_transformers import SentenceTransformer, CrossEncoder, util
docs = ["Reset your password from the account settings page.",
"Our office is closed on public holidays.",
"Two-factor authentication can be enabled under Security."]
query = "how do I change my login password"
# Stage 1: bi-encoder retrieval
bi = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
D = bi.encode(docs, normalize_embeddings=True)
q = bi.encode(query, normalize_embeddings=True)
hits = util.semantic_search(q, D, top_k=3)[0]
# Stage 2: cross-encoder re-ranking of the shortlist
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
scores = ce.predict([(query, docs[h["corpus_id"]]) for h in hits])
for h, s in sorted(zip(hits, scores), key=lambda t: -t[1]):
print(f"{s:6.2f} {docs[h['corpus_id']]}")| Model | Dims | Hosting | Typical use |
|---|---|---|---|
| all-MiniLM-L6-v2 | 384 | Local, CPU-friendly | Fast general-purpose similarity, prototypes, edge |
| multi-qa-mpnet-base | 768 | Local, GPU helps | Asymmetric question-to-passage retrieval |
| text-embedding-ada-002 | 1536 | OpenAI API | Managed embeddings, long inputs; superseded by text-embedding-3 |
Higher dimensionality is not automatically better: it raises index memory and latency linearly. Evaluate on your own queries (recall@k, nDCG) and consult public leaderboards such as MTEB only as a starting point.
The original use of BERT was not embeddings but fine-tuning: add a tiny task head and train the whole network on labeled data. Three heads cover most tasks:
Typical hyperparameters from the BERT paper are a learning rate of 2e-5 to 5e-5, batch size 16 or 32, and 2–4 epochs, with linear warmup and decay. Fine-tuning is sensitive to the random seed on small datasets; run a few seeds and report the mean. With a few hundred examples per class, a fine-tuned base-size encoder is often a strong, cheap alternative to prompting a large decoder model, and far faster at inference.
Take the input "the bank raised interest rates". During pretraining, the token "interest" might be selected and replaced with [MASK]. To predict it, the model must use "raised" and "rates" on the right as well as "bank" on the left. A left-to-right model predicting the same position would see only "the bank raised". The bidirectional objective is why encoder representations are strong for understanding: the vector for "bank" here is shaped by "interest rates", pulling it toward the financial sense rather than the river sense.
The 80/10/10 rule (of the 15% selected tokens, 80% become [MASK], 10% a random token, 10% stay unchanged) exists because [MASK] never appears at fine-tuning time; mixing in random and unchanged tokens forces the model to build a good representation for every position, not only masked ones. MLM learns from only about 15% of positions per sequence, which is one reason ELECTRA's replaced-token detection, which learns from every token, is more sample-efficient.
Classic BERT and RoBERTa use learned absolute position embeddings for 512 tokens, and self-attention cost grows quadratically with length. Options for longer inputs:
Truncation is silent in most tokenizers. Log how often inputs exceed the limit, because a classifier that only ever sees the first 512 tokens of a contract may miss the clause that matters.
Encoders are cheap to serve compared with generative models: one forward pass, no autoregressive loop. Practical steps, in order of effort: batch requests and bucket them by length so padding is minimal; export to ONNX Runtime or TensorRT; use fp16 on GPU or dynamic int8 quantization on CPU after confirming the accuracy change on your validation set; and finally distill into a smaller student (DistilBERT keeps about 97% of BERT's GLUE score with 40% fewer parameters, per its paper) if latency is still too high. Cache embeddings for static documents rather than recomputing them.
See also Self-Attention, Decoder-Only and Encoder-Decoder Transformers.