MiniLM is a family of small encoder-only Transformers produced by distilling a larger BERT-style teacher. Its best-known derivative, sentence-transformers/all-MiniLM-L6-v2, maps text to a 384-dimensional vector and is the default "small, fast, good enough" embedding model in many semantic-search and RAG prototypes.
This page explains how MiniLM is distilled, what the sentence-embedding variant does at inference time, the storage implications of 384 dimensions, and when to choose something larger such as multi-qa-mpnet or ada-002.
Classic distillation matches the teacher's output logits. MiniLM (Wang et al., 2020) instead transfers the teacher's self-attention behaviour from its last layer: the student is trained to reproduce the attention distributions softmax(QKT/√dk) and the "value relation" softmax(VVT/√dk), both via KL divergence. Because these are n × n matrices per head, the student's hidden size and depth can differ freely from the teacher's. MiniLMv2 generalized this to query-query, key-key and value-value relations across multiple heads.
The sentence-transformers checkpoint takes a 6-layer, 384-hidden MiniLM and fine-tunes it contrastively on a large mix of sentence pairs (question-answer, duplicate questions, title-body, caption pairs). Inference is tokenize, encode, mean-pool over non-padding tokens, L2-normalize. With normalized vectors, cosine similarity equals the dot product.
Input is truncated at 256 wordpieces (training used shorter sequences), so chunk documents to a paragraph or a few sentences. The tokenizer is uncased and English-centric; for other languages use a multilingual variant such as paraphrase-multilingual-MiniLM-L12-v2.
# pip install sentence-transformers faiss-cpu
import numpy as np, faiss
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
print(model.get_sentence_embedding_dimension(), model.max_seq_length) # 384 256
corpus = ["The cat sits outside", "A man is playing guitar",
"The new movie is awesome", "A woman watches TV",
"The feline rests on the porch"]
emb = model.encode(corpus, batch_size=64, normalize_embeddings=True,
convert_to_numpy=True).astype("float32")
index = faiss.IndexFlatIP(emb.shape[1]) # inner product == cosine on unit vectors
index.add(emb)
q = model.encode(["a cat on the porch"], normalize_embeddings=True).astype("float32")
scores, ids = index.search(q, k=2)
for s, i in zip(scores[0], ids[0]):
print(f"{s:.3f} {corpus[i]}")On a modern CPU this model embeds hundreds of short sentences per second, which makes it practical for local indexing without a GPU and for low-latency query embedding at search time.
Vector index memory, disk, and distance-computation cost all scale linearly with dimension. At 384 dims, MiniLM vectors are a quarter the size of 1536-dim vectors, which matters at tens of millions of chunks. Product quantization or int8/binary quantization can compress further, at some recall cost.
The distilled MiniLM checkpoint is a general language encoder; it becomes a sentence embedder through contrastive fine-tuning with the multiple negatives ranking loss (an InfoNCE-style objective). A batch contains N (anchor, positive) pairs. All 2N texts pass through the same encoder, producing an N×N matrix of cosine similarities between anchors and positives. Each row is treated as an N-way classification problem whose correct answer is the diagonal entry, so every other positive in the batch acts as a free negative. Similarities are multiplied by a scale (20 in the sentence-transformers default, equivalent to a temperature of 0.05) before the softmax so the loss is not too flat.
Two consequences matter in practice. First, larger batches mean more and harder negatives, which is why the public all-* models were trained with large batches on TPUs. Second, if you fine-tune on your own data, duplicates within a batch become false negatives; dedupe pairs or use a batch sampler that avoids repeats. Adding a mined hard negative per pair (a passage that looks relevant but is not) usually helps more than additional easy pairs.
Storage is simple arithmetic. A float32 384-dimensional vector is 384 × 4 = 1,536 bytes. Ten million chunks therefore need about 15.4 GB of raw vectors; a 1536-dimensional model would need about 61 GB for the same corpus. Converting to float16 halves that, int8 scalar quantization quarters it (about 3.8 GB), and binary quantization (one bit per dimension, 48 bytes per vector) reduces it 32x at a noticeable recall cost that is usually recovered by rescoring the top candidates with the full-precision vectors.
Index structures add overhead: HNSW graphs store neighbor lists per node (commonly tens of links), which can add a significant fraction on top of the vectors. IVF-PQ indexes store only compact codes plus centroids, trading accuracy for memory. For a corpus that fits in a few GB, a flat (exact) index is often fast enough and removes a tuning knob.
The model itself is about 22.7M parameters (around 90 MB in float32). With six layers it is several times faster than a 12-layer base model at the same sequence length; cost grows with sequence length, so short chunks embed fastest.
A bi-encoder like all-MiniLM-L6-v2 compresses each text into one vector independently, which makes retrieval fast but loses fine-grained interactions between query and passage. A standard fix is to retrieve generously (top 50–200) with MiniLM, optionally merged with BM25 results, and then rerank with a cross-encoder such as cross-encoder/ms-marco-MiniLM-L-6-v2, which reads the query and passage together. Reranking only a hundred candidates keeps latency bounded while recovering much of the precision of a larger model. See Hybrid Search and Reranking for the full pattern.
| Model | Dims | Size | Max tokens | Notes |
|---|---|---|---|---|
| all-MiniLM-L6-v2 | 384 | ~23M params | 256 | Fast, CPU-friendly, English |
| all-MiniLM-L12-v2 | 384 | ~33M params | 256 | Somewhat better quality, about half the speed |
| multi-qa-mpnet-base-dot-v1 | 768 | ~110M params | 512 | Trained for question-answer retrieval |
| text-embedding-ada-002 | 1536 | API | 8191 | Hosted; long inputs; per-token cost |
Public leaderboards such as MTEB rank many newer open models above MiniLM on retrieval quality. The honest guidance is to build a small evaluation set from your own queries (50–200 labeled query-passage pairs), measure recall@10 and nDCG for two or three candidates, and pick the cheapest model that meets the bar. MiniLM is frequently the baseline that the larger models must clearly beat to justify their cost.
normalize_embeddings=True) so scores equal cosine similarity.Further reading: Wang, W. et al. (2020), "MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained Transformers" (arXiv:2002.10957); Reimers, N. & Gurevych, I. (2019), "Sentence-BERT" (arXiv:1908.10084); Henderson, M. et al. (2017), "Efficient natural language response suggestion for smart reply" (in-batch negatives); Muennighoff, N. et al. (2023), "MTEB: Massive Text Embedding Benchmark".