multi-qa-mpnet (768-Dimensional Embeddings)

multi-qa-mpnet-base is a sentence-transformers model built on the MPNet base encoder (12 layers, 768 hidden) and fine-tuned on a large collection of question-answer pairs from sources such as search logs, forums and QA datasets. It produces 768-dimensional vectors and is designed for asymmetric semantic search: a short query against longer passages, the retrieval step of most RAG systems.

It sits between the compact MiniLM (384 dims) and hosted models such as ada-002 (1536 dims) on the size/quality curve.

1. The MPNet Backbone

MPNet (Song et al., 2020) unifies BERT's masked language modelling and XLNet's permuted language modelling. MLM predicts masked tokens independently of each other; PLM models dependencies between predicted tokens but each prediction lacks the position information of the full sentence. MPNet permutes the prediction order while also feeding position embeddings for all tokens, so the model sees the full sentence length during pretraining just as it will at fine-tuning time.

2. Asymmetric Retrieval

Symmetric similarity (paraphrase detection) compares texts of the same kind. Retrieval is asymmetric: "how do vaccines work?" should match a paragraph explaining immune response, which shares few words with the question. Training on (question, answer) pairs with an in-batch-negatives contrastive loss teaches the encoder to place questions near their answers rather than near other similar questions.

3. Variants and Scoring

The two common checkpoints differ in pooling and normalization. Configure your vector database to match: inner product for dot-v1, cosine (or inner product on normalized vectors) for cos-v1. Using cosine on a dot-trained model discards the vector-length signal; using raw dot product on mean-pooled unnormalized vectors favours long inputs.

4. Runnable Example

# pip install sentence-transformers
from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("sentence-transformers/multi-qa-mpnet-base-dot-v1")
print(model.get_sentence_embedding_dimension())          # 768

passages = [
    "Vaccines train the immune system by exposing it to a harmless antigen.",
    "The Eiffel Tower was completed in 1889 for the World's Fair.",
    "mRNA vaccines deliver instructions for cells to make a spike protein.",
]
P = model.encode(passages, convert_to_tensor=True)       # dot-v1: do NOT normalise
q = model.encode("how do vaccines work?", convert_to_tensor=True)

scores = util.dot_score(q, P)[0]
for i in scores.argsort(descending=True):
    print(f"{scores[i]:.2f}  {passages[i]}")

# Long documents: chunk first (stay well under the max sequence length)
def chunk(text, words=200, overlap=40):
    w = text.split()
    return [" ".join(w[i:i + words]) for i in range(0, max(len(w) - overlap, 1), words - overlap)]

The model accepts up to 512 wordpieces, but it was trained on shorter text, so passages of roughly 100–300 words with some overlap usually retrieve best. A GPU speeds up bulk indexing substantially; query-time encoding is fine on CPU.

5. Trade-offs and Pitfalls

• Versus MiniLM — roughly 5x the parameters and 2x the vector size, typically better retrieval quality on question-style queries. Benchmark on your own data before paying the cost.
• Versus hosted APIs — runs locally (no data leaves your network, no per-token fees) but you own scaling and upgrades.
• English-focused — use a multilingual model for other languages.
• Domain shift — specialised vocabularies (legal, medical, internal product names) may need fine-tuning on in-domain pairs or a hybrid BM25 + vector search.
• Re-ranking — pair it with a cross-encoder on the top 20–100 hits for best precision.

6. How the Model Is Trained

The fine-tuning objective is the multiple-negatives ranking loss (an in-batch contrastive loss related to InfoNCE). A batch contains B pairs (q_i, a_i). Every question is encoded, every answer is encoded, and a B × B score matrix S_ij = sim(q_i, a_j) is built. The loss is softmax cross-entropy where the correct "class" for row i is column i, so each question must score its own answer above the B − 1 other answers in the batch. Larger batches mean more negatives per step and generally better embeddings, which is why these models are trained with large batches on many accelerators.

Because questions and answers share one encoder, the model learns a single space where a short question and a long passage can be close. Pooling turns the per-token outputs into one vector: the cos variant averages token vectors and L2-normalizes them; the dot variant uses the [CLS] token and leaves length unnormalized, so the score can also reflect how much relevant content a passage carries.

A small worked example. With normalized vectors, cosine and dot product are the same number. If a query vector scores 0.62 against passage A and 0.41 against passage B, A ranks first. Those absolute values are not probabilities and are not comparable across models; a threshold such as "accept above 0.5" must be calibrated on labeled pairs from your own data.

7. Production Retrieval Pipeline

In a retrieval-augmented generation system the model sits in two places. Offline, every document chunk is encoded once and written to a vector index such as FAISS, pgvector or a managed vector database. Online, each user question is encoded with the same model and the index returns the nearest chunks. Because the bi-encoder never sees query and passage together, it misses fine-grained interactions; a cross-encoder reranker that reads both texts jointly is commonly added on the top 50 to 100 candidates. Combining dense scores with BM25 in a hybrid search recovers exact-match terms such as product codes, names and error strings that dense vectors tend to blur.

8. Operational Notes

9. Evaluating Retrieval Quality

Build a small labeled set: a few hundred real user questions, each with the passages that answer it. Measure recall@k (did any correct passage appear in the top k?), mean reciprocal rank, and nDCG@10. Compare against BM25 alone, MiniLM, and a hosted model on the same set; public leaderboards such as MTEB are a starting point, but domain vocabulary often reorders the ranking. Re-run the evaluation whenever chunking, the model, or the index parameters change. For end-to-end answer quality in a RAG system, see RAG evaluation with RAGAS.

10. When to Fine-Tune

If evaluation shows the model failing on your domain (legal citations, internal product names, medical terminology), fine-tuning on a few thousand in-domain (query, relevant passage) pairs with the same multiple-negatives loss often helps. Pairs can come from search click logs, FAQ pages, or questions generated by an LLM from your documents and then spot-checked. Mine hard negatives (passages that BM25 ranks highly but are wrong) to make training more informative, hold out an evaluation set before training, and confirm the tuned model beats the base model on it.

Further reading. Song et al. (2020), "MPNet: Masked and Permuted Pre-training for Language Understanding", arXiv:2004.09297; Reimers and Gurevych (2019), "Sentence-BERT", arXiv:1908.10084; Henderson et al. (2017), "Efficient Natural Language Response Suggestion for Smart Reply" (multiple-negatives loss); Karpukhin et al. (2020), "Dense Passage Retrieval for Open-Domain Question Answering"; Muennighoff et al. (2022), "MTEB: Massive Text Embedding Benchmark"; the sentence-transformers documentation at sbert.net.

11. Related Topics

• Transformer Architecture — parent section
• Encoder-Only (BERT, RoBERTa) — the model family
• MiniLM (384 dims) — smaller, faster alternative
• OpenAI ada-002 (1536 dims) — hosted alternative
• Retrieval-Augmented Generation (RAG) — where these embeddings are used
• Embedding Models — vector embeddings in RAG