Retrieval-Augmented Generation (RAG) Workflow

Retrieval-augmented generation answers a question in two stages: first find the passages most likely to contain the answer, then have a language model write the answer using those passages as context. Retrieval gives the model current and private knowledge it was never trained on and lets the system cite sources; generation turns the retrieved fragments into a fluent, direct answer. This page describes the standard query-time workflow, the offline indexing pipeline it depends on, a minimal runnable implementation, and the design decisions that most affect answer quality.

1. Two pipelines: indexing and query

The workflow below describes only the online half, but its quality is mostly decided offline. How documents are split into chunks, which embedding model is used, and what metadata is stored with each chunk determine what similarity search can find. The FAISS index stores vectors and integer ids; the text itself lives in a separate store (MongoDB, Elasticsearch, SQLite, or a simple dictionary), which is why step 4 of the workflow fetches raw text by id.

Standard Workflow of RAG

1. User Prompt
   • The system receives a user query or prompt as input.
2. Embedding Generation
   • The user prompt is tokenized and transformed into a vector representation using an embedding model (e.g., Sentence-BERT or a transformer encoder).
   • This vector captures the semantic meaning of the query.
3. Similarity Search in FAISS
   • The query vector is compared against stored vectors in the FAISS index to find the most similar documents based on semantic similarity (e.g., cosine similarity or Euclidean distance).
   • FAISS returns the top-N most relevant results, which include document IDs and their similarity scores.
4. Retrieval of Raw Text
   • The document IDs returned by FAISS are used to fetch corresponding raw text or content from an external database or data store (e.g., Elasticsearch, MongoDB).
5. Contextual Generation
   • The retrieved text and the original user prompt are fed into the generative model (e.g., T5, GPT).
   • The model conditions its output on the combined input of the query and retrieved documents to produce the final response.

Why Similarity Search Comes First

• Semantic Matching: Ensures the generative model is provided with semantically relevant content, enhancing response quality.
• Efficiency: Narrows down large datasets to a smaller set of relevant documents, optimizing processing.

When Text Search Might Be Used

• Supplementary Retrieval: May complement the vector-based search to apply specific filters or search for keywords.
• Hybrid Models:
  • Combines vector similarity search with full-text search for a balance between semantic and lexical matching.
  • Refines or filters results based on complex conditions after the initial vector search.

5. Hybrid retrieval and reranking

Vector search is strong on meaning and weak on exact tokens such as error codes, product SKUs, or rare names; BM25 is the opposite. Running both and merging the ranked lists with reciprocal rank fusion (score = sum of 1 / (k + rank), with k commonly 60) recovers most of the strengths of each without tuning score scales. A cross-encoder reranker then reads the query and each candidate together and produces a more accurate ordering, at higher cost, so it is applied only to the merged shortlist.

6. Worked example: minimal RAG in Python

# pip install sentence-transformers faiss-cpu numpy
import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

docs = {
    0: "FAISS is a library for efficient similarity search over dense vectors.",
    1: "BM25 ranks documents by term frequency and inverse document frequency.",
    2: "MongoDB stores JSON-like documents in collections.",
}
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Offline: embed and index (normalized vectors + inner product = cosine similarity)
vecs = model.encode(list(docs.values()), normalize_embeddings=True).astype("float32")
index = faiss.IndexIDMap(faiss.IndexFlatIP(vecs.shape[1]))
index.add_with_ids(vecs, np.array(list(docs.keys()), dtype="int64"))

# Online: embed query, search, fetch text, build prompt
question = "How does vector similarity search work?"
q = model.encode([question], normalize_embeddings=True).astype("float32")
scores, ids = index.search(q, 2)
context = "\n".join(f"[{i}] {docs[i]}" for i in ids[0] if i != -1)

prompt = f"""Answer using only the context. Cite sources as [id].
Context:
{context}

Question: {question}"""
print(prompt)   # send this to any chat model API or a local model

The generation call is left out on purpose: any hosted or local model works, and the prompt is where the important behaviour is set. Instructing the model to answer only from context and to cite ids makes unsupported answers easier to detect.

7. Design choices and trade-offs

8. Pitfalls

• Mismatched embedding models between indexing and querying produce meaningless similarity scores.
• Stale indexes. Updated documents need re-embedding; track a content hash per chunk.
• No evaluation set. Keep a set of questions with known source passages and measure retrieval recall before tuning prompts.
• Lost metadata. Without source URLs and titles stored per chunk, answers cannot cite anything.
• Prompt injection via documents. Retrieved text is untrusted input; do not let it override system instructions.

9. Conclusion

• Primary Approach: RAG begins with a similarity search against a vector database (e.g., FAISS) to identify semantically relevant documents.
• Text Search: Used as a complementary step for more complex retrieval strategies or filtering.

10. Related topics

• Retrieval-Augmented Generation
• RAG from Scratch
• Python Whoosh - BM25 keyword retrieval
• BM25 Probabilistic Model
• FAISS

Featured Videos