Search & Retrieval

BM25 Probabilistic Information Retrieval Modelinformation retrievalBM25 (Best Matching 25) is a ranking function used by search engines to rank documents based on their… Gunning Fog Indexinformation retrievalGunning Fog Index The Gunning Fog Index is a readability test that estimates the years of formal education… Retrieval-Augmented Generation (RAG) Workflowinformation retrieval RAG (Retrieval Augmented Generation)information retrievalRAG enhances the response generation process by retrieving relevant documents from an external knowledge base… TF-IDFinformation retrievalTF-IDF Value Breakdown Higher TF-IDF values: The higher the TF-IDF score for a term in a document, the more… Unstructured Data Indexinginformation retrievalThis process results in a structured, indexed data store that enables efficient search, retrieval, and… Python Whooshinformation retrieval

The retrieval half of RAG: classical ranking models, indexing strategies for unstructured data, and the readability measures used to judge what comes back.

Overview

Retrieval-augmented generation is only as good as its retrieval step. Before a language model can answer from private or current data, the system must find the few passages that actually contain the answer among thousands or millions of candidates. The pages in this section cover that half of the problem from first principles: how unstructured content is prepared and indexed, how classical ranking functions score documents against a query, how a small Python search library implements them, and how the RAG workflow combines retrieval with generation.

How the Topics Relate

Where Each Topic Fits in a Retrieval Pipeline Unstructured datadocs, email, PDFs Indexingchunk, summarise, embed RankingTF-IDF, BM25, vectors GenerationRAG prompt Evaluationreadability, relevance Indexing decides what can be found; ranking decides what comes first; evaluation checks whether the result is usable.

TF-IDF is the foundation: a term matters to a document if it is frequent there and rare elsewhere. BM25 refines it with term-frequency saturation and document-length normalisation and remains the default lexical ranking function in Lucene-based engines such as Elasticsearch and OpenSearch. Python Whoosh shows these ideas in a pure-Python full-text index. Dense vector retrieval with embeddings captures meaning rather than exact words, and most production RAG systems combine both in hybrid search. The Gunning Fog Index sits at the end of the pipeline as a simple readability measure for the text that is retrieved or generated.

Choosing a Retrieval Method

MethodStrengthWeaknessRead
TF-IDFsimple, transparent, fastno length normalisation or saturationTF-IDF
BM25strong lexical baseline, exact terms and codesmisses synonyms and paraphraseBM25
Full-text libraryindex and query in Pythonsingle-process scalePython Whoosh
Embeddings + vectorssemantic matchesweaker on rare exact terms, needs a modelRAG with pgvector
Indexing strategysummaries, metadata, chunkingextra preprocessing costUnstructured Data Indexing

Start with BM25 as a baseline, add embeddings when users phrase questions differently from the documents, and evaluate retrieval on real questions before tuning prompts. The end-to-end flow is described in the RAG workflow page, and readability scoring in the Gunning Fog Index page.

‹ AI ML