Python Whoosh

Whoosh is a full-text search library written entirely in Python. It builds an on-disk inverted index, scores matches with BM25F by default, and lets you query that index with a Lucene-like query language, all without a JVM or a separate search server. That makes it a practical choice for the lexical (keyword) half of a retrieval-augmented generation pipeline, for search inside a desktop or command-line tool, and for prototypes where Elasticsearch would be too much infrastructure.

1. What Whoosh is

Whoosh gives you the core pieces of a search engine as Python objects. A Schema declares fields and their types. An analyzer turns raw text into tokens (lower-casing, stop-word removal, stemming). A writer adds documents to segments on disk. A searcher opens a point-in-time view of those segments, and a QueryParser turns a user string such as quick fox into a query tree that the searcher scores and ranks.

Field types matter. TEXT fields are tokenized and indexed with positions, so they support phrase queries and relevance scoring. ID fields are stored as a single untokenized term, which suits paths, URLs, and primary keys; marking one unique=True lets update_document() replace an earlier version. KEYWORD holds space- or comma-separated tags, NUMERIC and DATETIME support range queries, and stored=True on any field keeps the original value so results can display it.

2. How indexing and search work

At index time each document passes through the field's analyzer. The StemmingAnalyzer used below splits text into words, lower-cases them, removes common stop words, and applies the Porter stemmer. The writer records, for every resulting term, a postings list: which documents contain it, how often, and at which positions. On commit() those postings are written as a new immutable segment in the index directory, along with stored field values and per-document field lengths that BM25 needs for length normalization.

At query time the parser runs the query string through the same analyzer, builds a query tree (terms combined with AND by default), and the searcher walks the postings lists for those terms. Each candidate document receives a score, and the top limit hits are returned as a Results object whose items behave like dictionaries of stored fields.

The table shows why the query quick fox returns documents /a and /c only: they are the only documents whose postings lists appear under both terms. Document /b never enters the candidate set, so it is never scored.

3. Worked example: index, search, output

The first script defines a schema with a stored title, a stemmed content field, and a unique path, creates an index in indexdir, adds three documents, and commits. The second script opens that directory, parses each query against the content field, and prints the hits. Install the library with pip install whoosh (the maintained fork is published as whoosh-reloaded and keeps the same import name).

Create Python BM25 Index

#!/usr/bin/env python
from whoosh.index import create_in
from whoosh.fields import Schema, TEXT, ID
from whoosh import qparser
from whoosh.qparser import QueryParser
from whoosh.analysis import StemmingAnalyzer
import os

# Define schema for indexing
schema = Schema(
    title=TEXT(stored=True),
    content=TEXT(stored=True, analyzer=StemmingAnalyzer()),  # Stemming for better search results
    path=ID(stored=True, unique=True)
)

# Create index directory if it doesn't exist
if not os.path.exists("indexdir"):
    os.mkdir("indexdir")

# Create the index
index = create_in("indexdir", schema)

# Add documents to the index
writer = index.writer()

# Example documents to index
documents = [
    {"title": "Document 1", "content": "The quick brown fox jumps over the lazy dog.", "path": "/a"},
    {"title": "Document 2", "content": "Whoosh is a fast search library implemented in pure Python.", "path": "/b"},
    {"title": "Document 3", "content": "The fox is quick and jumps high.", "path": "/c"}
]

# Add each document to the index
for doc in documents:
    writer.add_document(title=doc["title"], content=doc["content"], path=doc["path"])

writer.commit()  # Save changes to the index

print("Indexing completed.")

Search Index BM25

#!/usr/bin/env python

from whoosh.index import open_dir
from whoosh.qparser import QueryParser

def search(query_str):
    # Open the index
    ix = open_dir("indexdir")

    # Parse the query
    with ix.searcher() as searcher:
        query_parser = QueryParser("content", ix.schema)
        query = query_parser.parse(query_str)

        # Perform the search
        results = searcher.search(query, limit=10)

        # Print the results
        print(f"Search results for '{query_str}':")
        for result in results:
            print(f"Title: {result['title']}, Path: {result['path']}")
            print(f"Content: {result['content']}")
            print("-" * 40)

# Example search queries
search("quick fox")
search("search library")

Output

#!/usr/bin/env python
Search results for 'quick fox':
Title: Document 3, Path: /c
Content: The fox is quick and jumps high.
----------------------------------------
Title: Document 1, Path: /a
Content: The quick brown fox jumps over the lazy dog.
----------------------------------------
Search results for 'search library':
Title: Document 2, Path: /b
Content: Whoosh is a fast search library implemented in pure Python.
----------------------------------------

Document 3 ranks above Document 1 for quick fox even though both contain both terms. BM25 normalizes term frequency by field length, and Document 3's content field is shorter, so each matching term carries more weight there.

4. BM25F scoring and query syntax

Whoosh's default weighting is scoring.BM25F, a field-aware variant of Okapi BM25. For each query term it combines inverse document frequency (rare terms count more) with a saturating function of term frequency, controlled by K1 (how quickly repeated occurrences stop adding score) and B (how strongly long fields are penalized). You can tune those or switch to TF_IDF or Frequency:

from whoosh import scoring
from whoosh.index import open_dir
from whoosh.qparser import MultifieldParser, OrGroup

ix = open_dir("indexdir")
weighting = scoring.BM25F(B=0.75, K1=1.2)

with ix.searcher(weighting=weighting) as searcher:
    # Search title and content, OR terms together but reward documents matching more of them
    parser = MultifieldParser(["title", "content"], ix.schema, group=OrGroup.factory(0.9))
    q = parser.parse('fox OR "search library"')
    for hit in searcher.search(q, limit=5):
        print(round(hit.score, 3), hit["path"], hit["title"])
        print(hit.highlights("content"))   # snippet with matched terms marked

The parser accepts AND, OR, NOT, quoted phrases, field prefixes such as title:fox, wildcards (qui*), and ranges on numeric or date fields. By default terms are ANDed; OrGroup.factory(0.9) switches to OR while still boosting documents that match more of the terms, which is usually the better behaviour for natural-language questions coming from a RAG front end.

To keep an index current, open it with open_dir and use writer.update_document(...), which deletes any document with the same unique path before adding the new one. Many small commits create many small segments; writer.commit(optimize=True) merges them, at the cost of a slower commit.

5. Trade-offs and when to use it

In a RAG stack, Whoosh usually supplies the sparse retriever. Exact identifiers, error codes, and rare product names are matched reliably by BM25 even when an embedding model blurs them, so a common pattern is to retrieve candidates from both Whoosh and a vector index, merge them with reciprocal rank fusion, and pass the top chunks to the model.

6. Pitfalls

• Analyzer mismatch. Changing the analyzer of a field after indexing silently breaks matching; rebuild the index when the schema changes.
• create_in wipes the index. It creates a fresh index every time. Use open_dir (or exists_in first) for incremental updates.
• Concurrent writers. Only one writer may hold the lock. Use AsyncWriter or a single indexing process when several workers produce documents.
• Searcher staleness. A searcher is a snapshot; call searcher.refresh() or open a new one after commits.
• Project maintenance. The original package has not been actively developed for years; pin a version or use the whoosh-reloaded fork for current Python releases.
• Stored fields cost space. Storing full content duplicates the corpus on disk; store only what results need to display.

7. Related topics

• BM25 Probabilistic Model - the scoring formula behind Whoosh's default ranking
• TF-IDF - the simpler weighting BM25 refines
• Retrieval-Augmented Generation Workflow
• Unstructured Data Indexing
• Elasticsearch and OpenSearch - the server-based alternative

Featured Videos