Unstructured data indexing turns documents, web pages and transcripts into searchable indexes. The steps below describe each stage; the later sections show how they connect into a pipeline, how keyword and vector indexes are combined, and the trade-offs that matter in practice.
This process results in a structured, indexed data store that enables efficient search, retrieval, and analysis of previously unstructured information. By summarizing content and embedding semantic relationships, the system supports advanced applications like question-answering, information retrieval, and analytics.
The numbered steps above describe each stage in isolation; in a working system they form one pipeline that runs both for the initial backfill and incrementally as new documents arrive. Extraction is usually the hardest step: PDFs need layout-aware parsing and OCR for scanned pages, audio needs speech recognition, and HTML needs boilerplate removal. Chunking then splits documents into passages small enough to embed and to fit in a model prompt — typically a few hundred tokens with some overlap, split on headings or paragraphs rather than fixed character counts where possible. Enrichment attaches metadata such as source, date, author, section title and entities, which later powers filters like “contracts from 2025 mentioning Acme”.
Note that classical normalisation steps such as lowercasing and stopword removal help keyword indexes but should not be applied to the text you embed; modern embedding models expect natural text, and stripping it lowers retrieval quality.
An inverted index maps each term to the chunks that contain it and scores matches with BM25, which rewards rare terms and normalises for chunk length. A vector index stores one embedding per chunk and finds nearest neighbours with an approximate algorithm such as HNSW. Each fails differently: BM25 misses synonyms and paraphrases, while embeddings blur exact identifiers such as error codes, SKUs and names. Hybrid retrieval runs both and merges the ranked lists, commonly with reciprocal rank fusion, then optionally reranks the top candidates with a cross-encoder for precision.
import re
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
docs = {
"manual.txt": "Error E42 means the pump pressure sensor failed. Replace sensor P-100.",
"faq.txt": "If the machine stops pumping water, check the pressure sensor first.",
"policy.txt": "Warranty covers parts for two years from the date of purchase.",
}
def chunk(text, size=12, overlap=4):
words = text.split()
step = size - overlap
return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - overlap, 1), step)]
chunks = [(src, c) for src, t in docs.items() for c in chunk(t)]
texts = [c for _, c in chunks]
# keyword-style index (TF-IDF stands in for BM25 here)
vec = TfidfVectorizer(lowercase=True, token_pattern=r"[A-Za-z0-9-]+")
X = vec.fit_transform(texts)
def search(query, k=2):
scores = cosine_similarity(vec.transform([query]), X)[0]
best = scores.argsort()[::-1][:k]
return [(chunks[i][0], round(float(scores[i]), 3), chunks[i][1]) for i in best]
for hit in search("pump not working pressure sensor"):
print(hit)The example keeps the source file name with every chunk, which is the minimum metadata needed for citation. In production the TF-IDF matrix would be replaced by a BM25 index (Elasticsearch, OpenSearch, Xapian or a library such as Whoosh) and a second, embedding-based index would be queried in parallel and fused with the keyword results.
| Decision | Smaller / simpler | Larger / richer |
|---|---|---|
| Chunk size | Precise matches, less context | More context, diluted embeddings |
| Summaries per document | Cheaper ingestion | Better high-level questions, extra LLM cost |
| Index type | Keyword only: transparent, cheap | Hybrid: better recall, two systems |
| Embedding model | Small: fast, cheap to re-embed | Large: better quality, slower |
See BM25 and TF-IDF for keyword scoring, Python Whoosh for a pure-Python index, How Vector Embeddings Work in RAG, and the RAG workflow that consumes these indexes.