A RAG architecture diagram shows how a retrieval-augmented generation system answers questions from your own documents. This sample draws a typical internal knowledge assistant in two lanes: an offline ingestion path that turns documents into a searchable vector index, and an online query path that retrieves, reranks and hands the best chunks to an LLM. It is an architecture data-flow diagram, not UML, because the interesting part is where data moves and where it is stored.
1. The Diagram
Read the top lane once per document change and the bottom lane once per question. The only link between them is the yellow k-NN search into the index.
Documents (PDFs, HTML, wiki pages) are pulled in whenever they change.
Parse and clean extracts text plus metadata such as title, source URL and page number, which later become citations.
Chunk splits text into roughly 512-token pieces with a 64-token overlap so a sentence is never cut away from its context.
Embed turns each chunk into a dense vector with a bi-encoder model, and the vector index (HNSW) stores it alongside the text.
At query time the user question is embedded with the same model; a different model would put it in a different vector space.
Retrieve top-50 runs hybrid search: k-nearest-neighbour on vectors plus BM25 keyword match, merged with reciprocal rank fusion.
Rerank top-5 scores each (question, chunk) pair with a slower but more accurate cross-encoder.
Build prompt places the five chunks and their sources in front of the question, with an instruction to cite or refuse.
LLM generate writes the grounded answer, returned to the user with citations.
2. How to Read It
Boxes are processing steps, the cylinder is persistent storage and dashed rectangles are lanes that run on different schedules. Sub-labels carry the one design parameter that matters most for that step (chunk size, k, model type), which is where most RAG quality problems are found.
3. Variant: Corrective RAG
The basic pipeline always answers, even when retrieval found nothing useful. This variant adds a decision diamond: a small LLM grader checks whether the chunks are relevant. If not, the query is rewritten (for example with HyDE, which embeds a hypothetical answer) and retrieval runs again. After two failed attempts the system says it does not know and logs the gap so the content team can add a document. Red marks the error path.
4. Pros, Cons, and When to Use
Pros
Separates offline and online cost, latency and scaling concerns at a glance.
Shows exactly where an answer can go wrong: parsing, chunking, retrieval, reranking or generation.
Maps one-to-one to evaluation metrics (context recall, precision, faithfulness).
Easy for non-ML stakeholders to follow left to right.
Sub-labels document the key tuning parameters in one place.
Cons
Hides timing; a sequence diagram is better for latency budgets.
Does not show access control, which matters for multi-tenant document stores.
Fixed parameters on the diagram go stale as the team tunes them.
Agentic RAG with loops and tools quickly outgrows a two-lane picture.
When to use
Design reviews for a new RAG feature.
Explaining to stakeholders why an answer was wrong.
Onboarding engineers to an existing pipeline.
When not to use
Debugging one slow request (use a sequence diagram or trace).
Documenting the vector database internals.
5. As Diagrams-as-Code
The same diagram as a Mermaid flowchart with two subgraphs:
flowchart LR
subgraph Ingest[Offline ingestion]
D[Documents] --> P[Parse and clean] --> C[Chunk 512/64] --> E[Embed bi-encoder]
E --> I[(Vector index HNSW)]
end
subgraph Query[Online query]
Q[User question] --> QE[Embed query] --> R[Retrieve top-50 vector+BM25]
R --> RR[Rerank top-5 cross-encoder] --> B[Build prompt] --> G[LLM generate]
G --> A[Answer with citations]
end
I -. k-NN search .-> R