A Production Architecture for Retrieval-Augmented LLM Apps

Retrieval-augmented generation looks simple on a whiteboard: embed documents, retrieve relevant passages, add them to a prompt, and call an LLM. That sequence is useful for a prototype, but it is not an application architecture.

Production systems need to handle changing source data, permissions, malformed documents, ambiguous queries, model failures, latency budgets, and evidence requirements. They also need to tell engineers why an answer failed. Treating retrieval as a helper function inside an API endpoint makes all of those concerns harder to isolate and improve.

A stronger design separates the system into distinct data, retrieval, generation, and evaluation paths. Each path should have explicit contracts, independent observability, and replaceable components.

Start with the application contract

Before choosing a vector database or embedding model, define what the application promises.

For an internal policy assistant, the contract might be: answer only from approved documents, respect document-level permissions, cite the exact source sections, and abstain when evidence is weak. A support copilot may instead optimize for response speed, while allowing the agent to propose a draft that a human reviews.

Write down the operational constraints:

• Which users and workflows does the application support?
• What source systems are authoritative?
• How current must answers be?
• Is an unsupported answer worse than no answer?
• Which permissions must be enforced at query time?
• What latency and cost budgets apply?
• What evidence must be logged for audit or debugging?

These decisions shape the architecture. If access control is mandatory, metadata filtering cannot be an afterthought. If information changes hourly, a nightly ingestion job is insufficient. If every claim needs a citation, the generation contract must preserve passage identifiers through the entire request.

Separate the offline and online paths

RAG systems have two fundamentally different workloads.

The offline path prepares knowledge. It connects to sources, extracts content, normalizes formats, removes duplicates, splits documents, enriches metadata, creates embeddings, and updates indexes. This work is asynchronous and often compute-heavy.

The online path answers a request. It authenticates the user, interprets the query, retrieves permitted evidence, ranks it, builds model context, generates a response, verifies output constraints, and returns citations. This path is latency-sensitive.

Do not run ingestion logic in the request path. Connect the two through versioned stores and explicit events. An ingestion job should be idempotent: processing the same source revision twice must not create duplicate chunks. It should also record lineage from each chunk back to the source document, revision, parser version, and embedding version.

That lineage lets teams rebuild an index safely, compare chunking strategies, and remove content when a source is deleted. Without it, the vector index becomes an opaque copy of organizational data that nobody can reliably govern.

Design a retrieval pipeline, not a vector lookup

Embedding similarity is only one retrieval signal. It is effective for semantic matches, but it can miss exact product codes, names, error messages, dates, and domain-specific terminology. In many enterprise applications, hybrid retrieval is a better default.

A practical retrieval pipeline can include:

1. Query normalization and optional decomposition.
2. Permission and tenant filters.
3. Parallel lexical and vector searches.
4. Candidate merging and deduplication.
5. Reranking with a stronger relevance model.
6. Context selection under a token budget.

Keep the initial search broad and use reranking to narrow the candidate set. Sending every retrieved chunk to the LLM increases cost and can reduce answer quality by burying strong evidence among weak passages.

Chunking should follow document structure where possible. Fixed token windows are acceptable for an initial baseline, but they often split tables, procedures, or definitions from their qualifiers. Preserve headings, section paths, document dates, and neighboring references as metadata. For long manuals, retrieve smaller passages for ranking and expand to the containing section only after selection.

Retrieval should return a typed result rather than raw text: passage ID, content, source ID, revision, access scope, relevance scores, and citation metadata. This contract prevents provenance from disappearing during prompt construction.

Make orchestration explicit

The request orchestrator coordinates retrieval and generation, but it should not contain every implementation detail. Keep authentication, retrieval, prompt assembly, model access, and output validation behind clear interfaces.

A typical request flow is:

• Authenticate the caller and establish tenant and role context.
• Classify whether the request requires retrieval.
• Create one or more search queries.
• Retrieve and rerank evidence with mandatory access filters.
• Decide whether the available evidence is sufficient.
• Assemble instructions, conversation state, and evidence.
• Call the model through a centralized gateway.
• Validate the response schema and citation references.
• Persist traces, metrics, and user feedback.

Use deterministic code for deterministic rules. Permissions, monetary calculations, date comparisons, and schema validation do not belong in prompts. The model can interpret language; it should not become the only enforcement layer for business policy.

A model gateway is useful even when the system initially has one provider. It centralizes timeouts, retries, rate limits, token accounting, redaction, and model configuration. It also prevents provider-specific request formats from spreading across the codebase.

Treat evidence as untrusted input

Retrieved documents can contain malicious instructions, stale guidance, or accidental secrets. The model must treat passages as evidence, not as commands. Delimit source content clearly and state that instructions inside it are not authoritative.

Prompt wording alone is not a sufficient defense. Apply controls before and after generation:

• Enforce access filters during retrieval, not after generation.
• Remove or mask secrets that users are not allowed to receive.
• Restrict tools through code-level authorization.
• Validate structured outputs against schemas.
• Check that citations refer to passages supplied in the request.
• Log source revisions used for consequential answers.

For sensitive domains, add claim-level verification. Split the proposed answer into factual claims and check whether each is supported by retrieved evidence. This adds latency, so use it selectively rather than attaching it to every low-risk request.

Build observability around stages

A single end-to-end latency metric will not explain whether the bottleneck is search, reranking, prompt construction, or model generation. Trace each stage and attach stable identifiers for the request, retrieval configuration, index version, prompt version, and model version.

At minimum, monitor:

• Ingestion freshness, failures, and document coverage.
• Retrieval latency and candidate counts.
• Reranker latency and score distributions.
• Context size, model tokens, and cost.
• Citation validity and abstention rate.
• User feedback segmented by query type and source.

Store enough diagnostic context to reproduce failures, while respecting data retention and privacy requirements. Logging entire prompts by default can expose confidential content. Prefer structured metadata and controlled sampling, with tighter access to full traces.

Evaluate retrieval separately from answers

End-to-end answer ratings are necessary but insufficient. A poor answer may result from missing source data, failed retrieval, weak ranking, bad context assembly, or unsupported generation. Each stage needs its own tests.

Create a versioned evaluation set from real tasks. Include expected sources, permission scenarios, unanswerable questions, conflicting documents, exact-match identifiers, and time-sensitive queries. Then measure retrieval recall at a chosen candidate count, ranking quality, groundedness, citation correctness, abstention behavior, latency, and cost.

Run evaluations whenever chunking, embeddings, search settings, prompts, or models change. Compare configurations on the same dataset and release through a canary or shadow deployment. RAG quality is not one score, so establish thresholds for multiple dimensions and block changes that improve fluency while reducing evidence quality.

Production feedback should feed the evaluation set, but not automatically. Review failed queries, classify the cause, remove sensitive material, and add representative cases. Otherwise, the test suite will become noisy and reinforce transient user behavior.

Plan for change

Documents, models, and retrieval techniques will change independently. Version the boundaries that connect them.

Use blue-green indexes when changing embeddings or chunking. Keep the previous index available until the replacement passes offline evaluation and production canaries. Cache stable retrieval results carefully, including tenant, permissions, query, and index version in the key. Cache generated answers only when the source freshness and personalization rules make that safe.

Avoid premature microservices. A modular monolith plus background workers is often enough for an initial production release. Split services when scaling, ownership, security, or deployment requirements justify the operational cost—not because the architecture diagram looks cleaner.

Takeaway

A reliable RAG application is a governed data and retrieval system with an LLM at its boundary. Separate offline ingestion from online serving, preserve evidence lineage, enforce permissions in code, evaluate stages independently, and version every component that can change. That foundation matters more than any single model or vector database choice.