Design RAG Around Evidence Lifecycles, Not Vector Search

Teams often describe retrieval-augmented generation as a simple pipeline: split documents, create embeddings, retrieve similar chunks, and place them in an LLM prompt. That description is technically correct and architecturally misleading.

The hard part is not retrieving text. It is maintaining a defensible relationship between a generated answer and the evidence that should support it. Evidence changes, permissions differ, sources conflict, and a relevant passage may still be untrustworthy or obsolete.

A stronger architecture treats RAG as an evidence lifecycle. It defines how evidence enters the system, becomes searchable, participates in an answer, and eventually expires. This framing leads to better boundaries, clearer ownership, and fewer surprises in production.

Separate the application into control and evidence planes

A useful starting point is to divide the system into two logical planes.

The control plane manages behavior: user intent, model selection, prompt policy, tool routing, response constraints, and fallback decisions. It answers, “What should the application do?”

The evidence plane manages knowledge: source ingestion, parsing, access rules, versioning, retrieval, ranking, citation, and retention. It answers, “What information may support this response?”

These planes can share infrastructure, but they should not share responsibilities. For example, a prompt template should not encode document permissions, and an embedding index should not determine whether the application may answer a question.

This separation also makes failures easier to diagnose. If retrieval returns an old policy, that is an evidence-plane defect. If the model ignores a retrieved exception clause, that is an orchestration or response-generation defect. Without explicit boundaries, both problems appear as “the LLM gave a bad answer.”

Make ingestion preserve meaning and provenance

Most RAG quality is determined before a query arrives. Ingestion must produce units that are searchable without stripping away the context required to interpret them.

Fixed-size chunking is a reasonable baseline, not a production strategy. It can separate a table from its heading, a policy exception from its rule, or a code sample from the text defining its prerequisites. Prefer structure-aware parsing based on sections, paragraphs, lists, tables, and domain-specific entities.

Each retrievable unit should carry enough metadata to support governance and debugging. At minimum, record:

• A stable source identifier and canonical location
• Source version, ingestion time, and effective dates
• Document type, section path, and content language
• Tenant, group, role, or attribute-based access rules
• Parser, chunker, and embedding model versions
• Relationships to parent sections, tables, and adjacent chunks
• Deletion, supersession, or legal-hold status

Store original content separately from derived representations. Embeddings, summaries, keywords, and extracted entities are rebuildable artifacts. The source and its provenance are the durable record.

This distinction matters during re-indexing. If an embedding model changes, teams should be able to rebuild the index without re-fetching every external system or losing the ability to reproduce an earlier answer.

Retrieval should be a staged decision

A single nearest-neighbor query is rarely sufficient. Enterprise questions contain identifiers, product names, dates, legal terms, and exact phrases that semantic similarity can miss. Conversely, keyword search can find the right term in the wrong context.

Use retrieval as a staged decision:

1. Interpret the query. Identify entities, time constraints, source preferences, and whether the request requires current or historical evidence.
2. Apply authorization filters. Enforce access before candidates can enter the answer context. Post-retrieval filtering risks leakage through logs, caches, and model input.
3. Generate candidates. Combine lexical, semantic, metadata, and relationship-based retrieval where appropriate.
4. Rerank candidates. Score evidence against the actual question, not merely its embedding.
5. Assemble context. Remove duplicates, preserve useful ordering, include neighboring sections, and stay within a deliberate token budget.
6. Check sufficiency. Decide whether the evidence can support an answer or whether the system should ask a clarifying question, use another tool, or abstain.

Hybrid retrieval is usually the practical default. Dense vectors improve conceptual matching; lexical retrieval handles exact terminology. Metadata filters constrain the search space, while reranking spends more compute only on a small candidate set.

Do not hide these stages behind one opaque “search” method. Their inputs, outputs, latency, and policy decisions should be independently inspectable.

Treat context assembly as compilation

Retrieved chunks are not yet a prompt. Context assembly should behave more like a compiler: validate inputs, transform them into a defined representation, and reject combinations that violate constraints.

A context builder can group passages by source, attach citation identifiers, order sections chronologically, resolve duplicate versions, and flag conflicts. It can also reserve tokens for the user request, system rules, tool output, and the final response instead of allowing retrieval to consume the entire window.

This layer should be deterministic whenever possible. The LLM may summarize a long passage, but code should enforce permissions, token budgets, citation formats, and version rules.

Conflict handling deserves explicit design. If two approved sources disagree, the application should not silently select whichever passage ranked first. It may prefer the source with a later effective date, apply a documented authority hierarchy, or present the conflict to the user. The correct choice is domain-specific, but it must be visible and testable.

Generate claims that remain attached to evidence

A citation added after generation is cosmetic. The model may produce a correct-looking answer whose citation merely discusses the same topic.

Instead, make evidence identifiers part of the generation contract. Require the model to associate material claims with specific passages, then validate that cited identifiers were included in the authorized context. For higher-risk workflows, use a second pass to check whether each claim is entailed by its cited text.

The response contract should distinguish among:

• Claims directly supported by retrieved evidence
• Calculations or transformations derived from that evidence
• General model knowledge, if the product permits it
• Unsupported conclusions that should be removed or qualified

Some applications should answer only from supplied evidence. Others can combine retrieval with general knowledge. Either policy can work; ambiguity cannot. Users need to know what a citation means and what the system does when sources are insufficient.

Design freshness, deletion, and re-indexing up front

Knowledge systems decay. Policies are replaced, tickets are closed, customer data is deleted, and source permissions change. A RAG index that only supports additions will eventually serve incorrect or unauthorized content.

Build explicit workflows for update and removal. Source events should create new versions, mark older versions as superseded, invalidate relevant caches, and update indexes predictably. Deletion must propagate to lexical indexes, vector stores, derived summaries, evaluation corpora, and any prompt or response cache containing the content.

Avoid mutating records without history. Versioned evidence allows the system to answer time-bound questions and helps engineers reproduce incidents. A response record should identify the source versions, retrieval configuration, context-builder version, and model configuration used at that moment.

Re-indexing should also be routine rather than a migration crisis. Use versioned indexes, shadow retrieval, and controlled cutovers. Compare candidate sets before switching traffic, because similar aggregate metrics can conceal regressions for specific departments, languages, or document types.

Put ownership at architectural boundaries

RAG crosses application engineering, data engineering, security, and domain operations. Shared responsibility often becomes absent responsibility.

Assign clear owners for source connectors, access-control semantics, index rebuilds, retrieval policy, response contracts, and incident response. Domain teams should decide source authority and retention. Platform teams should provide reusable ingestion, retrieval, and tracing capabilities. Product teams should own user-visible behavior, including abstention and escalation.

Keep interfaces narrow. Retrieval should return evidence objects rather than concatenated prose. Generation should receive an explicit context package rather than query arbitrary stores. This lets teams replace search technologies or models without rewriting business rules.

Short takeaway

RAG is not a vector-search feature attached to a chatbot. It is a governed evidence system with an LLM at the final mile. Preserve provenance, authorize before retrieval, assemble context deterministically, bind claims to evidence, and make expiration a first-class workflow. Those choices matter more than marginal differences between embedding models.