Key Takeaways
- RAG grounds AI agents in your organisation’s documents — the difference between source-cited answers and confident hallucination.
- Production RAG is a six-stage pipeline: ingestion → chunking → embedding → hybrid retrieval → reranking → cited generation. Each stage has failure modes that compound.
- Structure-aware chunking beats fixed-size chunking on contracts, tenders and policies — clause-level chunks with parent-section context preserve meaning.
- Hybrid retrieval (BM25 keyword + dense vector) with a cross-encoder reranker is the 2026 default; pure vector search underperforms on exact terms, IDs and clause numbers.
- Without a golden-question evaluation harness measuring faithfulness, context precision and answer relevancy, you cannot tell improvement from regression.
- Access control must be enforced in the retrieval layer at query time — RAG inherits none of your document permissions by default.
Every enterprise AI agent that answers a question, drafts a document or evaluates a submission is only as trustworthy as the knowledge it retrieves. Industry analysts consistently find that the majority of enterprise generative-AI deployments now depend on retrieval-augmented generation (RAG) rather than fine-tuning alone — because RAG is cheaper to update, easier to govern and, crucially, auditable. Yet most RAG systems in production today were built as weekend prototypes and never re-engineered.
We build RAG pipelines for procurement platforms, banks and government analytics programmes. This is the architecture guide we wish every team had before their first index.
What Is Enterprise RAG?
Retrieval-augmented generation (RAG) is an architecture in which a language model’s answers are grounded in documents retrieved at query time from your own corpus — policies, contracts, tickets, wikis, tenders — instead of relying on the model’s training data. The model receives the retrieved passages as context and generates an answer with citations. Enterprise RAG adds what prototypes skip: permissioning, evaluation, monitoring, PII handling and index lifecycle management.
Why it matters commercially: boards approve AI agents that can show their sources. A support agent that cites the policy paragraph, a procurement assistant that links the clause it drafted from — these earn production budgets. Ungrounded agents stay in pilot purgatory.
The Anatomy of a Production RAG Pipeline
1. Ingestion and Document Hygiene
Most enterprises have five versions of every policy and no canonical source. Before embedding anything: deduplicate, mark authoritative versions, attach metadata (owner, effective date, classification, department) and build connectors that keep the index synchronised with SharePoint, DMS or file stores. This unglamorous work delivers more answer quality than any model upgrade — and it’s where our Data & Analytics engagements usually start.
2. Chunking — Architecture, Not Preprocessing
Chunking determines what retrieval can ever find. The main strategies:
| Strategy | How It Works | Best For |
|---|---|---|
| Fixed-size | Split every N tokens with overlap | Quick prototypes; uniform prose |
| Recursive | Split on paragraphs → sentences → tokens | General documents, wikis |
| Structure-aware | Respect headings, sections, tables, clauses | Contracts, tenders, policies, manuals |
| Semantic | Split at embedding-similarity boundaries | Long unstructured narratives |
| Parent-child | Retrieve small chunks, feed parent section to the LLM | Precision retrieval + full context |
For legal and procurement corpora, clause-level chunking with parent-section context consistently wins: retrieval is precise, and the model still sees enough context to reason correctly.
3. Hybrid Retrieval and Reranking
Pure vector search fails on the queries enterprises actually ask: exact clause numbers, invoice IDs, product codes, statutory section references. The 2026 default is hybrid retrieval — BM25 keyword search fused with dense vector search (reciprocal rank fusion), followed by a cross-encoder reranker that reorders the top candidates by true relevance. This combination typically lifts retrieval precision dramatically over vector-only baselines, at millisecond-scale cost.
4. Generation With Mandatory Citations
Constrain the model to answer only from retrieved context, cite chunk IDs inline, and say “not found in the knowledge base” rather than improvise. Route refusals to human escalation. This single prompt-and-policy discipline removes the majority of hallucination incidents in production systems.
Advanced Patterns: GraphRAG and Agentic Retrieval
Two patterns matured in the last eighteen months:
- GraphRAG: build an entity-relationship graph over the corpus so multi-hop questions (“which vendors appear in contracts affected by clause X?”) traverse relationships instead of hoping a single chunk contains the answer. Powerful for compliance, procurement and investigation workloads.
- Agentic retrieval: the agent decomposes complex questions, issues multiple targeted queries, evaluates the evidence and iterates — retrieval as a reasoning loop, not a single lookup. This is how our BidsInsight platform grounds RFP drafting: clause-aware chunks, hybrid retrieval and citation-first generation over an organisation’s tender history.
Evaluation: The Difference Between Engineering and Vibes
Build a golden set of 100–300 real questions with known correct sources, then measure on every pipeline change:
- Context precision / recall: did retrieval surface the right chunks, and how much noise came with them?
- Faithfulness: is every claim in the answer supported by the cited context?
- Answer relevancy: does the answer actually address the question?
- Citation coverage: what share of answer sentences carry a verifiable citation?
Log every production retrieval — query, chunks, scores, answer — for continuous evaluation and drift detection. Teams that skip this cannot tell whether last week’s “improvement” made things worse.
Security and Compliance in the Retrieval Layer
RAG inherits none of your document permissions by default. Three controls are non-negotiable:
- ACL-aware retrieval: enforce user-level permissions at query time inside the retriever, not in the UI.
- PII minimisation before indexing: detect and mask personal data in the ingestion pipeline — cheaper and safer than scrubbing embeddings later, and directly relevant to DPDPA obligations.
- Erasure that reaches embeddings: when a document is deleted or a data principal invokes erasure, remove derived chunks, vectors and caches, then re-index.
Common Failure Modes We See in Audits
| Symptom | Root Cause | Fix |
|---|---|---|
| Right document, wrong answer | Chunks split mid-table or mid-clause | Structure-aware chunking |
| Misses exact IDs and clause numbers | Vector-only retrieval | Hybrid BM25 + vector + reranker |
| Confident answers from stale policy | No version authority in metadata | Canonical-version flags; filter at query time |
| Restricted content leaking into answers | No ACL enforcement in retriever | Query-time permission filtering |
| Quality silently degrading | No golden-set evaluation | Eval harness in CI + production logging |
A 90-Day Implementation Roadmap
- Days 1–30: corpus audit, deduplication, metadata schema, ingestion connectors; pick one high-value use case with measurable questions.
- Days 31–60: structure-aware chunking, hybrid retrieval with reranking, citation-constrained generation; build the golden set and baseline the metrics.
- Days 61–90: ACL-aware retrieval, PII masking, production logging and dashboards; pilot with a real user group and iterate on eval failures.
Our Agentic AI team delivers this as a production system — pipeline, evaluation harness and governance included. If your AI agents can’t cite their sources, they’re not ready for production. Start with the knowledge system; the agents will follow.
Frequently Asked Questions
What is enterprise RAG?
Enterprise RAG (retrieval-augmented generation) grounds AI answers in documents retrieved at query time from your own corpus — policies, contracts, tenders, wikis — with citations. Enterprise-grade RAG adds permissioning, PII handling, evaluation harnesses, monitoring and index lifecycle management on top of the basic pipeline.
What is the best chunking strategy for RAG?
It depends on the corpus. For contracts, tenders and policies, structure-aware chunking at clause level with parent-section context performs best. For general documents, recursive chunking works well. Fixed-size chunking is only suitable for prototypes — it splits tables and clauses mid-thought and damages retrieval precision.
Why is hybrid retrieval better than pure vector search?
Vector search misses exact terms — clause numbers, invoice IDs, product codes, statutory references — that enterprise users query constantly. Hybrid retrieval fuses BM25 keyword search with dense vector search and applies a cross-encoder reranker, substantially improving precision on real enterprise query mixes.
How do you evaluate a RAG system?
Build a golden set of 100–300 real questions with known correct sources. Measure context precision and recall, faithfulness (are claims supported by cited context?), answer relevancy and citation coverage on every pipeline change, and log all production retrievals for drift detection.
How does RAG handle document permissions and data privacy?
Permissions must be enforced inside the retriever at query time (ACL-aware retrieval) — RAG inherits no permissions by default. PII should be detected and masked during ingestion, and erasure workflows must remove derived chunks, embeddings and caches, not just source records — a direct DPDPA requirement.
