
A chatbot can quote a policy accurately and still give an unacceptable answer: the policy might be withdrawn, or belong to a different customer. That is why our test for RAG chatbot architecture is not “does the demo answer?” It is “can the team prove why this user received this answer from this version of the evidence?” This production guide follows that evidence trail, with launch gates for each layer.
If you are new to RAG, start with our plain-English explainer, Build an AI Chatbot With Your Own Data. It also covers hosted file-search tools. This post assumes you have decided to build a custom pipeline and want it to hold up in front of customers.
Key takeaways
- Make document ownership, permissions and regression tests part of the build contract, not launch-week tasks.
- Authorise retrieval before any evidence reaches a model, reranker or shared cache.
- Combine exact-match and semantic retrieval, then calibrate answer and abstention rules on your corpus.
- Evaluate permission failures, deleted documents and correct refusals alongside answer quality.
- Eoxys estimates a narrow pilot at about $7,000–$12,600 and a production customer-facing build at about $19,600–$35,200 (hours × $25–$35 per hour). These are estimates, not quotes.
In this guide: Why retrieval, not fine-tuning · The architecture, layer by layer · Guardrail 1: access control · Guardrail 2: grounding and refusal · Guardrail 3: prompt injection and PII · Evaluation metrics · Cost and timeline · How to start narrow · FAQ
Why retrieval, not fine-tuning
Retrieval-augmented generation, introduced by Lewis et al. in 2020, fetches relevant passages at question time and gives them to the model as context. Fine-tuning changes the model’s weights instead. For company knowledge, our view is simple: if the facts change, use retrieval.
- Freshness: when a policy changes, you re-index one document. You do not retrain a model.
- Citations: the system knows which passage it used, so it can show the source. That makes answers checkable.
- Permissions: retrieval can be limited to what the signed-in user may read. A fine-tuned model cannot forget one department’s data for one user.
Fine-tuning still earns its place for tone, output format or a narrow classification task, and the two combine well. Our RAG vs fine-tuning comparison covers that decision in detail.
RAG chatbot architecture: eight layers and their launch gates

1. Ingestion and document governance
Treat ingestion as a publishing workflow. Extract headings, tables and page references; OCR scanned material; quarantine failed parses instead of silently indexing damaged text. A number without its row heading is not reliable evidence.
Give each document an owner, stable ID, version, effective period, jurisdiction, approval status and permissions. Reject draft or expired policies at query time. Index updates and deletions with monitored jobs. Launch gate: withdraw a policy and prove it disappears from search, cached answers and citations. Let the content owner resolve conflicts; similarity scores cannot decide company policy.
2. Chunking
Chunk by sections and meaning, not an arbitrary character boundary. Keep eligibility rules with exceptions and tables with headers. Store the heading path and parent section ID so a precise search result can expand into sufficient context. Anthropic documents adding contextual descriptions to chunks; test that approach rather than assuming it helps your documents. Launch gate: questions spanning a section boundary must still retrieve the complete rule.
3. Embeddings
Embeddings represent chunks for similarity search. Benchmark domain vocabulary and every supported language, not only general English questions. Version the model and preprocessing together. Changing the vector space requires a compatible re-index and query encoder, not a casual configuration change. Launch gate: compare retrieval on the same labelled questions before switching; retain a rollback index.
4. Storage and indexing
Store searchable text and vectors with the provenance and permission metadata. pgvector adds vector similarity search to Postgres; a dedicated search engine is another option. Choose using filtered recall, update speed, backups and operational ownership, not database fashion. Launch gate: a restored index must preserve versions, tenant boundaries and deletion state.
5. Hybrid search
Semantic similarity and exact-match search solve different problems. An error code needs lexical matching; a paraphrased policy question benefits from vectors. Combine the candidate lists. Elastic recommends reciprocal rank fusion (RRF) for this, and OpenSearch supports hybrid queries natively. Rewrite conversational follow-ups into standalone queries, but never let rewriting change identity or permissions. Launch gate: test paraphrases, exact identifiers and misleading near-matches separately.
6. Reranking
A reranker compares candidate passages against the question; Cohere Rerank is one hosted option. Start with a bounded candidate budget and measure recall, latency and cost as you vary it. Relevance scores are model-specific, not calibrated probabilities of truth. Research such as Lost in the Middle is a reason to test context selection and ordering rather than stuffing every match into a prompt. Launch gate: irrelevant near-matches must trigger abstention.
7. Generation with citations
Supply numbered evidence passages and require claim-level references. Resolve those IDs to approved source URLs in application code; do not accept invented citation links. For balances or live order status, use an authorised structured lookup instead of stale document prose. Launch gate: opening a citation must reveal the supporting section to an authorised user, not just a vaguely related document.
8. Verification and logging
Validate citation IDs, check claim support and reject unsupported outputs. A model-based verifier can help but is not a correctness guarantee. Record a trace ID, source versions, model and index versions, authorisation decision, latency and outcome. Minimise stored question text and redact sensitive values. Launch gate: an investigator can reconstruct a disputed answer without unrestricted access to private documents.
Planning a knowledge-base assistant for support, sales or internal teams? Get a real project estimate within 24 hours. NDA on request.
Guardrail 1: access control inside the query
Derive the tenant, identity and groups from your authenticated backend. Apply them to lexical and vector searches, parent-section expansion, reranking and citation access. A restricted document must not reach an external reranker or generation prompt. Never ask the model to enforce permissions.
Permission changes need an explicit propagation SLA agreed with the business. Key caches by tenant and entitlement scope; reauthorise conversation history too. Test a revoked user, changed group membership and cross-tenant follow-up. Fetching a forbidden chunk and hiding it in the UI is not a security boundary.
Guardrail 2: grounding and refusal
Use an answerability decision before generation: is there current, authorised evidence that covers the question? If retrieval returns nothing, skip generation and offer a useful handoff. If documents conflict, state the conflict and escalate. A relevant passage is not necessarily sufficient evidence. Calibrate thresholds using both answerable and deliberately unanswerable questions.
Show a neutral refusal that does not disclose that a restricted document exists. Track incorrect refusals as well as unsupported answers, including separate language and role slices.
Guardrail 3: prompt injection and PII
Anything you index can contain text that reads like an instruction, such as a support ticket, a supplier PDF or a crawled web page. OWASP lists this as indirect prompt injection under LLM01, its top risk for LLM applications. Clever prompt wording will not stop it. The defences have to be architectural:
- Wrap retrieved passages in clear delimiters and tell the model they are reference data, never instructions.
- Tag each source’s trust level at ingestion (approved policy, user-submitted, external) and down-weight or exclude low-trust content for sensitive questions.
- If the bot can take actions, allow-list them per role and enforce limits in application code. Prompts can describe policy, but only server-side checks can enforce it.
- Validate outputs against an expected shape before passing them to other systems.
For personal data, detect and mask names, phone numbers, IDs and account numbers before text reaches a hosted model, where policy requires it. Open-source tools such as Microsoft Presidio handle PII detection and anonymisation. Keep audit logs for as long as your complaint and compliance process needs them, and no longer. NIST’s AI Risk Management Framework is a useful structure if you need to document these controls for auditors.
Evaluation: the metrics that matter
Build a labelled test set from consented, sanitised support questions and content-owner review. Label expected sources, answer requirements and allowed roles. Include questions with no answer, malicious documents, outdated policies and permission changes. Separate development cases from a held-out release set.
| Metric | What it tells you | Layer it points to |
|---|---|---|
| Retrieval recall (context recall) | Did the right passage appear in the results at all? | Chunking, embeddings, hybrid search |
| Context precision / rank | How high did the right passage rank, and how much noise came with it? | Reranking, query rewriting |
| Groundedness (faithfulness) | Is every claim in the answer supported by the retrieved text? | Prompt, verification step |
| Answer relevance | Did the answer address the question asked? | Generation, query rewriting |
| Refusal rate and correct refusals | Does the bot decline when the answer is not in the corpus, and only then? | Threshold, corpus coverage |
| Escalation and rephrase rate (production) | Are real users giving up or asking again straight after an answer? | Whole pipeline |
Ragas documents recall, precision, faithfulness and groundedness metrics. Specify the exact implementation: retrieval recall@k is relevant labelled passages retrieved divided by all relevant labelled passages; it is not identical to every framework’s context-recall score. Groundedness asks whether the answer is supported, not whether the source is true. Human-review high-risk cases and run regressions on every pipeline change.
Cost and timeline (Eoxys estimates)
These are Eoxys illustrative planning estimates for alternative scopes, shown as delivery hours × a blended rate of $25–$35 per hour from our team in India. They are not quotes. They exclude model API, embedding, hosting and reranker usage fees, which depend on your volume and provider.
| Scope | What is included | Hours × rate | Estimate (USD) | Timeline |
|---|---|---|---|---|
| Narrow pilot | One document set, one user group, web chat, hybrid search, reranking, citations, refusal threshold, a golden set of about 200 questions | 280–420 × $25–$30 | $7,000–$12,600 | 4–6 weeks |
| Production, customer-facing | Several sources with scheduled ingestion, SSO and access control in the query, PII masking, claim verification, evaluation in CI, monitoring dashboard | 700–1,100 × $28–$32 | $19,600–$35,200 | 10–14 weeks |
| Enterprise, multi-source | Many systems and permission models, multilingual evaluation, API look-ups and allow-listed actions, audit and retention controls | 1,400–2,200 × $30–$35 | $42,000–$77,000 | 4–6 months |
Include evaluation and content-owner review explicitly. Running cost combines ingestion and embedding updates, index hosting, retrieval, reranking, generation tokens, optional verification and observability. Compare cost per resolved question, alongside latency and refusal quality, then multiply by volume. Our chatbot cost guide covers other build types.
How to start narrow
- Pick one slice: one document set with a clear owner, one user group and one type of question, such as billing policy questions from support agents.
- Fix governance for that slice first: owners, versions, effective dates and permissions.
- Build the golden set from real questions before writing prompts.
- Ship internally with a strict refusal threshold, so the bot declines often and is right when it answers.
- Review the pilot data: recall, groundedness, refusals, escalations and unanswered questions, then fix the weakest layer first.
- Expand one dimension at a time: a new source, channel, language or action, each with its own test cases and launch gate.
Build it with Eoxys
Eoxys IT Solution has delivered 700+ projects since 2009, with a 55+ person team in Jaipur, India. Our AI and machine learning team builds retrieval pipelines, evaluation suites and guardrails. You can also hire chatbot developers or hire AI developers to work inside your team. Engagements can be fixed cost, a dedicated team or hourly, and you keep the code, indexes and provider accounts.
FAQ
What is a RAG chatbot architecture?
It is the pipeline that lets a chatbot answer from your own documents. Content is ingested and governed, split into chunks, embedded and indexed with metadata and permissions. At question time, hybrid search and a reranker find the best passages, the model answers only from them with citations, and a verification step checks the claims.
Should I use RAG or fine-tuning for company knowledge?
Use RAG when the facts change or users have different permissions. You update a document instead of retraining, answers can cite sources, and retrieval respects access rules. Fine-tuning is better for tone, format or narrow classification. Many production systems use both: a tuned model answering from retrieved context.
Why does my RAG chatbot give wrong answers?
Usually because retrieval missed the right passage or found an outdated version. Check what was retrieved before changing the prompt. Common fixes are structure-aware chunking, hybrid search for exact terms, reranking, version and effective-date metadata, and scheduled re-indexing so the index does not go stale.
How do I stop a RAG chatbot from leaking restricted documents?
Carry each document’s access-control list into the index and filter by the user’s permissions inside the search query, so restricted chunks are never retrieved. Do not rely on filtering after retrieval or on prompt instructions. Sync permission changes on a schedule and test with real role accounts before launch.
How do you measure RAG chatbot quality?
Use a golden set of real questions with known source passages and answers. Measure retrieval with context recall and precision, and generation with groundedness and answer relevance. Track refusals and, in production, escalation and rephrase rates. Run the suite on every change to chunking, embeddings, search, reranking or prompts.
How much does a custom RAG chatbot cost?
Eoxys estimates about $7,000–$12,600 for a narrow pilot, $19,600–$35,200 for a production customer-facing build and $42,000–$77,000 for an enterprise multi-source system. Each figure is delivery hours × $25–$35 per hour. These are estimates, not quotes, and exclude model, embedding and hosting usage fees.
How long does it take to build a RAG chatbot?
Plan about 4–6 weeks for a narrow internal pilot, 10–14 weeks for a production customer-facing build and 4–6 months for an enterprise system. The biggest schedule risk is usually document governance and permission clean-up, not the model, so start that work in week one.
How should I brief Eoxys for an estimate?
Share source formats, sample sanitised documents, user roles, expected question volume and deployment constraints through the Eoxys contact page. We can scope ingestion, permission handling, evaluation and handover explicitly. Ask for a real project estimate within 24 hours, NDA on request, or call +91 96678 22661.
Ready to put your knowledge base to work safely? Get a real project estimate within 24 hours, NDA on request, or call +91 96678 22661.
Related reading:
- Build an AI Chatbot With Your Own Data: RAG Explained for Business
- RAG vs Fine-Tuning in 2026: Which One Your AI App Needs
- AI Chatbot Development Cost
Sources
Checked on 20 Nov 2026 against arXiv, OWASP GenAI Security Project, Anthropic, Elastic, OpenSearch, Cohere, Ragas, pgvector, Microsoft Presidio and NIST. Cost and timeline figures are Eoxys illustrative planning estimates, not quotes.
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv)
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv)
- Anthropic: Introducing Contextual Retrieval
- OWASP GenAI: LLM01 Prompt Injection
- Elastic: Hybrid search
- OpenSearch: Hybrid search
- Cohere: Rerank overview
- Ragas: Available metrics
- pgvector: open-source vector similarity search for Postgres
- Microsoft Presidio
- NIST AI Risk Management Framework
16+ Years Building Digital Products That Ship





16+ Years Building Digital Products That Ship


