GuideAIRAGLLMCustomer support

Building an AI support agent with RAG: architecture, evaluation and pitfalls

How to build a retrieval-augmented (RAG) AI support agent that answers from your knowledge base: architecture, chunking, evaluation, guardrails, handover and cost.

By INTELLIA AI & Data Team · Reviewed by INTELLIA Engineering Team

Published October 20, 2026

Last updated November 1, 2026

3 min read

Short answer

A RAG support agent answers customer or staff questions by first retrieving relevant passages from your own knowledge base and then asking a language model to write an answer using only those passages, with citations. Building one well requires clean source content, good chunking and retrieval, an evaluation set of real questions, guardrails, human handover and cost monitoring.

How to build a RAG support agent: step by step

1
Collect and clean knowledge sources
Gather help articles, policies, product data and resolved tickets; remove outdated and duplicate content.
2
Chunk and index
Split documents into meaningful sections with metadata and index them for keyword and vector search.
3
Build an evaluation set
Collect 100 to 300 real questions with expert-approved answers and source passages.
4
Implement retrieval and generation
Retrieve top passages, re-rank them, and prompt the model to answer only from them with citations.
5
Add guardrails and handover
Detect low confidence, sensitive topics and requests for a human, and hand over with context.
6
Monitor and improve
Score answers continuously, review failures weekly and update content and prompts.

What a RAG agent is, and is not

A RAG agent is a search engine and a writer working together. The search step finds the passages in your knowledge base most relevant to the question; the writing step turns those passages into a clear answer. It is not a model that "knows" your business. If the right passage is not retrieved, the answer will be wrong or missing, which is why most of the work lies in content and retrieval rather than in the model.

Reference architecture

Layer	Responsibility	Typical components
Channels	Where questions arrive	Website chat, WhatsApp, helpdesk, intranet
Orchestrator	Conversation state, routing, tools	Application service (Python or Node.js)
Retrieval	Find relevant passages	Keyword index, vector index, re-ranker
Generation	Write grounded answers	Hosted or self-hosted language model
Guardrails	Safety, PII, policy	Classifiers, redaction, rules
Observability	Quality, latency, cost	Logs, evaluation runs, dashboards

Step 1: Fix the knowledge base first

The most common reason for poor answers is poor source material: outdated articles, contradictions between documents, and answers that exist only in agents' heads. Before writing code:

Remove or archive outdated content.
Resolve contradictions and assign an owner to each article.
Turn frequent ticket resolutions into articles.
Add metadata: product, region, audience, effective date.

Step 2: Chunking and retrieval

Split documents along their natural structure, such as headings and numbered steps, rather than fixed character counts. Keep chunks small enough to be specific but large enough to make sense alone, and store the document title and section path with each chunk.

Use hybrid retrieval: keyword search catches exact terms like product codes; vector search catches paraphrases. A re-ranking step then orders the combined results by relevance. Filter by metadata so a customer in the UAE does not receive a Pakistan-only policy.

Step 3: Evaluation before optimization

Create an evaluation set of real questions, drawn from tickets and chat logs, each with an approved answer and the source passage. Score every change against it:

Retrieval recall: did the right passage appear in the top results?
Groundedness: is every claim in the answer supported by a retrieved passage?
Correctness: does the answer match the approved one?
Helpfulness and tone: reviewed by experts on a sample.

Without this set, teams tune prompts by intuition and regress without noticing.

Step 4: Prompting for grounded answers

Good prompts for support agents instruct the model to answer only from the provided passages, cite them, say clearly when the information is not available, keep answers short and follow a defined tone. Store prompts in version control and evaluate every change before release.

Step 5: Guardrails and handover

Low confidence: if retrieval scores are weak, do not guess; offer a human.
Sensitive topics: complaints, legal threats, medical or financial advice go to people.
Personal data: mask it before sending to external models where policy requires.
Explicit requests: "talk to a person" always works.

When handing over, pass the conversation summary and retrieved sources to the agent so the customer does not repeat themselves. In CX Hub, the summary appears in the ticket.

Step 6: Tools, not just answers

Many support questions need live data: "Where is my order?", "What is my balance?" Tool calling lets the agent call your APIs with the customer's verified identity. Keep tools narrow (look up one order, not query the database), log every call, and require confirmation for actions that change data.

Step 7: Cost and latency

Track cost per resolved conversation. Practical levers: use smaller models for classification and routing, cache answers to frequent questions, limit retrieved context to what is needed, and stream responses so users see progress immediately.

RAG vs fine-tuning vs rules

Approach	Best for	Weakness
Rules and menus	A handful of fixed flows	Breaks on unexpected phrasing
RAG	Answers from changing documents	Depends on content quality
Fine-tuning	Consistent tone and format	Expensive to keep current
RAG + tools	Answers plus live account data	More integration and security work

Pitfalls we see often

Launching to all customers without a pilot group.
Measuring deflection without measuring satisfaction.
No owner for the knowledge base after launch.
Treating handover as failure instead of a designed path.

Where to start

Pick one channel and one topic area, build the evaluation set, and pilot with internal agents first using AI-suggested replies. Our AI and machine learning service runs this as a four-to-eight-week pilot, SCRIBA applies the same retrieval to internal documents, and LOOMX brings the agent to WhatsApp.

Frequently asked questions

What does RAG stand for?

RAG stands for retrieval-augmented generation: a language model generates an answer augmented with information retrieved from a specific set of documents, rather than relying only on what it learned during training.

Is RAG better than fine-tuning for customer support?

For answering questions from changing knowledge such as policies and product details, RAG is usually better, because you update documents instead of retraining. Fine-tuning helps with tone, format or domain vocabulary and can be combined with RAG.

How do you stop the AI from making things up?

Instruct the model to answer only from retrieved passages, require citations, detect when retrieval finds nothing relevant, and hand over to a human instead of guessing. Measure groundedness on an evaluation set before launch.

How many documents do you need?

There is no minimum. Even a few dozen well-written articles can support a useful agent. Coverage of the questions customers actually ask matters more than volume.

Can a RAG agent take actions like checking order status?

Yes. Combining RAG with tool calling lets the agent look up orders, create tickets or book appointments through your APIs, with permissions and logging.

About the author

INTELLIA AI & Data Team

AI, machine learning and analytics

The team responsible for INTELLIA's conversational AI, retrieval-augmented generation, analytics dashboards and data pipelines.

Named author byline pending

Keep reading

All insights

Have a project in mind?

Tell us what you are trying to achieve. A solution architect will reply within one business day with next steps — not a sales script.

Start a project Request a demo