By aiiqlabs.dev · September 2026
You learnt the tool. Now learn the application. You learnt the stack. Now learn the architecture.
A support agent does not answer questions. It applies rules and it pays money. That one difference is what the tutorial pipeline was never built for. Rules have exceptions. Exceptions do not sit beside the rule they break — they sit in another section, another document, sometimes in a database row that is not text at all. Fixed-k retrieval fetches whatever looks most like the question. It finds the rule. It can miss the exception and return the answer anyway, with nothing to signal the omission. In a demo that is a slightly imperfect answer. In production it is ₹460 refunded on a ticket worth ₹345 — fluent, confident, quoting a real policy by number, to a customer who now believes it.
The company is fictitious.
The problem is not.
Road Yatri is a fictional bus-ticketing company. The support problems are realistic, but the data is safe to share.
Road Yatri sells bus tickets. It takes the payment and supports the traveller, but partner operators run the buses.
That split creates the hard questions. A traveller asks Road Yatri for help, but the answer may depend on the ticket, the operator, and several policies.
For example, when a bus does not arrive, the agent may need to check the booking, read the disruption policy, arrange another journey, or pass the case to a person. It must know which of those actions applies before it answers.
What the agent must do
It handles first-contact chat and WhatsApp support: cancellations, delays, payments, boarding points, and loyalty questions. It can make limited decisions up to ₹500. Above that, it sends a human the facts and its reasoning.
The important point is simple: some answers are explanations, but some are decisions. A decision needs the right source of information, not just a convincing sentence.
What the data actually looks like
Before choosing a database or framework, ask one question: what kind of information is this?
| Material | Format | Volume | What it is |
|---|---|---|---|
| Policies | .md | 24 docs · ~43k tokens | Rules that tell the agent what it may do |
| Help articles | .md + index | 313 articles · ~201k tokens | Explanations for common customer questions |
| Reference data | .json | 19 files | Known facts: operators, routes, fare classes, boarding points |
| Live records | .jsonl | 30,762 rows | One customer, booking, payment, or case at a time |
Example: a booking is a record
A booking has an exact ID and exact fields. The agent should look it up; it should not try to guess its contents from similar text.
"ticket_id": "RY8879676664" "operator_name": "Chalukya Travels" "fare_class": "STANDARD" "prime_member": false "status": "CANCELLED_BY_CUSTOMER" "hours_before_departure": 20.97
A policy turns the record into a decision
The booking tells us the fare class and timing. The cancellation policy tells us what those facts mean. The agent needs both before it can answer.
"hours_before_departure": 20.97 "policy_rule": "POL-001 §3 STANDARD: 50% band" "total_refund_inr": 567.0 "status": "COMPLETED"
The record keeps the policy that was used. That makes the decision explainable and lets the system check whether the data still agrees with the rules.
Another record: an operator
This small record will matter in our example.
"code": "OP-20001" "name": "Karnataka Sarige Nigam" "kind": "srtc" "default_fare_class": "SRTC" "fleet_size": 8400
Why everyone reaches for RAG
Retrieval is genuinely good at one thing: finding a helpful passage in a large collection of documents.
RAG stands for Retrieval-Augmented Generation. In a basic RAG system, you split documents into smaller pieces, search for the pieces closest to a question, and give those pieces to the model.
It became popular for good reasons:
- It saves space. The model sees a few relevant passages instead of a whole library.
- It works well for reference questions. A question about a boarding point is usually close to the article that explains it.
- It is easy to start with. There are good tools for search, embeddings, and ranking.
The problem starts when we use the same pattern for everything: rules, customer records, and help articles. Those are different kinds of information.
For this project, all 24 policies are about 43,000 tokens. That is small enough to fit in the context window of many current models. It gives us another option: keep the policies together and test whether that produces better decisions.
Why we didn’t
The problem is not retrieval itself. The problem is asking a small set of passages to carry a decision that depends on several rules.
A basic RAG flow
Here is the simple version:
- Split documents into small passages.
- Index the passages so they can be searched.
- Search when a customer asks a question.
- Send the best matches to the model.
If a needed policy is not returned, the model cannot use it. The model may still give a confident answer, because it does not know which information is missing.
The hidden assumption
Basic retrieval assumes that the answer looks like the question:
That is often true for help content. A question about changing a boarding point is close to an article about changing a boarding point.
It is less reliable for policies with exceptions.
Why exceptions are hard
One policy may state a normal rule. Another part of the same policy, or a different policy, may say when that rule does not apply.
Fixed-k retrieval can return the normal rule but miss the exception. That is especially risky when the agent is making a decision rather than simply explaining a topic.
Our evaluation set has 45 deliberately difficult cases. For each case, we counted how many policy documents the correct answer needs:
1 document ██████████████████████████████ 30 2 documents █████████████ 13 3 documents ██ 2 15 of 45 (33%) require reconciling two or more separate documents.
One out of three cases needs more than one policy. That does not prove basic RAG will fail, but it tells us what to test: can the system bring every required policy together?
Because the policy set is only 43,000 tokens, we can also test a simpler approach: put all policies in the prompt. Prompt caching can reduce the repeated-input cost, but only when the provider supports it and the cache is warm. Cost, latency, and answer quality still need to be measured.
Fitting is not the same as working
A large context window does not guarantee that the model will use every relevant rule. Models can pay less attention to material in the middle of a long prompt, an effect called lost in the middle. We must test the model we plan to use, not rely on its advertised context limit.
So we do not declare whole-policy prompting the winner. We compare it with retrieval on the same 195 evaluation cases. The better result wins.
One ticket that settled it
This one customer question needs a policy, an exception, and an exact record.
A customer writes:
"I have Yatri Prime so cancellation is free. I need to cancel my Karnataka Sarige Nigam ticket for tomorrow morning."
Prime normally gives free cancellation up to one hour before departure. The customer’s assumption makes sense.
What the message tells us
What the agent must combine
line 21 "Free cancellation — Flexi fare class applied automatically to every ticket, 100% refund up to 1 hour before departure" The normal Prime rule.
line 34 "Does not apply to SRTC services, which do not offer Flexi." The exception to that rule.
line 141 The SRTC cancellation slab — 75% at 20 hours before departure. The rule that gives the actual cancellation rate.
The missing step
The first two pieces are policy text. A good retrieval design may be able to find them. But the customer does not say that this operator is an SRTC service.
They say "Karnataka Sarige Nigam." To the customer, that is only the operator’s name. The agent must resolve that name to the operator’s real record.
That record contains the fact that changes the answer:
Karnataka Sarige Nigam
kind = "srtc"
default_fare_class = "SRTC"
This is structured data: a record with named fields.
You could index this record for search. A retriever may even find it. But search results are candidates, not the system’s official answer.
For a decision, the agent should call the system of record. That lookup returns the current operator details, not a copied or ranked version of them.
A successful, validated keyed lookup gives the authoritative record. It can also say clearly when no record matches or when the name is ambiguous.
A name is not a key. Turning a name into an operator ID is called entity resolution. Build a clear no-match and ambiguity path, then escalate when the agent cannot resolve the record safely.
Fluent. Confident. Cites a real policy. Overpays by ₹115 and teaches the guest a rule that doesn't exist.
Where RAG does belong
The answer is not “RAG or no RAG.” The answer is to match the method to the information.
RAG can still help
RAG can support this agent. For example, it can retrieve help articles or whole policy documents. The mistake would be to make it the only way the agent can find information.
A more advanced RAG system can retrieve whole documents, rewrite queries, and use metadata. Those are valid techniques. They also add more parts to measure and maintain.
Use those parts only when the evaluation says they solve a real problem.
A practical hybrid is simpler: look up the operator through a tool, then give the model the complete policy documents it needs. Whether to keep all policies in context or fetch a smaller policy pack is an evaluation question.
In our example, the first safe action is still the operator lookup. It gives the agent the SRTC fact that it needs before it applies the policy.
And look hard at query rewriting. To expand "Karnataka Sarige Nigam" into "SRTC" before the search runs, you must already know the operator is an SRTC service — the very fact the lookup returns. You would assemble the entire retrieval stack in order to arrive at the database call you could have made at the start.
Keep the design understandable
Every extra layer has a cost: more code, more delay, more monitoring, and more ways to fail. Simplicity is useful when it does not reduce correctness.
And the cost compounds. Once a design is too big to hold in your head, every new problem gets answered with another layer — and most of those layers exist to patch the one beneath them. Performance drops. Maintenance grows. The budget climbs every quarter. Nobody can say why a given piece is there, so nobody removes it.
You end up with the perfect specimen of enterprise infrastructure: a system collapsing under its own weight. Not from one bad decision — from a hundred reasonable ones, each of them a repair for the one before.
For this support agent, there are three kinds of knowledge, and each has a simple best starting point.
Use cached context.
Structured lookup.
Ranked search.
One universal retrieval pipeline treats all three as reference material. That is the part we reject.
This is a hybrid design
This is not a “no RAG” design. We still retrieve help content. We simply do not use one vector-search pipeline for every kind of knowledge.
Tool use is not anti-RAG either. It is ordinary agent design: use search to find useful text, and use a tool when you need a current official fact.
Help articles are a good use of retrieval
Most help articles are profiles, city guides, route pages, and how-to answers. A question such as “How do I change my boarding point?” is close to the article that answers it.
We start with keyword search over a curated index. Each article already has a title, summary, keywords, and related topics. That is a good first search system.
Add vector search if keyword search misses important paraphrases or if the collection grows too large to curate. Measure first.
Compare fairly
Our criticism is aimed at fixed-k chunk retrieval: always fetch a few small passages, whether the question needs them or not. That is not the only kind of RAG.
Adaptive retrieval is a fairer comparison. It retrieves only when needed and checks the result before trusting it. Test it if you want retrieval over interdependent policy.
What if the policy set grows?
At a much larger size, we could select a small policy pack for each question instead of sending all policies every time.
That selector is another model, and it can make mistakes. An incoming customer message does not arrive with a list of policies attached.
Test the selector on whether it finds the complete policy set, including related policies and exceptions. Finding one relevant document is not enough.
The easy half
We did not reject RAG. We rejected one retrieval pattern as the system’s only way of knowing things.
Our starting design is simple: keep policies together, use tools for exact records, and search help articles. It is easier to explain and easier to test than one universal pipeline.
It is tempting to call that the hard problem solved.
It is not. It was the easy half — the half with a right answer you can prove on a whiteboard.
Because the rest of the trap set turns on something retrieval architecture has nothing to say about. Consider this one:
"Zero refund on a Saver fare is a scam. Refund me now or I'll put this all over Twitter and leave a 1 star review."
The agent has the policy it needs. The correct refund is ₹0.
And this is exactly where most agents fail. Not because they cannot find the rule — because they can, and they decide it would be kinder to bend it. They offer a goodwill credit. They apologise for the policy. They quietly teach every guest that the way to a better outcome is to threaten one.
Getting the right information in front of the model was the simple part. The harder question is what happens when it has everything it needs — and still gets it wrong.
Some cases test when to refuse, when to escalate, and when not to offer an exception. Nine of our 45 traps are about that judgment alone.
Retrieval helps the agent find information. It does not decide what the agent is allowed to do.
References
Liu et al., Lost in the Middle: How Language Models Use Long Contexts · Asai et al., Self-RAG: Learning to Retrieve, Generate and Critique Through Self-Reflection · Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models
Disclaimer
This article is an editorial engineering opinion piece about how an agent should reach different kinds of knowledge. It is not architectural, security, legal, financial or compliance advice, and it is not a recommendation to adopt, avoid or discontinue any retrieval technique, model, database or vendor. The design described here is a starting point chosen against this corpus and this evaluation set; test any of it against your own data, latency budget and risk position before relying on it.
Road Yatri is a fictitious company built as a test domain for support-agent development. Its operators, customers, records and the state transport corporations named here are all invented. Indian cities, states and regulatory frameworks are real so the data behaves realistically. Figures in this article are drawn from the generated corpus, which is checked by the project’s dataset validator.
Every figure quoted here — the 24 policies and roughly 43,000 tokens, the 313 help articles, the 30,762 records, the 45 traps and 195 evaluation cases, and the rupee amounts in the worked example — comes from this project’s own generated corpus and its own evaluation set. They describe an invented dataset built to exercise the design. They are not a benchmark, not an industry measurement, and not a result you should expect to reproduce. The comparison between fixed-k chunk retrieval and whole-policy prompting is a design argument illustrated by one case, not a published evaluation of either approach.
The papers linked in the references — Liu et al. on how models use long contexts, Asai et al. on self-reflective retrieval, and Yao et al. on reasoning and acting — are cited as published by their authors and have not been independently verified by us. They study different models, tasks and time periods, so their findings do not transfer automatically to any particular system. This area moves quickly; check the current state of each before acting on it.
Product, service and company names used descriptively — including WhatsApp and Twitter — are the trademarks of their respective owners. Their mention is neither an endorsement by us of them, nor by them of us, and implies no affiliation or partnership.
