Giving an AI Agent Sources for Company Questions

The contract has no refund clause at all, yet the agent says "in general, a refund should be possible." The answer reads smoothly and it's dangerous. One kind of agent error is especially misleading: stating something uncertain with confidence.

Real tasks often depend on private files, business records and fresh information that the model has never seen. For example:

What does our company's travel policy say?
What did this customer complain about last time?
Does this contract have an automatic renewal clause?
How should this course be split into chapters?
Where does the process break down in the project documents?

These questions go beyond general knowledge. The answers sit in specific files, records, policies, contracts or project documents. A large model learns general knowledge in training, so it can explain concepts, summarize common practice and give general advice.

But it may lack your private material and the latest information: a policy the company updated today, an email a customer sent yesterday, the wording of clause 12 in a contract, or what a project's code actually says.

Even if the model knows a company's public information, it can't infer the contents of a particular contract or yesterday's email. That material has to reach the model through input, files or a retrieval system, and for facts that change you also need to check the date and the version.

An agent that relies only on the model's general knowledge tends to write guesses as facts. Get the right material first and answer from it, and the conclusion can be checked. When the material is already fully provided, just read it.

When something is missing, use retrieval or a business lookup.

01 Which questions require looking things up

Some questions can go straight to the model, and others need sources:

The test fits in two lines:

When the answer depends on specific material, find the supporting source first.
If the material doesn't contain it, say the information is insufficient.

When in doubt, ask yourself:

Without this material, would the agent's answer be nothing but a guess?

If the answer is yes, look it up first. If the conversation already contains everything, reading it directly is enough and saves a retrieval.

02 Retrieval, file search and RAG form one chain

All three terms address the same problem: how to find the right basis for an answer.

Put together, they form a chain:

User question
  -> find relevant material
  -> hand the material to the model
  -> answer based on the material
  -> flag what's uncertain or insufficient

Each side does what it does well: retrieval finds the evidence, and the model builds an answer from it. When the material is short and fully provided, read it directly. When the library is large, retrieve the relevant parts first and give those to the model. For business data such as the latest order status, query the system that holds it. Implementation details such as vector databases can come later. First get a clear picture of how material gets into an answer.

Now a help-center example. A user asks, "After my membership expires, can I still download my past invoices?" If the agent skips the help documents and relies on common sense, it might say "usually yes." The help center's actual rule is "invoices can be downloaded within 30 days after membership expires, and after 30 days you need to contact support." The user says "past invoices," while the document may say "billing record download." The keywords don't match exactly, and semantic search can link related content by meaning even when the words differ.

Finding "billing record download" is a step, not the finish. You still need to open the clause and see what it lets users download. If it only mentions spending details, there's still no basis for invoices. Retrieval gives you candidate material, and before answering you check the file type, who it applies to and any time limits.

03 Choose the material you hand over

Handing all the material to the agent seems to save the step of choosing. But when irrelevant content, outdated rules and the current basis are mixed together, the key condition gets harder to find. Start with four questions:

Which material does this task need?
What question does each kind of material answer?
Which material can't serve as a basis?
What happens when material is outdated or conflicting?

For example, a customer-service policy agent could organize its material like this:

Mix in marketing copy, internal chatter and outdated policies, and the agent may cite the wrong basis. Check each item for relevance, authority, timeliness and access permission. When marketing copy says "refunds supported," go find the actual policy, because the policy itself decides which rule applies. Internal handling records may help, but first confirm that the current user is allowed to see them.

04 Read the version's scope

Suppose a company has two versions of its refund policy:

2024 version: refundable within 7 days of delivery
2026 version: some categories refundable within 15 days of delivery

An agent that can't tell the versions apart will cite the outdated rule. Choosing the right version depends on three things:

1. Effective date: the 2026 version took effect on January 1
2. Scope: it covers only orders delivered after that date, in some categories
3. Event date: when the order was delivered

So an order delivered in December 2025 still follows the old 7-day rule, while one delivered in March 2026 also needs a check on whether the item falls into a category the new version covers. This works when the policy states its scope clearly, and the publication date is only one input. What if the material says nothing about a transition period? Say "version to be confirmed," ask for verification, and list the two versions' rules side by side.

Try it on a concrete case:

Old version: all categories refundable within 7 days of delivery.
New version: applies only to books delivered after January 1, 2026, with a 15-day period.
Order: delivered in March 2026, category not provided, 10 days since delivery.
Question: can you judge it refundable under the new version?

The delivery date meets the new version's condition, but the category is still unconfirmed, so look up the order's category before choosing a rule. If you already have the full policy and the order information, just check them directly and skip the extra retrieval.

05 When the material falls short, leave the gap visible

A reply with a basis keeps the gap in view (simulated):

No explicit refund clause was found in the current material. The contract attachments need to be added, or a person should confirm.

Anthropic's guide to reducing hallucinations lists giving the model explicit permission to say "I don't know" as its first basic strategy, and suggests having it pull direct quotes from the material before answering, which fits the approach above. You can give an agent rules like these:

Answer only from the material provided.
When the material gives no explicit basis, mark "insufficient information."
Facts in the files take priority over the model's general knowledge.
Every conclusion must be traceable to specific material.

Now a customer-feedback case. (For how to break this kind of task into steps you can check one by one, see making an agent work in steps.) You give the agent 200 pieces of customer feedback and ask what users are least happy about. It might find only three quotes, all mentioning delivery delays:

3 quotes -> can show:    delivery delays occurred
         -> can't show:  they're more common or more serious than other issues

To decide which issue ranks first, fix what you're comparing, how often it appears or how bad its consequences are, and then go through all 200 using one set of categories and counting rules. With only these three pieces of evidence, the conclusion you can deliver is "a delivery delay problem was found." Keep the issues you found separate from a complete ranking, and the report stays within what the evidence supports.