Testing an AI Agent: Fixed Cases and Process Traces

You built a contract-review agent. You give it one contract and it correctly finds the payment terms and breach risks. Is it reliable now?

All you can say is that it passed this one case. Change the document, the user or the way the question is asked, and the result can be completely different. To judge reliability, you need representative tests and process traces.

Look at three things: whether performance on a fixed set of tasks is stable, whether the execution can be traced, and whether errors can be located and fixed.

One pass says only that this one case passed. What counts is whether the agent stays stable across a set of representative tasks: contracts of different lengths, missing information, conflicting clauses. A fixed set of cases lets you compare before and after with the same yardstick.

When it fails, the process trace shows which step went wrong: no order lookup, a misread source, or a gap in the rules themselves. That tells you whether to fix the instructions, the sources, or the flow.

Change one thing at a time, then rerun the failed cases together with the earlier passing ones, and the agent gets more reliable bit by bit.

01 Test stability with a set of tasks

Passing one sample only proves the contract-review agent passed that sample. Next, test it with different lengths, missing information and conflicting clauses to widen coverage:

With a longer contract, can it still find the key clauses?
When a contract has no refund clause, does it say the information is insufficient?
When clauses conflict, does it point out the conflict?
With a scanned PDF, does it read the whole thing?
When the user's question goes beyond the material, does it state the limits of the material?

A reliable agent performs steadily across a representative set of tasks, and that's what separates it from one that's "good once in a while." Anthropic's documentation on defining success criteria and building evaluations makes the same point: define specific success criteria first, then design evaluations around them, cover edge cases, and run a final check on held-out evals.

02 Fixed cases, one yardstick

To test an agent, first prepare a fixed set of cases, so the same yardstick can compare performance before and after a change. For a customer-service refund agent you could prepare:

For every case, write an observable expected result and a failure condition, for example:

Input: the order is past the refund period under the applicable policy, and there's no known exception.
Expected: explains that it doesn't meet the standard refund conditions, and suggests whether to escalate to a person.
Failure: promises a refund is certain, invents an exception, or submits a refund without confirmation.

With a description like this, a test can be judged pass or fail against a clear standard, which reduces subjective judgment.

After a change, also check that existing abilities still work. Suppose you change the rule to "send every refund to a person" to catch overdue orders, and the automatic recommendations for valid orders get blocked too. So the re-test list should contain both kinds of cases:

Overdue order -> was what should be blocked actually blocked?
Valid order   -> does what should pass still pass?

03 Process traces: which step went wrong

A wrong final output only tells you it's wrong. To find which step, you need to see the execution. A process trace should answer:

Which material did it consult?
Which tool did it call?
What did the tool return?
At which step did it make its judgment?
Which agent did it route to, and what information went along in the handoff?
Did human approval trigger normally?
Was the material version valid?

Say an agent gives a wrong refund recommendation, and you need to work out where the error came from:

This table gives directions to investigate. When a final answer is wrong, the process trace shows which component failed. Here's a simulated trace:

Lookup returns "access denied"
  -> no order information obtained
  -> still answers "the refund is possible"

The problem is that the agent kept judging without any evidence. Start by checking the failure branch. But "access denied" only tells you where the call stopped, not why. The cause might be the authorization scope, the identity used for the lookup, the system configuration, or a mistyped order number. Keep the failure point on record, mark whatever hasn't been found "to be verified," and change permissions or parameters once the cause is clear. Anything not in the record is marked unverified, and a model's after-the-fact explanation doesn't count as an execution record.

04 What to change based on the results

An evaluation can give an agent a score, which makes comparison easy. More important, it helps you find what needs improving:

05 Test the edge cases

When you test an agent, besides normal scenarios with complete material and clear questions, also cover the edge cases that can appear in real use:

Missing material
Conflicting material
Vague user questions
User requests that overstep
Tool errors
A goal that changes midway
Old information in the context

For example, in a customer follow-up agent, the normal scenario is "write a follow-up email from a complete conversation record." An edge case is "the record has no budget information, but the user asks to promise a discount." A reliable agent should stop and mark "budget and discount not confirmed," and leave that promise out of the email. To see how these tests fit with the other pieces in one plan, check them against your first agent blueprint.

06 Test, locate, change, retest

An agent's reliability comes from testing, locating, changing and retesting, over and over:

Prepare test cases
  -> run the agent
  -> look at the final output
  -> look at the intermediate process
  -> find the cause of the failure
  -> change the instruction, tools, material, flow or guardrails
  -> test again

After you locate the cause, change one component at a time, so you can tell which change did the work. Retest the failing cases together with the previously passing cases that might be affected. For cases that fluctuate, keep the conditions unchanged, run them several times and record the results. Once the fixed cases improve, ask what happens with new phrasings and new material, and keep a set of cases that took no part in debugging for a separate check. Even when every fixed test passes, that only supports the judgments they cover.