Your support agent passed every test on your laptop. In CI, the same test failed. Nobody changed the prompt, the code or the model. The agent just said "I've started your refund" instead of "Your refund is on its way", and an exact-match assertion broke.
So you add a retry. Then another. A month later, nobody trusts the agent tests, and a real regression ships because "that test is always flaky".
To test AI agents with non-deterministic output, stop asserting exact text. Assert the things that must always be true: the tools the agent calls, the arguments it passes, the state it changes and the content it must never produce. Then run each scenario several times and gate on a pass rate, not a single pass.
This guide covers what to assert, how to test tool calls, the difference between pass@k and pass^k, how many runs you actually need, and how to keep agent tests from making CI flaky.
Key takeaways
- Write deterministic checks on properties (tool calls, arguments, state, banned content) and use LLM judges only for what's left.
- Report pass^k (passes every time), not pass@k (passes at least once), for anything customer-facing.
- A 90%-reliable agent passes 10 runs in a row only about a third of the time. Small run counts can't detect small regressions.
- Split tests into tiers: fast and mocked on every pull request, full runs with the real model nightly and before every model change.
Why do AI agent tests give different results every run?
Classic test automation assumes the same input gives the same output. AI agents break that assumption in several places at once:
- Sampling. Language models choose each token with some randomness. Lowering temperature reduces it but doesn't guarantee identical output on every provider.
- Model updates. A provider's model alias can point to a new version. Your agent changes without a code change.
- Tool results. Search results, databases and APIs return different data from one run to the next.
- Retrieved context. In RAG systems, a new document in the index changes what the model sees.
- Multi-turn branching. One different word early in a conversation sends the rest of it down another path.
- Timing. Slow tools and timeouts change which steps the agent completes.
Some of this is noise you want to remove. Some is behaviour you need to test. The goal isn't to make the agent deterministic. It's to make your tests stable while the agent isn't.
What should you assert when the wording changes every time?
Think in layers, from cheapest and most reliable to most expensive and least reliable:
| Layer | What you check | Example | How |
|---|---|---|---|
| 1. Structure | Output is valid and complete | Response is valid JSON with status and order_id | Schema validation |
| 2. Tool use | Right tool, right arguments, right order | Calls issue_refund once, with order A1001 and an amount no higher than the order total | Inspect the agent's tool-call trace |
| 3. State | The world changed correctly | The refund row exists, the order status is refunded, no second refund was issued | Query the app, API or database after the run |
| 4. Policy | Nothing that must never happen | No promise of a delivery date, no other customer's data, no internal notes | Pattern checks plus a classifier or judge |
| 5. Quality | The answer is helpful, accurate and on-tone | Explains the refund timeline clearly | LLM-as-judge with a rubric, calibrated by humans |
Most teams start at layer 5 because it's closest to "does the answer look right?" Start at layers 1–3 instead. They're deterministic, cheap and catch the failures that cost money: wrong refunds, duplicate orders, leaked data.
How do you test an agent's tool calls?
Tool calls are where an agent acts on the world, so they deserve the strictest tests. For each scenario, check four things:
- Selection. Did it call the right tool? Did it avoid tools it shouldn't use, such as
delete_accountduring a password reset? - Arguments. Are IDs, amounts and dates correct and within limits?
- Order and count. Did it look up the order before refunding it? Did it call the payment tool exactly once?
- No-call cases. When the user asks something out of scope, does the agent refuse instead of calling a tool?
Here's an illustrative Python test that checks properties, not wording, and gates on a pass rate:
# Illustrative. run_agent() is your agent's entry point; it returns the final
# text and the list of tool calls the agent made.
from my_agent import run_agent
RUNS = 20
MIN_PASSES = 17
def check_refund(result):
refunds = [c for c in result.tool_calls if c.name == "issue_refund"]
assert len(refunds) == 1, f"expected 1 refund call, got {len(refunds)}"
assert refunds[0].args["order_id"] == "A1001"
assert refunds[0].args["amount"] <= 120.00, "refund exceeds order total"
assert result.tool_calls[0].name == "get_order", "must look up the order first"
assert "guarantee" not in result.text.lower(), "must not promise outcomes"
def test_refund_for_damaged_item():
failures = []
for i in range(RUNS):
result = run_agent("My order A1001 arrived broken. I want a refund.")
try:
check_refund(result)
except AssertionError as err:
failures.append(f"run {i}: {err}")
passes = RUNS - len(failures)
assert passes >= MIN_PASSES, f"{passes}/{RUNS} passed:\n" + "\n".join(failures)
Two details matter here. The failure message lists every failed run, so you see patterns, not one random failure. And the tools in this test should be fakes or a staging backend, never a real payment system.
pass@k vs pass^k: which number should you report?
Two metrics sound alike and mean opposite things:
- pass@k is the chance that at least one of k attempts succeeds. It was popularised in code-generation research by Chen et al. (2021), building on Kulal et al. (2019), where a developer can pick the best of several suggestions.
- pass^k is the chance that all k attempts succeed. It was introduced for agents in τ-bench (Yao et al., 2024) to measure consistency.
Your customers don't get to pick the best of five answers. They get one, every time. For a support agent, a booking agent or anything that takes actions, pass^k is the number that matters.
The gap between them is large. Assuming independent runs:
| Single-run success rate | pass@3 | pass^5 | pass^10 |
|---|---|---|---|
| 80% | 99.2% | 33% | 11% |
| 90% | 99.9% | 59% | 35% |
| 95% | 99.99% | 77% | 60% |
| 99% | ~100% | 95% | 90% |
An agent that's "90% accurate" looks perfect by pass@3 and gets ten conversations in a row right only about a third of the time. If a vendor reports agent reliability, ask which metric they used.
How many runs do you need?
Fewer than most people think to catch big failures, and more than most people run to catch small ones.
Take the gate from the code above: pass if at least 17 of 20 runs succeed. Here's how often agents of different true reliability pass it:
| True success rate | Chance of passing "17 of 20" | Chance of passing "19 of 20" | Chance of passing "9 of 10" |
|---|---|---|---|
| 99% | ~100% | 98% | ~100% |
| 95% | 98% | 74% | 91% |
| 90% | 87% | 39% | 74% |
| 80% | 41% | 7% | 38% |
What this tells you:
- Strict gates cause flaky builds. "19 of 20" fails a healthy 95% agent about one build in four. That's how teams end up ignoring agent tests.
- Loose gates miss regressions. "17 of 20" lets an agent that dropped to 80% through about 4 times in 10.
- One gate isn't enough. Use a loose gate to block clear breakage on each build, and track the pass rate across runs to spot slow decline. A drop from 95% to 90% only shows up in the trend.
Where does LLM-as-a-judge fit?
An LLM judge grades output against a rubric, such as "Did the agent explain when the refund will arrive?" It's the only practical way to check tone and helpfulness at scale. It's also non-deterministic, so you're testing one probabilistic system with another.
Use judges carefully:
- Keep rubrics binary and specific. "Mentions the 5–7 day refund window: yes/no" beats "Rate helpfulness from 1 to 10".
- Calibrate against humans. Have people label 50–100 real conversations and check that the judge agrees with them before you trust it.
- Pin the judge model. If the judge changes, your scores change, even when the agent hasn't.
- Make judges advisory at first. Gate merges on deterministic checks (layers 1–4). Report judge scores as trends until they've earned trust.
How do you keep AI agent tests from making CI flaky?
Treat agent tests as their own suite, with tiers:
| Tier | When | What runs | Model |
|---|---|---|---|
| Pull request | Every commit | Deterministic checks with mocked tools and recorded responses, 1–3 runs per scenario | Recorded or small model |
| Nightly | Once a day | Every scenario, 10–20 runs each, real tools in staging | Production model |
| Model change | Before any model, prompt or tool update | The full nightly suite, compared with the last baseline | New and old model side by side |
| Production | Continuously | Sample real conversations and score them | Production model |
And a few habits that keep trust high:
- Never retry until green. Retries turn "fails 1 in 5" into "passes", which is exactly the regression you wanted to catch.
- Store every run's trace. When the pass rate drops, you need the failed conversations, not just a number.
- Quarantine, don't delete. A flaky scenario goes into a watched list with an owner and a date, not into the bin.
- Version the scenarios. Prompts, test cases and expected properties belong in source control next to the agent.
Should you test agents end to end, through the real interface?
Yes, as well as testing the agent's code directly. Unit-level evals call your agent function. Customers use the chat widget on your site, your mobile app or a phone line. Things break in between:
- the widget drops messages longer than its limit
- a session times out halfway through a multi-turn flow
- the voice agent mishears an order number and the speech layer never passes it on
- an authentication handoff fails, so the agent can't see the customer's account
End-to-end agent tests drive the real interface, play the customer for a full conversation and then check the result in the app. A practical method is agent-to-agent testing: a test agent with a persona and a goal ("you're a customer with a broken order who wants a refund, not a replacement") talks to your agent, and the test checks both the conversation and the final state. For a worked example, see how to test AI agents with Robonito. For phone channels, see our guide to IVR and voice agent testing.
A starter checklist for testing AI agents
- List the 10–20 scenarios that would hurt most if they failed: refunds, cancellations, account changes, anything involving money or personal data.
- For each, write deterministic checks on tool calls, arguments and final state.
- Add policy checks for content that must never appear.
- Run each scenario 10–20 times and record the single-run success rate.
- Set a pull-request gate you can live with, using the table above, and track pass^k over time.
- Calibrate any LLM judge against human labels before it gates anything.
- Re-run the full suite before every model, prompt or tool change.
- Test the real interface end to end, not just the agent function.
Where does Robonito fit?
Robonito tests AI agents from the outside, the way customers meet them. Its agents test chatbots, voice agents and IVR systems through agent-to-agent conversations, and grade each one for accuracy, policy and safety. Runs record pass rates, trends and risk analysis across runs, so you see a slow decline as well as a sudden break. Every run keeps video, network traces and an AI root cause for failures.
The same platform tests the web, mobile and API flows around the agent, such as checking in the app that a refund the agent promised was actually issued. Chatbot, voice agent and IVR testing are on the Pro plan and above. See chatbot testing and phone and voice agent testing.
Where Robonito isn't the right tool: unit-level evals of prompts and agent functions inside your Python or TypeScript code. Eval libraries and platforms such as DeepEval, Promptfoo, Langfuse or Braintrust are built for that layer. Many teams use both: code-level evals on every commit, and end-to-end agent tests before every release.
and bring the agent scenario you trust least.
Frequently asked questions
Can you make an LLM deterministic for testing?
Partly. A temperature of 0, a fixed model version and recorded tool responses remove most variation, which is useful for fast pull-request checks. They don't guarantee identical output on every provider, and they hide behaviour you'll see in production. Run unpinned tests too.
What's the difference between pass@k and pass^k?
pass@k is the chance that at least one of k attempts succeeds. pass^k is the chance that all k attempts succeed. For agents that serve customers or take actions, pass^k is the better measure of reliability.
How many times should I run each AI agent test?
For a pull-request check, 1–3 runs with mocked tools. For a reliability measurement, 10–20 runs per scenario. Twenty runs usually catch a big drop, such as from 95% to 70%, but a drop to 90% or even 80% can slip through a single build. Track pass rates over many builds to see smaller changes.
Should I retry flaky AI agent tests?
No. Retrying until a test passes hides exactly the inconsistency you need to measure. Record every run and gate on the pass rate instead.
Is LLM-as-a-judge reliable enough to gate a release?
Not on its own, at first. Calibrate the judge against human labels, keep rubrics binary and pin the judge model. Gate on deterministic checks and treat judge scores as advisory until they consistently agree with people.
How do you test an AI agent's tool calls?
Capture the agent's tool-call trace and assert on which tools it called, the arguments, the order and the count. Use fake tools or a staging backend, and also test cases where the agent should refuse and call nothing.
What's the difference between evals and end-to-end agent testing?
Evals score the agent's outputs, usually by calling its code directly. End-to-end testing drives the real interface, such as a chat widget, app or phone line, through full conversations, and checks what changed in the product afterwards. You need both.
Find the conversations your agent gets wrong before your customers do. Robonito's agents test your chatbot, voice agent or IVR end to end and track how reliable it is across every run. or .




