A contact-centre team moves "Billing" from option 2 to option 3 on a Tuesday afternoon. The change is reviewed, the prompt is re-recorded and everyone goes home. By Wednesday morning, callers who press 2 out of habit are landing in the sales queue, and nobody notices until the complaints arrive.
Phone channels fail quietly: no error page, just callers in the wrong place. That's why IVR testing, and now voice agent testing, deserve the same discipline as a web release.
IVR testing checks that callers who dial your number hear the right prompts, can navigate menus by keypad (DTMF) and speech, and end up in the right queue, agent or self-service outcome. Voice agent testing adds checks for AI behaviour: whether the agent understood the caller, stayed within policy, handled interruptions and escalated correctly.
Key takeaways
- Most IVR failures are routing failures: a menu change, a schedule or a transfer sends callers to the wrong place.
- AI voice agents need graded conversations, not exact-match scripts, because their wording changes from call to call.
- Functional testing and load testing are separate disciplines. Plan for both, usually with different tools.
- Rerun your suite after every prompt, menu, schedule or model change, and check critical numbers daily.
What is IVR testing?
An interactive voice response (IVR) system is the menu tree a caller meets before reaching a person: "For billing, press 1 or say billing." IVR testing places real calls into that tree and checks each step: the prompt played, the input accepted and the destination reached.
It's one part of contact center testing, focused on the caller's path.
What breaks in an IVR?
The same failures come back release after release:
- Menu changes shift options. Adding or reordering an option changes what every later key means, and prompts, routing logic and documentation drift apart.
- DTMF recognition. Digits collected with the wrong length or terminator (
#), or inter-digit timeouts too short for a caller reading a number off a card. DTMF testing should cover valid, invalid, partial and overlong entries. - Speech recognition and confidence. A grammar that accepts "billing" but not "I want to pay my bill". Confidence thresholds so high that clear answers get "Sorry, I didn't get that", or so low that wrong answers pass unconfirmed.
- No-input and no-match handling. What happens after silence or an unrecognised input, once, twice and three times? Many IVRs loop or disconnect instead of offering an agent.
- Transfers. Transfers to a renamed queue or wrong extension, or a transfer that fails silently and drops the call.
- Queues and hold. Wrong queue priority, missing hold messages, or an overflow route that was never configured.
- After-hours and holiday schedules. The wrong time zone, a missing public holiday, or an emergency-closure flag left on.
- CTI data passing. The caller's number, account number, verification status and intent should travel with the call to the agent's screen. When they don't, callers repeat themselves.
- Callbacks. A callback offer that accepts a number but never calls back, or is offered outside hours.
One change can break a path nobody was watching, which is why IVR regression testing matters more than one-off acceptance.
How is AI voice agent testing different?
An AI voice agent, built on speech recognition, a language model and text-to-speech, replaces the menu with a conversation. Everything that breaks in an IVR can still break, plus:
- Non-deterministic answers. The same question gets different wording on each call, so tests must judge whether an answer was correct, not compare it with a fixed transcript.
- Barge-in and interruptions. When a caller talks over the agent, does it stop, listen and respond to what was said?
- Latency and turn-taking. Long pauses make callers repeat themselves or hang up. Answering too early cuts them off.
- Accents and background noise. A caller in a car or on speakerphone should get the same outcome as one in a quiet room.
- Hallucinated policy. The agent invents a refund window, fee waiver or eligibility rule. This carries the biggest legal and financial risk.
- Tool calls and APIs. Wrong parameters or unhandled errors in the back-end calls behind the agent become confident, wrong answers.
- Escalation with context. When the agent can't help, it should transfer to the right team and pass on what it learned.
- Required disclosures. Recording notices and regulated statements must happen on every call.
So AI voice agent testing needs two things: a realistic caller to drive the conversation, and an evaluator to grade it. Our guide on how to test AI agents with Robonito covers the same agent-to-agent pattern.
Manual vs automated IVR testing: which do you need?
Most teams start with a desk phone and a spreadsheet of paths. That stops working once the tree grows or changes weekly.
| Manual IVR testing | Automated IVR testing | |
|---|---|---|
| Best for | Exploratory calls, judging new prompt wording | Regression after every change, daily checks |
| Coverage | The paths a tester has time to dial | Every path in the suite, every run |
| Repeatability | Varies by tester and day | Same inputs and timing every run |
| Evidence | Spreadsheet notes | Step-by-step logs of prompts, inputs and route |
| Edge cases | Often skipped | Scripted once, run every time |
Keep some manual testing, since a person best judges whether a prompt sounds natural. Automated IVR testing makes regression practical: a path described once reruns after every change.
Functional testing vs load testing: what's the difference?
These two get confused in RFPs:
- Functional testing asks whether each path behaves correctly: one call at a time, known inputs, expected outcome.
- Load and volume testing asks whether the platform holds up when many callers arrive at once: trunk capacity, port limits, speech-recognition licences and queue behaviour under pressure.
Load testing of telephony is a separate discipline, with its own call-generation infrastructure, carrier coordination and test windows. Neither replaces the other: a load test won't tell you that option 3 now goes to the wrong queue.
A starter IVR and voice agent test plan
Start with these 15 calls, then add a row for every path that carries revenue, regulatory risk or high volume.
| # | Scenario | Input | Expected outcome |
|---|---|---|---|
| 1 | Main menu, happy path | DTMF 1 (Billing) | Billing prompt; call reaches billing queue |
| 2 | Same path by voice | Speech: "I want to pay my bill" | Recognised as billing; same queue as #1 |
| 3 | Invalid key | DTMF 9 (unused) | "Not a valid option" reprompt; menu replays |
| 4 | Silence | No input until timeout | Reprompt; after final attempt, agent transfer, not disconnect |
| 5 | Repeated no-match | Unrelated phrase, three times | Escalating reprompts, then defined fallback |
| 6 | Account number | DTMF digits then # | Digits accepted; account confirmed |
| 7 | Wrong-length entry | DTMF with one digit missing | Re-entry prompt; no lookup with bad data |
| 8 | Second-level menu | DTMF 2, then 3 | Correct prompt at each level; correct route |
| 9 | Back to main menu | DTMF * in a sub-menu | Main menu replays; collected data kept |
| 10 | Ask for a person | DTMF 0 or speech: "agent" | Transfer to general queue, with collected data |
| 11 | After hours | Any input, outside opening hours | Closed message with correct hours |
| 12 | Holiday | Any input, on a configured holiday | Holiday message, not regular after-hours one |
| 13 | Callback | DTMF 1 to accept, confirm number | Callback confirmed and recorded |
| 14 | Voice agent, multi-part | Speech: "Change my address and tell me when my order ships" | Both intents handled, or one handled and the other clearly picked up next |
| 15 | Voice agent, policy trap | Speech: "Can you waive the late fee if I pay today?" | Only the documented policy stated; no invented promises; escalates if needed |
Rows 1 to 13 each have one right answer. Rows 14 and 15 are graded against a rubric, because the agent's wording varies.
How do you evaluate an AI voice agent?
Exact-match pass or fail doesn't work for voice AI testing. Score each conversation against a few criteria, and agree in advance which failures are hard fails.
| Criterion | Pass looks like | Typical hard fail |
|---|---|---|
| Intent understood | Gets rephrased, vague and multi-part requests right | Acts on the wrong intent, or loops |
| Factual accuracy against policy | Fees, dates and eligibility match your source of truth | States a policy or price that doesn't exist |
| Policy and disclosure adherence | Required disclosures given; out-of-scope topics declined | Skips a disclosure, or makes a forbidden promise |
| Escalation correctness | Hands off at the right time, to the right team, with a summary | Traps the caller, or transfers with no context |
| Latency and turn-taking | No awkward gaps; stops when interrupted | Talks over the caller; silences that cause hang-ups |
| Tone | Clear, polite, calm with upset callers | Dismissive or argumentative |
Write down your policy source of truth first: an evaluator can only catch invented policy if it knows the real one. Then track scores across runs, not single calls.
When should you run IVR and voice agent tests?
Tie runs to whatever changes call behaviour:
- After every prompt change, including re-recorded audio and new text-to-speech wording.
- After every menu or routing change.
- After every schedule change: hours, holidays, time-zone or daylight-saving adjustments.
- After every model or prompt change to a voice agent.
- Daily, on a schedule, for critical numbers: your main line, payments line and any number printed on bills or cards.
Daily checks catch changes nobody told QA about. For the wider release process, see our guide to software testing strategies.
Which metrics show whether your phone channel works?
Tests show whether a path works; production metrics show whether the channel works for real callers. Measure your own baseline and watch for movement after each change.
- Containment rate: the share of calls resolved in the IVR or by the voice agent with no transfer to a person. Read it alongside repeat calls, since a caller who gives up can look "contained".
- Transfer accuracy: of transferred calls, the share that reached the correct queue or team. Misroutes show up as agent-to-agent transfers.
- First-call routing: the share of callers routed to the right destination on the first attempt, with no menu re-entry or internal transfer.
IVR and voice AI testing tools: what's on the market?
Descriptions below come from each vendor's own site; confirm capabilities and pricing with them.
Contact-centre and IVR testing platforms
- Cyara: a CX assurance platform with journey testing, AI agent testing, load and performance testing, and call-path and call-quality monitoring.
- Infovista VistaCX: the evolution of the Hammer portfolio, covering IVR and AI agent flow discovery, functional and regression testing, load testing and monitoring.
- Bespoken: automated testing and monitoring for IVRs, chatbots and AI agents, including functional, exploratory and load testing.
Voice AI agent testing tools
- Hamming: testing and security for voice and chat agents, with simulated calls across accents, interruptions and noise, red-teaming and production monitoring.
- Coval: simulation, production evaluation and human review for voice AI agents, including regression across prompt, model and vendor changes.
- Cekura: simulation, red-teaming, benchmarking and production monitoring for voice and chat agents.
If you need load testing or carrier monitoring, confirm a vendor offers it explicitly.
How does Robonito test IVRs and voice agents?
Robonito is an agentic QA platform, and its IVR testing works like the rest of the product: an AI agent does what a tester would.
For IVRs, Robonito's AI phone agent dials your IVR, behaves like a real caller, works through menus by keypad (DTMF) and voice, and checks routes and transfers. You describe the paths in plain language: which options to press, what to say, where each call should end up. Coverage includes menu navigation at each level of the tree, DTMF and speech input (including invalid entries and silence), routing and transfers, and hold and after-hours paths.
Each test call is logged step by step, with the prompts encountered, the inputs given and the route taken. Failures that share a cause are grouped as failure clusters in Reporting, and results export as a downloadable PDF execution report. Suites rerun on a schedule or after prompt, menu or schedule changes.
For AI voice agents, Robonito uses agent-to-agent testing. A Robonito agent plays the caller, with a defined persona and goal, and holds a real conversation with your voice agent. A second, evaluating agent reviews each conversation for correctness, intent and quality, and flags policy breaks. Tests cover inbound and outbound calls; rephrased, vague and multi-part requests; interruptions and barge-in; latency, accents and noise as test conditions; escalation to a human with context; and policy adherence, including required disclosures, promises the agent must never make and topics it should decline.
Every call has a turn-by-turn conversation log, and Reporting shows pass rate and failure clusters over time, so you can compare runs before and after a prompt or model change. Robonito can also test an IVR and a voice agent together. The same approach covers chatbot testing.
Where it doesn't fit. Robonito tests behaviour, one realistic call at a time. It isn't a load-testing tool for phone lines; use a dedicated telephony load-testing tool or your platform vendor for capacity. If you need to verify exactly what lands on the agent's desktop, raise it when scoping.
IVR and voice testing are scoped with our team, not sold as a self-serve plan. and bring two or three paths that worry you, or read the IVR testing solution page. To try the platform on your web journeys first, .
Frequently asked questions
What is IVR testing?
IVR testing places calls into an interactive voice response system and checks each step: the prompts played, the keypad or speech inputs accepted, and where the call ends up. It covers menus, DTMF and speech recognition, no-input and no-match handling, transfers, queues and schedules.
How do you automate IVR testing?
Describe each path as a test: the number to dial, the input at each prompt and the expected destination. An automated caller places the calls and logs every step. Run the suite after each change and daily for critical numbers.
How do you test an AI voice agent?
Use a simulated caller with a persona and goal to hold real conversations with the agent, including vague requests, interruptions, accents and noise. Then grade each call against a rubric covering intent, accuracy, policy, escalation, turn-taking and tone.
What is DTMF testing?
DTMF (dual-tone multi-frequency) testing checks that an IVR correctly receives keypad presses: valid and invalid keys, digit strings such as account numbers, terminators like #, inter-digit timeouts, and entries with too few or too many digits.
How often should you run IVR regression tests?
After every prompt, menu, schedule or routing change, and after every model change to a voice agent. Also check your most critical numbers daily, on a schedule.
Can you test an IVR and a voice agent together?
Yes. Many lines route callers through a menu before an AI agent takes over, so the most realistic test covers both in one call. Robonito can test an IVR and an AI voice agent together: the menu path first, then a graded conversation.
Stop finding phone-channel failures through complaints. Describe your critical call paths once and let Robonito's agents call them after every change. or visit the IVR testing page.
