How to Test AI-Generated Code: A QA Playbook for the Vibe-Coding Era (2026)

The pull request is 1,400 lines long. Claude Code wrote it in an afternoon, along with 38 unit tests, all green. The reviewer scrolls, sees sensible names and tidy functions, and approves. Two days later, support reports that discount codes stack when they shouldn't. Every test passed, because every test checked what the code did, not what the business wanted.

That's the core problem with testing AI-generated code. It looks finished long before it's correct.

To test AI-generated code, verify it against intent, not against itself. Run static analysis and dependency checks, then review what the AI-written unit tests actually assert. Use mutation and property-based testing to measure test strength, add contract tests at API boundaries, and run independent end-to-end tests written from requirements. Finish with a security review.

In AIDLC: when AI builds faster than teams can test, we argued that the bottleneck has moved from writing code to verifying it. This is the practical follow-up: what to check on every AI-assisted pull request, with which tools, and the minimum for solo vibe coders. AI writes the code. Someone still has to prove it works.

Why does testing AI-generated code need a different strategy?

Human code has bugs too. Testing code written by AI differs because of the bugs' shape, and how easily reviewers miss them. Across pull requests from Cursor, Claude Code, GitHub Copilot, Codex and Windsurf, the same patterns recur.

Confident but wrong logic

AI-generated code reads well: clear names, polite comments, textbook structure. That fluency is the trap. An off-by-one in a date range looks exactly as confident as the correct lines around it.

Happy-path bias

Ask for "a refund endpoint" and you'll get one that works for a valid order and amount. A refund larger than the payment, or two requests in the same second, only get handled if someone asked.

Plausible but wrong APIs, and packages that don't exist

Models call methods that sound right but don't exist. Type checks catch some; running the code catches the rest.

Packages are a sharper risk. A USENIX Security 2025 study by Spracklen et al. generated 576,000 Python and JavaScript samples and found that at least 5.2% of packages suggested by commercial models, and 21.7% by open-source models, didn't exist: 205,474 unique made-up names. The authors frame this as a new form of package confusion attack, now often called slopsquatting. An attacker publishes a malicious package under a name models keep inventing, and the next install pulls it in.

Duplicated logic

Agents often don't know a formatCurrency helper already exists three folders away, so they write another. Duplicated logic is easy to miss in review because each copy looks fine on its own. The copies then drift apart, and only one gets the bug fix.

Security patterns that look fine

This is the best-documented risk. Veracode's 2025 GenAI Code Security Report (July 2025) tested 80 coding tasks across more than 100 LLMs: the generated code introduced security vulnerabilities in 45% of cases, and failed to defend against cross-site scripting 86% of the time. Models, Veracode noted, are getting better at coding accurately but not at security.

Academic work agrees. NYU's Asleep at the Keyboard? (IEEE S&P 2022) found about 40% of 1,689 Copilot-generated programs vulnerable. In a Stanford study by Perry et al. (ACM CCS 2023), participants with an AI assistant wrote significantly less secure code, yet were more likely to believe it was secure.

What we see most: missing authorisation checks (logged in, but is it their invoice?), string-built SQL, and API keys pasted into config "for now".

Tests that share the code's blind spots

When one agent, in one context, writes both code and tests, the tests encode the same assumptions. If it misread the requirement, the tests faithfully check the misreading. The AIDLC post called this correlated assumptions, and it's why a green run means less than it used to.

Diffs too big to read

Generating 1,000 lines costs an agent nothing. Reviewing them properly costs a human an hour they rarely spend. Google's 2024 DORA report found that rising AI adoption was accompanied by an estimated 1.5% drop in delivery throughput and a 7.2% drop in delivery stability. That's a correlation, not proof of cause, and DORA points to missing basics: "small batch sizes and robust testing mechanisms".

What does a quality gate for AI-generated pull requests look like?

No single check catches everything. A layered gate gets close, and most layers are cheap and automatic. Put the fastest first so the pipeline fails early.

1. Static analysis, types, linters and dependency scanning

Strict types catch invented methods and wrong signatures. Block new dependencies until someone confirms each one exists, is intended and is maintained. OSV-Scanner checks for known vulnerabilities, and Gitleaks catches hard-coded secrets.

2. Unit tests, and what they actually assert

Don't count tests. Read their assertions. Three red flags:

• Tests that assert the implementation. The expected value is recomputed with the same formula as the code, so it can never disagree.
• Mocks that hide the bug. The payment client is mocked to return exactly what the code expects, so the interesting failure never happens.
• No negative cases. Nothing checks the empty cart, the expired token or the duplicate request.

// Weak: mirrors the implementation
expect(applyDiscount(100, 0.1)).toBe(100 - 100 * 0.1);

// Stronger: states the business rule
expect(applyDiscount(100, 0.1)).toBe(90);
expect(() => applyDiscount(100, 1.5)).toThrow();

3. Mutation testing to check test strength

Mutation testing makes small changes to your code, such as flipping > to >=, and reruns the tests. If they still pass, they weren't checking that behaviour. It's the most honest answer to "are these AI-written tests any good?". Use Stryker for JavaScript, TypeScript and C#, PIT for Java, or mutmut for Python, on changed files only.

4. Property-based testing

Instead of a few hand-picked examples, generate hundreds of inputs and check a rule that must always hold, such as "a refund never exceeds the payment". It finds the edge cases happy-path code ignores. Try Hypothesis for Python or fast-check for JavaScript and TypeScript.

5. Contract and API tests

Agents happily rename a field or change a status code while "cleaning up". Contract tests with Pact, or schema checks against your OpenAPI definition, catch these breaking changes early.

6. Independent end-to-end tests of user journeys

This layer checks intent. Tests for sign-up, checkout, permissions and billing should come from the ticket and acceptance criteria, not the PR's code, and run against a real environment. Our guide to automated regression testing covers choosing and maintaining these journeys.

7. Security review and SAST

Run static application security testing such as CodeQL or Semgrep on every PR, mapped to the OWASP Top 10. Then have a human review anything touching authentication, payments or personal data. Scanners find injection; they rarely notice an endpoint that never checks who owns the record.

Should the AI that wrote the code also test it?

It can help, but it shouldn't be the only check. The principle is independent verification: the agent that wrote the code shouldn't be the sole judge of whether it works.

That means separating two things:

• Intent comes from requirements, tickets and acceptance criteria. It describes what should happen, before any code exists.
• Implementation is what the agent produced. Its own tests describe what does happen.

Let the coding agent write unit tests. But derive at least one layer, usually end-to-end and acceptance tests, from intent, using a different person, a different tool, or at least a fresh agent context that has seen the ticket but not the code. When the two disagree, you've found a bug self-written tests could never surface.

AI code review vs testing: what's the difference?

AI code review tools read the diff and flag style issues, obvious bugs and some security smells, at scale.

But review is reading, and testing is running. No reviewer, human or AI, can tell you whether checkout totals are right for a customer paying with a gift card in another currency. Only running that scenario can. Use AI review to speed up human review, and tests to prove behaviour.

A PR checklist for AI-assisted pull requests

Add this to your PR template for any change labelled AI-assisted.

How do you test a vibe-coded app?

Andrej Karpathy coined "vibe coding" in February 2025 for coding where you "fully give in to the vibes" and stop reading the code closely. It's great for prototypes and risky for a business, because nobody has read the code users depend on.

Vibe coding testing can stay small. Here's minimum viable testing for solo builders and prototypes:

1. Write down five critical journeys in plain sentences: sign up, log in, the core action, payment, account deletion.
2. Automate those five end to end and run them before every deploy.
3. Turn on dependency and secret scanning in your repository.
4. Check authorisation by hand: log in as user A and try to open user B's data by changing the ID in the URL.
5. Try bad input on every form: empty, very long, script tags.
6. Keep secrets in environment variables, never in code or prompts.
7. Ask the agent to break its own tests. "Change one line so these tests fail" is a cheap mutation test.

Production teams need the full seven-layer gate.

What team practices keep AI-generated code quality high?

Teams that ship AI-assisted code safely tend to:

• Keep PRs small. One change per agent task: DORA's "small batch sizes" applied to AI.
• Require acceptance criteria before generation. The criteria become your prompt and your tests.
• Label AI-assisted PRs, so reviewers know where to look harder and you can measure outcomes.
• Track escaped defects by origin. Tag production bugs by whether the change was AI-assisted, and you'll soon see where the gate is weak.
• Make the gate the CI default. Checks that rely on memory don't run. See our guides to CI/CD test automation and software testing strategies.

Where does Robonito fit in QA for AI coding assistants?

Robonito is layer six: the independent end-to-end layer. It doesn't replace unit tests, static analysis, mutation testing or security review. What it does is keep intent separate from implementation:

• Tests from requirements, not code. In AI Studio, you describe a journey in plain language, reference a Jira story or paste a URL. It can also generate tests from requirement documents, Jira epics, Swagger/OpenAPI definitions or the live app. Because the tests start from intent rather than your PR's code, they don't inherit the coding agent's blind spots.
• Runs on every merge from Jenkins, GitHub Actions, GitLab CI or Azure DevOps (Starter and above), in parallel cloud workers.
• Self-healing when the UI changes. Steps adapt when labels, layouts or locators move, and healed steps are logged for review.
• Evidence for every run: video, network traces and an AI root cause for each failure.
• More than web. Web is on every plan; mobile, desktop and API testing are on Pro and Custom.
• Testing your own AI features. Robonito's agents test chatbots and voice agents agent-to-agent, grading for accuracy, policy and safety.

Try Robonito free, or book a demo and bring a Jira story your team recently shipped with AI.

Frequently asked questions

How do you test AI-generated code?

In layers: static analysis and dependency scanning, unit tests whose assertions you've reviewed, mutation testing to measure their strength, property-based and contract tests, independent end-to-end tests written from requirements, and a security review.

Is AI-generated code safe to use in production?

It can be, with checks at least as strict as for human code. Veracode's 2025 report found AI-generated code introduced vulnerabilities in 45% of test cases, so treat AI output as an untrusted draft: scan it, test it independently and review sensitive paths.

Should AI write its own tests?

Yes for speed, no as the only check. An agent's tests share its assumptions, so they can pass while the feature is wrong. Keep at least one layer, usually end-to-end tests, written from the requirements by someone else.

What is vibe coding and how do you test it?

Vibe coding is building software by prompting an AI and accepting its output without closely reading the code. At minimum, automate your five most critical journeys end to end, scan dependencies and secrets, and check authorisation by hand.

What's the difference between AI code review and testing?

AI code review reads the diff and flags likely problems. Testing runs the code and proves whether it behaves as required. Use both: review catches obvious bugs fast, and only tests confirm a real journey works.

Which tests catch AI coding mistakes best?

Mutation testing shows whether AI-written tests detect defects at all. Property-based testing finds missed edge cases. Independent end-to-end tests catch logic that's self-consistent but wrong for the business. SAST and dependency scanning catch security and supply-chain issues.

AI writes the code. Someone still has to prove it works. Robonito checks your critical journeys on every merge, with evidence. Try Robonito free or book a demo.