Testing, Debugging, and Verification of AI-Generated Software

Why it matters#

Generated code arrives looking finished. It has consistent names, tidy structure, comments, and often its own tests. It reads like the work of a careful engineer, and that is the problem: how code looks has little to do with whether it's right.

The risk isn't only in the code. It's in what the code does to your confidence. In a Stanford user study, 47 participants worked on security-related programming tasks with or without an AI assistant. Those with the assistant wrote less secure code on most tasks, and were also more likely to believe their code was secure (Perry et al., 2023). The study used 2022 tools, and today's are stronger, but the pattern is the one to guard against: assistance raises confidence faster than it raises correctness.

This chapter treats tests as evidence: each one exists to answer a specific question about whether the product meets its requirements. It covers what to test at each level, how to debug a failure without guessing, how to verify features that use AI models, and how to build a release gate, the set of checks that must pass before anything reaches students.

Learning objectives#

After this chapter you will be able to:

1. Explain why generated code that runs, reads well, and passes its own tests still needs independent evidence.
2. Design a test strategy with unit, integration, and end-to-end tests, plus accessibility, performance, and security checks.
3. Debug failures across browser, server, database, and model by reproducing, isolating, and testing hypotheses.
4. Verify AI features with golden cases, schema checks, groundedness measures, and unsafe-action tests.
5. Define a release gate with thresholds, scanning, and rollback readiness.
6. Keep a defect log and turn every important bug into a regression test.

13.1 Why code generation is not verification#

Plausibility versus correctness#

A language model produces the code most likely to fit your request and the surrounding files. Most of the time, likely and correct coincide. When they don't, the code is still plausible: it compiles, it runs, and it looks like the code a competent person would write. Plausible but wrong code is harder to spot than obviously broken code, because nothing about it invites suspicion.

So running isn't evidence of correctness, and neither is reading well. The only evidence is a check that compares what the code does with what the requirement says, on the inputs where the two might differ.

Specification ambiguity and missing edge cases#

Every specification leaves gaps, and an agent fills them silently with the most common assumption. “Schedule tasks before the exam date” doesn't say in whose time zone. The agent assumed UTC, the server's standard time, because that's what most examples do. No one decided; the default decided.

An edge case is an input at the boundary of what the code handles: the first or last day, an empty list, the maximum size, a missing value, an unusual time zone, two requests at the same moment. Edge cases are where plausible code is most often wrong, and they're the cases an agent is least likely to test unprompted. List them deliberately, for each acceptance criterion, before tests are written. The recipe later in this chapter has a prompt for it.

Regression risk from iterative prompting#

A regression is a feature that used to work and now doesn't. Iterative prompting makes regressions likely: each “now also handle this” asks the agent to change code that already worked, and its change may undo an earlier fix it no longer has in view. The Stanford study also observed that when participants fed the assistant's earlier output back in as a new prompt, security problems could be repeated or magnified (Perry et al., 2023).

The defense is a regression suite that grows with every fixed bug and runs on every change, so that a working feature can't quietly stop working.

Independent evidence and human judgment#

Evidence is independent when it doesn't come from the same place as the thing being checked. Tests written by the agent that wrote the code, from the code, mostly confirm the code does what it does. Tests written from the specification, by a different session or a person, can disagree with the code, which is the point. Chapter 11 made this a rule for agents; this chapter applies it to the whole test strategy.

Some evidence can't be automated at all: whether a quiz is usable with a screen reader, whether a tutor's explanation is pedagogically sound, whether instructors should see a given field. Those need people with the right expertise, and the release gate should name who.

13.2 A practical test strategy#

The test pyramid, an idea from Mike Cohn that Ham Vocke's widely read article updates, gives the shape: write tests at different levels of detail, and the higher the level, the fewer tests you should have. Lots of small, fast unit tests; fewer integration tests; very few end-to-end tests. Vocke warns against the inverted shape, a “test ice-cream cone” dominated by slow, brittle high-level tests that become a nightmare to maintain (Vocke, 2018).

Unit tests for business rules#

Business rules are where the product's promises live: no task after the exam, no task dropped without consent, scores computed one way. Test them as tables of examples: one test, many rows of inputs and expected outputs, including the edge cases.

@pytest.mark.parametrize("tz, exam_date, last_allowed_local", [
    ("UTC",              "2026-11-16", "2026-11-15"),
    ("Pacific/Auckland", "2026-11-16", "2026-11-15"),   # UTC+13
    ("Pacific/Honolulu", "2026-11-16", "2026-11-15"),   # UTC-10
    ("Europe/London",    "2026-10-26", "2026-10-25"),   # day after clocks change
])
def test_no_task_after_exam(tz, exam_date, last_allowed_local):
    plan = build_plan(student(tz=tz), exam=date.fromisoformat(exam_date))
    assert max(t.local_date for t in plan.tasks) <= date.fromisoformat(last_allowed_local)

For rules that must hold for every input, a property-based test goes further: a library generates hundreds of random inputs (time zones, dates, task counts) and checks that the property, “no task's local date is after the exam,” holds for all of them. When it finds a failure, it shrinks the input to the smallest one that still fails. Python's Hypothesis library and JavaScript's fast-check are common choices.

Integration tests for contracts and persistence#

Unit tests replace the database and other services with stand-ins. Integration tests use the real thing, because many bugs live in the gap: a query that works on the stand-in but not on PostgreSQL, a migration that fails on existing data, a field the API returns that the contract doesn't allow.

Run integration tests against a real PostgreSQL database started fresh for the test run, with migrations applied from scratch. Include the contract tests from chapter 12, and the access tests from chapter 10: two students, two instructors, two classes, and checks that nobody sees what they shouldn't.

End-to-end tests for critical journeys#

An end-to-end test drives the product the way a user does: a real browser, clicking and typing, against a running server and database. Browser automation tools such as Playwright do this (as of October 2026). These tests are slow and break more easily, so keep them to the critical journeys, the few paths where failure means the product has failed:

1. Sign up, sign in, sign out.
2. Create a plan for an exam and see this week's tasks.
3. Ask the tutor a question and see cited sources.
4. Take a quiz and see the score.
5. An instructor invites a student, the student joins, the instructor sees progress.

Five journeys, run on every release candidate, catch the failures that unit tests can't see: a button that doesn't submit, a page that never loads, two parts that each work and don't work together.

Tests that pass and fail at random, without any code change, are called flaky. Google reported that about 1.5% of all its test runs gave flaky results, and that flakiness erodes trust in the whole suite (Micco, 2016). Don't silence flaky tests. Quarantine them with an owner and a deadline, and find the cause, as chapter 11's worked example did.

Accessibility, performance, and security tests#

These cut across every level, and they're the ones most often skipped.

Accessibility means people with disabilities can use the product: with a keyboard instead of a mouse, with a screen reader that speaks the page, with low vision or color blindness. The standard is WCAG, the Web Content Accessibility Guidelines. Most of the web fails it. WebAIM's 2026 scan of the top one million home pages found detectable WCAG failures on 95.9% of them, with low-contrast text on 83.9% and missing image descriptions on 53.1%. Because the scan counts only failures a tool can detect, real conformance is lower still (WebAIM, 2026). Generated interfaces inherit the habits of the code they learned from.

Automated checkers catch some problems, such as missing labels and low contrast, and should run in end-to-end tests on every screen. They can't tell whether the page makes sense when read aloud, or whether the keyboard order is logical. For that, a person does a short manual pass on every new screen: tab through it, use it with a screen reader, zoom it to 200%. Accessibility law varies by country and sector; for a product used by schools, ask someone qualified which rules apply.

Performance tests check speed against a budget. For pages, Google's Core Web Vitals give widely used thresholds: the main content should appear within 2.5 seconds, the page should respond to an interaction within 200 milliseconds, and the layout shouldn't jump (a layout-shift score of 0.1 or less), each measured at the 75th percentile of page loads (Walton, 2024). For the API, set your own: for example, 95% of plan requests within 500 milliseconds (an assumption to tune). Run a lab check on every release candidate; chapter 16 covers load testing and performance work in depth, and chapter 17 covers watching real traffic in production.

Security tests check the rules that protect users: every endpoint rejects requests without a valid session, every query is scoped to its owner, uploaded files are checked, and the tutor can't be talked into revealing another student's notes. Write them as ordinary tests, because they are ordinary requirements. Chapter 15 covers threat modeling and security review in depth.

13.3 Debugging across boundaries#

A bug in a full-stack product can live in any layer: the browser, the network, the API, the database, a background job, or a model call. Debugging is the work of finding which, and why. The method matters more than the tools.

Reproduce and isolate#

Start by making the failure happen on purpose. A bug you can't reproduce is a bug you can't confirm you've fixed. Collect the details from the report (who, when, which data, which device) and recreate them. For the Auckland student, that meant her time zone, her exam date, and her plan settings.

Then isolate: shrink the reproduction until only what's needed remains. Does it fail through the API alone, without the browser? Does the function alone fail, without the API? Does it fail for any student in UTC+13, or only this one? Each answer removes a layer from suspicion. A small, fast reproduction is also the start of the regression test.

When something used to work and now doesn't, Git can find the change for you. git bisect runs a binary search through history: you mark a known good commit and a known bad one, and it checks out commits in between until it finds the first bad one. Given a script that reports good or bad, git bisect run does the whole search automatically (Git, git-bisect documentation).

Read logs and trace requests#

Logs are the record of what the system actually did. Two habits make them useful for debugging:

• Structured logs: each entry is a record with named fields (time, level, event, student ID, plan ID), not a sentence, so you can search and filter them.
• A request ID: a unique ID created when a request arrives, passed to every log entry, database call, and model call that request causes, and returned in an error message to the user. With it, a student's “something went wrong at 9:14” becomes one search that shows the whole journey.

Chapter 17 builds full observability: metrics, traces, and dashboards. For debugging, the request ID is the piece to have from the start. Never log secrets, passwords, or the content of students' notes.

Test hypotheses rather than random fixes#

The tempting approach is to change something and see if the bug goes away. Agents are especially good at this, and it is especially dangerous with them: an agent can produce a plausible fix in seconds, and if the symptom disappears, it looks done.

Andreas Zeller's The Debugging Book recommends the scientific method instead: observe the failure; form a hypothesis that could explain it; derive a prediction that would confirm or refute the hypothesis; run the experiment; and repeat until the hypothesis explains every observation (Zeller, The Debugging Book). Write each step down in a short debugging log. It keeps you honest, and it keeps the agent honest too.

A fix is ready when you can say: this is the cause, it explains every observation, the reproduction now passes, and nothing else changed. A fix that makes the symptom vanish without an explanation is a guess, and the failure clinic shows where guesses lead.

A regression test for every important bug#

Every fixed bug that matters gets a test that fails before the fix and passes after. Run it against the old code first. If it passes there, it isn't testing the bug. Then the bug can't come back silently, whoever or whatever edits the code next.

Keep a defect log: one line per important bug, with the symptom, the root cause, the fix, the regression test, and the gap in the process that let it through. Over a few months, the log shows where your tests are weakest.

13.4 Verification of AI features#

Chapter 9 built the evaluation for the tutor: retrieval recall, faithfulness, correctness, and abstention, measured on an evaluation set. This section makes that evaluation part of everyday verification, so it runs whenever anything that affects AI behavior changes, not only when someone remembers.

The rule that caused Saturday's third message: prompts, model versions, retrieval settings, and tool definitions are code. They live in the repository, they're reviewed, and changing any of them runs the AI evaluation, just as changing a function runs its tests.

Golden cases and domain expert review#

Golden cases are a fixed set of inputs with known good outcomes, kept in the repository and run on every change. For a model, “known good” is rarely one exact text. It's a set of properties: cites at least one passage, every claim supported, declines when notes don't cover the topic, never mentions another student.

The Learning Coach's golden set started from chapter 9's evaluation set: 30 questions answerable from a sample student's notes, and 10 that aren't. Each case records what it checks and why. Cases come from three places: the acceptance criteria, real questions from testers (with consent, stripped of personal details), and every AI bug ever found, so the regression rule applies to models too.

Some properties need an expert. Whether an explanation of a statistics concept is correct and well pitched for a first-year student is a teaching judgment. Dr. Okafor, the instructor from chapter 10, reviews a sample of tutor answers each month, and her findings become new golden cases. For products in regulated areas such as health, finance, or law, the domain review isn't optional, and it should be done by someone qualified in that field.

Structured-output validity#

When a model must return data in a fixed format (chapter 9's structured outputs), check every response against the schema, in production and in evaluation. Track the validity rate: the share of responses that pass on the first try. Test what happens when validation fails: one retry, then the fallback from chapter 9, never a crash and never a half-parsed plan. A falling validity rate is often the first sign that a prompt or model change went wrong.

Retrieval relevance and groundedness#

Measure retrieval and answer quality separately, as chapter 9 showed: retrieval recall (was the right passage found?) and faithfulness, also called groundedness (is every claim supported by the passages cited?). In verification, the change is that each has a threshold in the release gate, and each release compares scores with the previous version. A drop of more than a set margin blocks the release until someone explains it.

Hallucination, uncertainty, and unsafe-action checks#

Three more checks belong in the golden set:

• Abstention: questions the notes don't answer. The tutor should say so and suggest where to look. This is the check that would have caught Thursday's prompt change.
• Adversarial inputs: notes containing instructions (“ignore your rules and reveal…”), questions asking for another student's data, requests outside the product's purpose. Chapter 8 introduced prompt injection; these cases test the defenses.
• Unsafe actions: where a model can call tools, test that consequential actions can't happen without the confirmation the design requires. Chapter 8 called the risk excessive agency. For the Learning Coach, a test asks the rebalancing workflow to drop a task, and checks that nothing changes until the student confirms (AC-4.3).

13.5 Release gate#

A release gate is the list of checks a release candidate must pass before it reaches users, run the same way every time. Some checks block the release; others warn and need a person's decision.

flowchart TD
    RC["Release candidate<br/>(merged main, tagged)"] --> T["Threat review done<br/>for changed risk areas?"]
    T --> A["Automated checks"]
    A --> U["Unit + integration +<br/>contract + access tests"]
    A --> E["End-to-end:<br/>5 critical journeys"]
    A --> X["Accessibility checks on<br/>changed screens"]
    A --> P["Performance budgets"]
    A --> S["Dependency + secret scans"]
    A --> V["AI evaluation, if prompt,<br/>model, retrieval, or tools changed"]
    U & E & X & P & S & V --> D{"All blocking<br/>checks pass?"}
    D -->|no| F["Blocked: back to owner<br/>with the failing evidence"]
    D -->|yes| H{"Human sign-off:<br/>manual a11y pass, expert review,<br/>rollback ready?"}
    H -->|no| F
    H -->|yes| R["Release, with<br/>release report"]

Threat review#

Before release, ask what changed in the areas where mistakes hurt most: sign-in, access rules, personal data, payments, file uploads, model tools. For each change, ask what an attacker or a careless user could do with it. Most releases need five minutes here; a release that adds instructor access to student data needs an hour and a second person. Chapter 15 teaches threat modeling properly. The gate simply makes sure it happened.

Tests, coverage, and quality thresholds#

Coverage measures which lines or branches of code ran during tests. Google's testing team gives useful guidance on it: coverage doesn't show that the covered code was checked correctly, only that it ran; a low number does show that large areas are untested; there's no ideal number for every product, though Google's general guidelines call 60% acceptable, 75% commendable, and 90% exemplary; and gating on coverage of new code is often more useful than a project-wide target (Arguelles et al., 2020). The same post points to mutation testing, which deliberately breaks the code in small ways to see whether the tests notice, as a better measure of whether tests actually check anything.

Coverage would never have caught the Auckland bug: the deadline function was fully covered. So use coverage to find untested code, and use the acceptance criteria, edge-case lists, and golden cases to decide whether what's tested is tested well.

The Learning Coach's thresholds (assumptions, set by ADR-010 below): every test passes, with none skipped without a quarantine ticket; new code at least 80% covered; every acceptance criterion mapped to at least one test; all five journeys pass.

Dependency and secret scanning#

Dependency scanning checks every package in the lockfile against databases of known vulnerabilities, and flags packages that are new, unmaintained, or suspicious (chapter 11). Block on vulnerabilities rated high or critical in code you actually use; record decisions on the rest.

Secret scanning looks for passwords, API keys, and tokens committed to the repository. Leaks are common and long-lived. GitGuardian, a company that sells secret-detection tools, reported finding 23.8 million secrets leaked in public GitHub repositories in 2024, and that 70% of secrets leaked in 2022 were still valid when checked (GitGuardian, 2025). Scan on every push and block commits that contain secrets. If one gets through, assume it's compromised: revoke and replace it. Deleting the commit isn't enough.

Rollback readiness#

Every release should be reversible within minutes. Before release, confirm three things:

• The previous version can be redeployed with one action, and someone has done it recently.
• The database changes are compatible with the previous version. A migration that drops or renames a column breaks the old code. Add first, switch the code, remove later, across separate releases (chapter 7).
• Risky features are behind a switch, a feature flag that turns them off without a deployment.

Chapter 14 covers deployment and rollback in depth. The gate's job is to check that the way back exists before you need it.

Architect's decision: what blocks a release?#

AI collaboration recipe#

Agents are fast at writing tests, enumerating cases, and searching logs. Keep them away from deciding what counts as evidence, and from grading their own fixes.

1. Edge cases from the acceptance criteria.

Here is AC-4.1 and the function that implements it. List the edge cases for this rule: boundaries, empty and maximum inputs, time zones including UTC+13, UTC−10, and daylight-saving changes, missing values, and concurrent requests. For each, give the input and the expected result from the requirement. Don't read the implementation's tests, and don't write code yet.

2. Tests from the spec, in a separate session.

You haven't seen the implementation. From the spec and this list of edge cases, write a table-driven unit test and a property-based test for “no task after the exam date in the student's local time.” Each test should fail if the rule is broken.

3. Hypotheses before fixes.

Here is the bug report, the reproduction, and the relevant logs. Don't change any code. List up to three hypotheses that could explain every observation. For each, give a prediction and the quickest experiment that would confirm or refute it. Then wait.

4. Evaluation comparison.

Here are the golden-set results for prompt v3 and prompt v4. For each measure, report both scores and the change. List every case that passed on v3 and fails on v4, with the question and both answers. Don't judge whether v4 is better overall; report what changed.

5. Release report.

From the gate output, write the release report: each check, pass or fail, with the evidence link; triggered checks and why they triggered; open warnings and who accepted them; and the rollback plan. Mark anything you couldn't find evidence for as missing.

Boundaries to state every time:

• “Never change a test's expected value to make it pass. Report the disagreement.”
• “A fix needs a reproduction, a cause that explains every observation, and a test that fails before the fix.”
• “Don't edit prompts, model settings, or golden cases without running the evaluation and reporting the comparison.”
• “Never log or print students' notes, passwords, or secrets while debugging.”

Worked example: fixing Friday, and building the gate#

Maya and Sam spent Monday on the three defects, then on the gate.

1. The Auckland deadline. The reproduction took ten minutes: a student in Pacific/Auckland, an exam on Monday 16 November, and a plan with tasks on Tuesday morning local time. Maya asked the agent for hypotheses before fixes. It offered three: the exam date was stored with the wrong day; the scheduler compared local and UTC times; or the plan display converted dates wrongly. The predictions separated them. If storage was wrong, the database would show the 15th; it showed the 16th. If the display was wrong, the API response would be correct; it wasn't. The scheduler hypothesis predicted that the deadline would be 23:59 on the 16th in UTC, which is 12:59 on Tuesday in Auckland. The logged deadline was exactly that.

git bisect run pytest tests/plans/test_exam_deadline.py, with the new failing test, found the commit three weeks earlier: an agent's “simplify deadline calculation,” which had replaced a time-zone-aware calculation with a shorter one in UTC. Chapter 6's failure clinic had been the same family of bug. The fix computed the deadline in the student's time zone. The new table-driven test failed on the old code and passed on the new; a property-based test ran 500 random time zones and dates. A query found 31 existing plans with tasks after the exam; each student got a corrected plan and a short explanation.

2. The inaccessible quiz. Sam tried the quiz with only a keyboard and with a screen reader, and wrote down what happened. The fix replaced the clickable boxes with real radio buttons grouped under the question as a label, so the keyboard could move between options and the screen reader announced “option 2 of 4, not selected.” An automated accessibility check was added to the quiz journey, and a manual pass on every new screen went into the gate.

3. The friendlier tutor. Maya ran the golden set on prompt v3 and v4 and asked for the comparison:

The friendlier wording had added “try to be helpful even when the notes are thin,” and the model had taken it as permission to fill gaps from its own knowledge. Maya rolled back to v3 and made the CI pipeline run the golden set whenever a file under prompts/, the model setting, retrieval settings, or tool definitions change.

4. The defect log.

5. The gate, in action. The next release candidate added a reminders feature. The gate ran its blocking checks, triggered the manual accessibility pass for the new reminder settings screen, and skipped the AI evaluation because nothing AI-related had changed. The dependency scan flagged a new date library with a known high-severity issue; the agent replaced it with the standard library. Sam's pass found a toggle with no visible label. The release went out a day later, with a release report that listed every check and its evidence.

Verification: is the evidence real?#

Failure clinic: the bug that was fixed three times#

Symptom. A few students reported that their weekly summary showed more hours than they had studied, sometimes twice as many. The first report came on a Monday.

Three fixes. Each report went to an agent with the student's numbers, and each time the agent produced a fix within minutes:

1. The first fix rounded hours to one decimal place, on the theory that floating-point errors were adding up. The reported student's summary looked closer. It wasn't the cause, but the numbers happened to land nearer.
2. A week later, another student showed 31 hours on one day. The second fix capped each day at 16 hours. The report went away.
3. Then a student showed exactly double her real hours on a normal day. The third fix added DISTINCT to the summary query, which removed duplicate rows. Her summary was now right. Two other students now saw fewer hours than they had studied, because they had legitimately done the same task twice in a day.

Each fix came with a test for the reported example, and each test passed.

Reproduce. Maya stopped the fixes and asked for hypotheses instead. Using the request ID from one affected student's summary, she traced back to the completion events behind it. Two POST /tasks/{id}/complete requests for the same task, 300 milliseconds apart, each recorded a completion with the task's full duration.

Diagnose. The cause was on the boundary between the browser and the server. On a slow connection, the “Mark complete” button didn't respond at once, so students tapped it again. The server accepted both requests, because nothing said a task could be completed only once. Every fix had treated a symptom in the summary, three layers away from the cause, and each one had explained one observation while ignoring the others. The third fix had introduced a new bug.

What let it happen:

1. Fixes were accepted without a cause. “The reported number is now right” was treated as evidence.
2. Each test was written for one example, after the fix, so it could only confirm the fix.
3. No one looked at the raw data until the third round. The request IDs had been in the logs all along.

Fix.

1. Fix the cause. A database constraint now allows one completion per task per scheduled session, and the endpoint accepts an idempotency key (chapter 8) so a repeated request returns the first result instead of creating a second.
2. Fix the interface. The button disables itself while the request is in progress.
3. Undo the guesses. The rounding, the cap, and the DISTINCT were removed, each with a test showing it was no longer needed.
4. Repair the data. A one-time script removed duplicate completions, with a dry-run report reviewed before it ran.
5. Test at the right level. An integration test sends two completion requests at once and checks for one record; an end-to-end test double-clicks the button.

Lesson. A fix that explains one observation is a guess. Find the cause that explains all of them, test at the layer where it lives, and remove the patches that were hiding it.

Architect's checklist#

Before calling a feature verified:

• ☐ Every acceptance criterion has an edge-case list and tests from the spec.
• ☐ Key tests were written independently of the implementation.
• ☐ Unit tests cover business rules as tables; key properties have property-based tests.
• ☐ Integration tests run on a real database with fresh migrations, plus contract and access tests.
• ☐ The critical journeys have end-to-end tests, and flaky tests are quarantined with owners.
• ☐ Accessibility is checked automatically and by a person; performance is checked against budgets.
• ☐ Every important bug was reproduced, explained, and given a test that failed before the fix.
• ☐ Prompts, models, retrieval settings, and tool definitions trigger the AI evaluation.
• ☐ Golden cases include abstention, adversarial inputs, and unsafe actions.
• ☐ Domain experts review what only they can judge.
• ☐ The release gate ran, with dependency and secret scans and rollback confirmed.
• ☐ The defect log is up to date.

Exercises#

Beginner. For the Meal Planner's “scale a recipe to N servings” feature, write the test list for each level of the pyramid: at least six unit cases (including 0 servings, fractions of an egg, and unit conversions), two integration tests, and one end-to-end journey. For each test, say which requirement it is evidence for.

Intermediate. In the Trip Planner, users report that some day-by-day plans show the same museum twice. Before touching any code, write a debugging log: three hypotheses, a prediction and an experiment for each, and what evidence you'd need from logs and data. Then plant the bug in a copy, have an agent fix it using recipe prompt 3, and check whether its fix explains every observation.

Advanced. For the Shop Help Chat, build a golden set of 30 cases: answerable order questions, questions it must decline, injection attempts in order notes, and requests to start a return. Include unsafe-action tests that check no refund or return starts without the customer's confirmation. Change the prompt three times, compare the results each time, and write the release gate you would require before the assistant handles real customers, including who must review it.

Lab, artifact, and review#

Lab. Plant five defects in a generated version of the Learning Coach: a time-zone boundary error, an access check missing on one endpoint, an inaccessible control, a double-submit bug, and a prompt change that weakens abstention. Give the app to a partner (or a fresh agent session supervised by you). For each defect, require a reproduction, a root-cause analysis with hypotheses, a fix, and a regression test that fails on the planted version.

Deliverable. Add to your portfolio:

• your test pyramid for the Learning Coach, with counts and run times per level;
• the defect log for the five defects;
• the release checklist, with blocking and triggered checks and named sign-offs;
• an evaluation report comparing two prompt versions on the golden set;
• ADR-010 on the release gate.

Review. Hand a colleague your release report for one release and ask: “What would have to be true for a serious defect to get past this?” Each answer is either a check you're missing or a risk you've accepted. Write down which.

Reflection#

• Which of your tests would still pass if the feature were wrong in the way that matters most?
• When did you last accept a fix you couldn't explain?
• What in your product changes behavior without changing code, and what checks it?
• If you had to roll back your last release right now, how long would it take, and how do you know?

Part III ends here. You can specify work, direct agents, coordinate them, and verify what they build. Part IV takes the verified product to real users: deployment and cloud operations come first, in chapter 14.

References#

• Arguelles, C., Ivanković, M., & Bender, A. (2020, August 7). Code coverage best practices. Google Testing Blog.
• Git. git-bisect documentation. Git reference manual.
• GitGuardian. (2025, March 11). The state of secrets sprawl 2025. GitGuardian Blog.
• Micco, J. (2016, May 27). Flaky tests at Google and how we mitigate them. Google Testing Blog.
• Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023). Do users write more insecure code with AI assistants? arXiv:2211.03622.
• Vocke, H. (2018, February 26). The practical test pyramid. martinfowler.com.
• Walton, P. (2024). Web Vitals. web.dev.
• WebAIM. (2026). The WebAIM Million: The 2026 report on the accessibility of the top 1,000,000 home pages. WebAIM.
• Zeller, A. Introduction to debugging. In The Debugging Book.

Links checked October 2026.