Path to AI Architect · Chapter 18 of 25
Maintaining and Evolving AI-Built Systems
Manage technical debt, redesign safely, keep architecture and agent context current, and avoid depending on one tool.
27 min read Slides for this chapter
Opening scenario: five kinds of week#
On the Monday after the clocks moved forward for daylight saving time, Dr. Okafor opened her class progress page and found 38 of her 140 students marked Behind. The week before, it had been 6. She did what the page invited her to do: she sent the 38 a kind, worried message offering extra office hours.
By noon, Maya had a dozen emails. Students who were on track, with every task ticked, wanted to know why their instructor thought they were failing. Their own plan pages said On track. Priya forwarded hers with one line: “Which one is lying?”
Maya asked an agent to fix the Behind calculation. It found the bug in minutes: the class progress code computed the start of the week from a fixed offset from UTC, saved when each student joined the class, instead of from the student's time zone. When the clocks changed, the offset was an hour wrong, and Sunday evening's ticks landed in the previous week. The agent wrote a correct calculation into the class progress code, with tests, and opened a pull request.
Sam reviewed it and asked one question: “Where else do we work out what ‘this week’ means?”
The answer took an afternoon to assemble:
| Where | Built in | How it found “this week” |
|---|---|---|
plans |
Chapter 6, fixed then | Student's time zone, Monday 00:00 local |
Weekly summary (summary) |
Chapter 12 | Its own helper, student's time zone, plus an odd 3 a.m. rule |
Class progress (classes) |
Chapter 10 | Fixed UTC offset saved at joining |
Reminders (notifications) |
Chapter 8 | Server time in UTC |
Practice quizzes (quizzes) |
Chapter 12 | The browser's clock, sent with the request |
Five implementations of one rule, written by different agent sessions since launch, each correct for the test cases its session thought of. The pull request would have made six. Chapter 6's failure clinic had already fixed one version of this problem, making sure the student's time zone was really saved and used by plans. Nobody had told the code written later.
There was a second finding. The class progress code didn't ask the plans module for students' progress; it read the tasks table directly, with its own query. So even a correct week calculation in plans couldn't have helped it.
Fixing the dashboard took an hour. Making sure “this week” could never again mean five things took three weeks, and that's what this chapter is about.
Why it matters#
Most of a successful product's life happens after its first release. Since its launch, the Learning Coach has gained classes, quizzes, summaries, shared notes, a university pilot, and thousands of students, and lost some early decisions to time.
AI changes the economics of that life. Writing code is now cheap, and reading it is not. An agent asked to fix or add something writes new code that works, quickly, and rarely spends the effort to find out whether the codebase already has a way to do it. GitClear's analysis of 211 million changed lines from 2020 to 2024 found that lines associated with refactoring fell from 25% of changes in 2021 to under 10% in 2024, while copied and pasted lines rose from 8.3% to 12.3% (GitClear, 2025). More code is being added; less is being consolidated.
Duplication isn't ugly; it's dangerous. When one rule lives in five places, every change to that rule is five changes, and the system disagrees with itself until all five are made. Dr. Okafor's email was the cost.
This chapter is about keeping an AI-built system changeable: recognizing debt, paying down the parts that matter, changing structure without changing behavior, and keeping the knowledge of why things are the way they are where the next person, or the next agent, will find it.
Learning objectives#
After this chapter you will be able to:
- Plan the work of a system after its first release: fixes, features, scale, migrations, and dependency and model updates.
- Recognize architectural debt typical of AI-generated code and prioritize which to pay down.
- Change a system's structure safely with characterization tests, incremental interfaces, and safe schema migrations, and decide between refactoring and rewriting.
- Keep architecture documents, decision records, and agent instructions current as the system changes.
- Hand a system over to new people and new tools without losing what the team knows.
- Produce characterization tests, a refactoring plan, and updated decision records and diagrams.
18.1 The lifecycle after the first release#
Bug fixes, features, scale, migrations#
After launch, work arrives in four streams:
- Fixes: things that don't work as promised.
- Features: new things users or the business want.
- Scale and operations: work that keeps the product running as usage grows (chapters 16 and 17).
- Migrations: moving to new versions of databases, libraries, models, or platforms, or to a new structure.
Features shout loudest; migrations and structural work are easy to postpone until they're urgent. Give the quieter streams a standing share of time. Maya and Sam reserve one day in five for structural work and migrations (a starting assumption, adjusted by what the debt register says), and the error budget policy from chapter 17 can take more when reliability slips.
Product feedback and roadmap decisions#
Chapter 17 built one ranked list from production signals. Roadmap decisions draw on it, plus what students and instructors ask for. The architect's contribution is the cost side: what a feature will cost to build in this codebase, and what it will cost to keep. A feature that needs a sixth definition of “this week” costs more than it looks. Sometimes the best roadmap item is the structural fix that makes the next five features cheap.
Dependency and model updates#
Everything you depend on keeps changing. Libraries release new versions, some with security fixes you need. Platforms change their features. Model providers retire models: Anthropic's documentation, for example, describes models moving from active to legacy, deprecated, and retired, after which requests to them fail, and promises at least 60 days' notice before retiring a publicly released model (Anthropic, Model deprecations). The same page shows that upgrades aren't only a name change: on its newer models, setting certain sampling parameters, such as temperature, to a non-default value returns an error, so code that worked with an older model can break against a newer one. (Provider lifecycles change; last reviewed October 2026.)
Treat updates as routine, not events:
- Update in small steps, often. One library at a time, through the pipeline, with its release notes read. Chapter 17's extraction bug came from a minor version; small steps make the cause obvious.
- Treat model changes as migrations. Pin the version (chapter 9), watch the provider's deprecation notices, run the evaluation set on the replacement well before the retirement date, and roll out behind a flag (chapter 17).
- Keep a dependency calendar: each significant dependency, its current version, its support end date, and who checks it.
Documentation maintenance#
Documentation decays the moment it's written unless it's updated with the code it describes. Chapter 10's rule still holds: when a change alters a rule, update the spec, the instruction file, and the decision record in the same pull request. Section 18.4 turns that rule into checks.
18.2 Architectural debt in AI-generated code#
Technical debt is a metaphor from Ward Cunningham's 1992 experience report: shipping code that isn't quite right is like borrowing, fine if you repay it promptly by improving the code, and dangerous if you don't, because every minute spent working around not-quite-right code is interest on the debt (Cunningham, 1992). Martin Fowler later sorted debt along two lines: deliberate or inadvertent (did you know you were taking it on?) and prudent or reckless (was it a reasonable tradeoff?). He argues that even excellent teams take on prudent, inadvertent debt, because you often understand the right design only after building the system (Fowler, 2009).
AI-assisted development produces a great deal of inadvertent debt quickly. Nobody chose five definitions of a week. Each agent session made a locally reasonable choice. The patterns are predictable, which means you can look for them.
Duplicate implementations and inconsistent abstractions#
Agents duplicate because writing is easier than searching. Joel Spolsky observed long before AI that it's harder to read code than to write it, which is why everyone on a team ends up with their own favorite function for the same job (Spolsky, 2000). An agent with a limited view of the codebase has that tendency at full strength.
Look for:
- Rules implemented more than once: dates and weeks, money, permissions, progress calculations, text cleaning.
- Similar helpers with different names:
get_week_start,week_bounds,start_of_week. - Inconsistent abstractions: three modules talking to the AI provider, each with its own retry and timeout settings (chapter 8 warned about retries at more than one layer).
Hidden coupling and implicit state#
Coupling is how much one part depends on another's internals. Chapter 5 drew module boundaries; agents cross them quietly. Class progress reading the tasks table directly is hidden coupling: nothing in the plans module's interface says classes depends on its table layout, so a change to plans can break classes without anyone knowing.
Implicit state is information a piece of code relies on that isn't passed to it: a global setting, a value cached in memory (chapter 16's failure clinic), the UTC offset saved when a student joined a class and never refreshed. Each makes behavior depend on history rather than on inputs, which is hard to test and hard to see in review.
Incomplete tests and false confidence#
Tests written by the session that wrote the code tend to check what the code does, not what it should do (chapter 11's test that knew it was a test). Each of the five week functions had tests, and each test passed, because each tested its own author's assumptions: no daylight saving change, no student west of UTC, no Sunday at 11:30 p.m. Coverage was high and confidence was false (chapter 13).
Refactoring and modularization priorities#
Refactoring is changing the structure of code without changing its behavior. You can't refactor everything, and you shouldn't try. Prioritize debt where interest is highest:
- Hotspots: code that changes often and is hard to change. Your version history shows which files change most; your experience and your defect log show which hurt.
- Rules that matter to users: duplicated logic in progress, permissions, deletion, and money first.
- Boundaries being crossed: direct access to another module's tables or internals.
Record debt in a debt register, like chapter 15's risk register: what it is, where, the interest it charges (bugs, slow changes, confusion), the fix, and its rough size. Debt in code nobody touches can wait indefinitely; Fowler notes that prudent debt in a rarely touched part of the code may not be worth paying down at all (Fowler, 2009).
18.3 Controlled modernization#
Stabilize behavior with characterization tests#
Before changing structure, pin down behavior. Michael Feathers describes characterization tests: instead of testing what code should do, you record what it actually does. Write a test with a placeholder expected value, run it, see what the code really produces, and make that the expected value. His key point is that a system in production becomes its own specification: users may depend on behavior you'd call a bug, so you need to know whenever you change existing behavior, whether or not you think it's right (Feathers, 2016).
Characterization tests are a natural agent task, with one warning, which the failure clinic expands on: the tests must be generated by running the old code, not by reading it or the spec. An agent that writes tests from its understanding will record its understanding.
Extract interfaces incrementally#
To replace one implementation with another without a risky big switch, use branch by abstraction. Fowler describes it: put an abstraction layer between the code that uses something and the thing it uses; move callers onto the layer one at a time; build the new implementation behind the same layer; switch callers over gradually; then delete the old implementation. Throughout, the system builds and releases normally. He notes a useful variation: run both implementations and check they return the same results (Fowler, 2014).
flowchart TB
subgraph S1["1. Today: five implementations"]
P1["plans"] --> W1["week logic A"]
SU1["summary"] --> W2["week logic B"]
C1["classes"] --> W3["week logic C<br/>+ reads tasks table directly"]
N1["notifications"] --> W4["week logic D"]
Q1["quizzes"] --> W5["week logic E"]
end
subgraph S2["2. One interface, callers moved one at a time"]
API2["plans.calendar.week_bounds()<br/>and plans.progress API"]
P2["plans"] --> API2
SU2["summary"] --> API2
C2["classes"] --> API2
N2["notifications"] --> API2
Q2["quizzes"] --> API2
end
subgraph S3["3. Old logic deleted; boundary enforced in CI"]
API3["plans.calendar"]
CHK["Import check: only plans<br/>may touch plans tables"]
end
S1 --> S2 --> S3
For larger replacements, a whole module or system, the same idea scales up as the strangler fig approach: build new parts alongside the old system and move behavior across gradually until the old one can be removed. Fowler reports that wholesale replacement plans usually go badly, because replacements take long, users can't wait, and the details of existing behavior are hard to discover, much of it behavior nobody actually wants. He also warns that the transitional code needed for old and new to coexist looks like waste, but the lower risk and earlier value are worth it (Fowler, 2024).
Migrate schemas safely#
Structural changes often need data changes. Chapter 14's expand, migrate, contract applies: replacing the stored UTC offset with a time zone name means adding the new column, filling it, reading from it, and only later removing the old one, each step a separate release that the previous code can run against.
Two additions for larger data changes:
- Backfill in batches, during quiet hours, with a lock timeout (chapter 14), so filling a column on 7,000 students or a million activity rows doesn't block the app.
- Verify with queries, not hope. Before the contract step, run a query that counts rows where old and new disagree. Zero is the condition for removing the old column.
Cost and risk of rewrite versus refactor#
AI makes rewriting tempting. An agent can regenerate a module in an afternoon, so why untangle the old one? Because the code isn't where the value is. Spolsky's famous argument against rewrites is that the ugly parts of old code are often bug fixes, each found through real use, and that a rewrite throws that knowledge away and is likely to repeat old mistakes (Spolsky, 2000). An agent rewriting from a spec writes what the spec says, not what months of real students taught the code.
Rewrite only when all of these hold: the part is small and well bounded; its behavior is captured in characterization tests; and the old structure blocks something you need. Otherwise refactor in steps. For the five weeks, that meant one shared function, reached by branch by abstraction, not a rewrite of four modules.
18.4 Preserving architectural knowledge#
Living architecture documents#
Architecture diagrams are only useful if they match the system. The C4 model, which chapter 5 used, organizes diagrams in levels from system context down to containers, components, and code, and is deliberately independent of any notation or tool (Brown, C4 model). That makes it easy to keep diagrams as text (Mermaid) in the repository, beside the code, changed in the same pull requests.
Keep them living:
- Update the diagram in the pull request that changes what it shows. A new module dependency without a diagram change is a review comment.
- Check what can be checked. If the component diagram says
classesdepends only on theplansAPI, a CI check that fails whenclassesimports fromplansinternals keeps the diagram honest. - Review them on a schedule. Once a quarter, compare the diagrams with the code and fix the drift.
Decision logs and tradeoff records#
Michael Nygard's original proposal for architecture decision records observes that a newcomer who doesn't know why a decision was made can only accept it blindly or change it blindly, and both can be harmful. His answer is short records of significant decisions, numbered and never reused, kept in the repository, and, when a decision is reversed, kept and marked superseded rather than deleted (Nygard, 2011).
The Learning Coach has fourteen ADRs. Maintenance adds two habits:
- Supersede, don't rewrite. When a decision is reversed, write a new record and mark the old one superseded. A clarification that leaves the decision standing can be added as a dated note; the original text stays.
- Record rules that have no other home. The week definition, including the 3 a.m. rule the failure clinic uncovers, now lives in a spec, an ADR, and a test. Before, it lived in one function's body.
Updating agent instructions after changes#
The instruction file is architecture knowledge in the form agents read. The AGENTS.md convention treats it as living documentation and has agents read the nearest file in the directory tree, so each module can carry its own (AGENTS.md). It decays like any document, and chapter 10's failure clinic showed what a stale rule does.
After the consolidation, three lines changed:
## Dates and weeks
- Never compute week or day boundaries yourself. Use
plans.calendar.week_bounds(student) and plans.calendar.local_day(student).
- Other modules read plan data only through the plans API (plans/api.py),
never from plans tables. CI enforces this.
And two checks keep instructions honest: a CI step that fails if the instruction files mention a path or command that no longer exists, and a search, run in CI, that flags new week arithmetic (such as adding seven days to a date) outside plans/calendar.py. Instructions advise; checks enforce (chapter 10).
Handoff to new humans and tools#
In April, Lena joined two days a week: a developer finishing a part-time computer science degree, and the third person on call (chapter 17's revisit trigger). Her first week was a test of everything above. Could she understand why the system was the way it was, from the repository alone?
Plan handoffs as deliberately as releases:
- A reading path: the PRD, the C4 diagrams, the ADR index, the instruction files, the runbooks, the debt register, in that order.
- A first task that touches the structure: small, real, and reviewed, not a toy.
- Questions become documents. Every question Lena asked that the repository couldn't answer became a line in a document the same week.
Tools need handoffs too. Coding tools change quickly, and their features, prices, and file conventions with them. Reduce dependence on any one:
- Keep knowledge in plain files in the repository: specs, decisions, instructions, prompts, task cards. The AGENTS.md format exists precisely so one file works across many tools (AGENTS.md).
- Enforce important rules in CI, which applies to every tool and every person, rather than in one tool's settings.
- Avoid making one tool's chat history the only record of a decision. If it matters, it goes in an ADR or a spec.
- Run the exit test once a year: could you switch coding tools in a week? Try one real task with a different tool and note what was missing.
Architect's decision: how do we fix “this week” for good?#
| Context | Five implementations of “this week” across five modules; class progress reads the tasks table directly; a wrong Behind flag reached an instructor and 38 students. Lena starting soon. |
| Constraints | No change in behavior students rely on, except the bug. Releases continue during the work. Two people plus one part-time. Every week rule must handle daylight saving changes and every time zone (AC-11.3). |
| Option A | Fix the class progress calculation in place, as the agent proposed. An hour of work; leaves four other implementations, the direct table access, and the next bug. |
| Option B | Rewrite classes, summary, notifications, and quizzes around a new shared calendar. Clean result; weeks of work in one risky release, and old behavior lost where nobody wrote it down. |
| Option C | Consolidate by branch by abstraction: characterization tests for all five implementations, run against the old code; one plans.calendar module and a plans progress API; move callers one pull request at a time, comparing old and new results in production before each switch; migrate stored offsets to time zone names with expand, migrate, contract; enforce the module boundary in CI; delete the old code. |
| Decision | Option C, recorded as ADR-015, the first record of the week rule; it links to AC-11.3 and ADR-007 (class membership). Ship option A's fix immediately as a stopgap, through the new function. |
| Tradeoffs | About three weeks of part-time work instead of an hour. Transitional code (comparison logging, two implementations side by side) that exists only to be removed. Some rules found by the comparison will need product decisions, not code. |
| Validation | Characterization tests pass against old and new code for every agreed behavior. Comparison logging shows zero unexplained disagreements for one week per caller before its switch. A daylight saving test runs for students in five time zones in CI. The import check blocks direct access to plans tables. |
| Revisit trigger | Any new implementation of date ranges outside plans.calendar (the CI search catches it), a new kind of calendar need (school terms, holidays), or the comparison finding a disagreement nobody can explain. |
AI collaboration recipe#
Agents are excellent at the tedious parts of maintenance: finding every occurrence of a pattern, generating characterization tests by running code, moving callers one at a time, and updating references. They're poor at deciding which odd behavior is a bug and which is a rule. Keep that decision for people.
1. Find the duplicates.
Without changing anything, find every place in this repository that computes a date range, week or day boundary, or converts between time zones. Include SQL queries and frontend code. For each: file, function, what it computes, which time source it uses (student's zone, stored offset, server time, browser time), and its tests. Group them by what they compute and flag every group with more than one implementation.
2. Characterization tests from real behavior.
For the function
summary.week_window, generate characterization tests by running the current code, not by reading it or the spec. Use inputs covering: every hour from Sunday 9 p.m. to Monday 6 a.m. local time; the days around both daylight saving changes; time zones from UTC−10 to UTC+13; a year boundary. Record the actual outputs as the expected values. List any outputs that look surprising, but don't change them.
3. Boundary map.
List every place where a module reads another module's database tables or imports another module's internal files, rather than its public API. For each, propose the API function that would replace it and whether that function exists.
4. Documentation drift.
Compare the C4 component diagram, the instruction files, and the ADR index with the code. List every statement in them that no longer matches, with the evidence. Propose the edits as a single pull request.
Boundaries to state every time:
- “Structure only: no change in behavior unless a test and a person say so.”
- “Characterization tests come from running the old code.”
- “If old and new disagree, report it; don't decide which is right.”
- “Update the diagram, ADR index, and instruction files in the same pull request as the change.”
Worked example: one week, one home#
1. Inventory. Recipe prompt 1 found the five implementations from the opening, plus two more nobody had listed: a SQL view used by the cost dashboard (chapter 17) that grouped activity by UTC week, and a frontend helper that labeled the plan page's date range. Seven in all.
2. Decide the rule. Before writing code, Maya wrote down the rule in the spec, as AC-11.3 extended: a student's week runs from Monday 00:00 to Sunday 24:00 in her current time zone, recomputed on every request, never stored as an offset. Daylight saving changes make that week 167 or 169 hours long, and that's correct.
3. Characterize. Recipe prompt 2 ran against each implementation: 2,400 recorded cases each. The surprises list was short and valuable: the summary's 3 a.m. rule (the failure clinic), the quizzes' trust in the browser's clock, and the reminders arriving an hour off from 7:30 for half the year.
4. Build the home. plans/calendar.py got week_bounds(student) and local_day(student), tested with the characterization cases that matched the decided rule. plans/api.py got progress_for(student_ids, week), so classes could stop reading tasks.
5. Move callers one at a time. Each caller moved in its own pull request, in order of risk: the frontend label, quizzes, reminders, class progress, the cost view, and last the weekly summary. Before each switch, the caller ran both old and new for a week in production, logging disagreements as IDs and timestamps only. Most disagreements were the bug. Each one that wasn't went to Maya for a decision.
flowchart LR
A["Caller uses old logic"] --> B["Add call to new function<br/>alongside old; old result still used"]
B --> C["Log disagreements<br/>IDs and timestamps only"]
C --> D{"Every disagreement<br/>explained and decided?"}
D -- "No" --> E["Person decides: bug or rule?<br/>Rule goes into spec and tests"]
E --> C
D -- "Yes, for one week" --> F["Switch caller to new result<br/>behind a flag"]
F --> G["Delete old logic<br/>after one quiet release"]
6. Migrate the data. The stored offsets in class_memberships were replaced by reading the student's time zone from her profile: add nothing, switch reads, verify with a disagreement query, then drop the column two releases later.
7. Enforce the boundary. An import check in CI now fails if any module other than plans imports from plans/models.py, and the week arithmetic search flags new date math outside plans/calendar.py.
8. Delete and document. The old implementations were deleted. The C4 component diagram lost four arrows and gained one. ADR-015 recorded the decision. The instruction file got the lines shown in section 18.4. The debt register closed one item and opened another: three modules still call the AI provider with their own retry settings.
9. Hand off. Lena's first task, two weeks later: add “this week” to the weekly email. She found week_bounds from the instruction file, used it, and had a pull request open on her first day. Her one question, “why is the summary's week different on Monday mornings?”, was already answered in ADR-015. That was the test passing.
Verification: is the system still changeable?#
| Check | How |
|---|---|
| Time for structure | A standing share of time for structural work and migrations, adjusted by the debt register. |
| Dependencies | Updated in small steps; a dependency calendar with support dates and owners. |
| Models | Pinned; deprecation notices watched; evaluation run on replacements before retirement dates. |
| Duplicates | No important rule implemented in more than one place; CI search for known patterns. |
| Boundaries | Modules use each other's public APIs only; a CI check enforces it. |
| Debt register | Each item has its interest, fix, and size; reviewed with the roadmap. |
| Characterization | Tests recorded from running code before any structural change to untested areas. |
| Incremental change | Large changes done by branch by abstraction or strangler fig, released continuously. |
| Data migrations | Expand, migrate, contract; batched backfills; disagreement queries before removal. |
| Diagrams | Changed in the same pull request as the structure they show; reviewed quarterly. |
| Decisions | ADRs for significant decisions; reversed decisions marked superseded. |
| Agent instructions | Updated with changes; CI checks referenced paths and commands exist. |
| Handoff | A newcomer can make a real change from the repository's documents alone. |
| Tool independence | Knowledge in plain files; rules in CI; the exit test done yearly. |
Failure clinic: the rule that lived in one function#
Symptom. The Monday after the weekly summary moved to the shared calendar, students wrote in about their summaries. Hours studied were down. Several streaks, counted in weeks with at least one completed task, had reset to zero. Jun's summary said he'd done nothing the previous week; he had ticked four tasks on Sunday night, the last at 1:40 a.m.
Reproduce. Sam ran the old summary helper and the new week_bounds for Jun's ticks. The old helper counted anything before 3 a.m. Monday, local time, as the previous week. The new function, following the decided rule, started the week at midnight. Four ticks between midnight and 3 a.m. had moved from last week to this one.
Diagnose. Soon after the summary launched, students had complained that late-night study on Sunday “didn't count” toward their week. A small fix added the 3 a.m. cutoff to the summary helper. It had a commit message, a test buried among the helper's others, and nothing else: no spec line, no ADR, no mention in the instruction file. It was a product rule living in one function's body.
The consolidation should have caught it, and nearly did. Recipe prompt 2 listed it among the surprises. But for the summary, a different session had written the characterization tests earlier, and it had written them from the spec, which said Monday 00:00. Its tests encoded the spec, not the code. The comparison logging, step 5 of the worked example, did flag the disagreement for the summary, but Maya reviewed those disagreements during a busy week, saw that most were the daylight saving bug, and approved the batch.
The pattern is the one Spolsky warned about: the odd line in old code is often a fix someone needed (Spolsky, 2000), and Feathers' point that production code is its own specification (Feathers, 2016). Four things combined:
- The rule existed only in code, so the decided rule in step 2 contradicted it without anyone knowing.
- Characterization tests were written from the spec, not by running the old code.
- Disagreements were approved in bulk, so a real rule was treated as part of the bug.
- The summary was switched with a known surprise unresolved.
Fix.
- Restore and decide. The 3 a.m. rule went back, deliberately this time: Maya decided that for the weekly summary and streaks, tasks completed before 3 a.m. Monday count toward the previous week, while plans and progress use midnight. It became AC-11.4, with examples, and ADR-015 gained a dated note saying so.
week_boundsgained agraceparameter, used only by the summary. - Repair the data. Summaries and streaks for the affected week were recalculated, and students whose streaks had reset saw them restored with a note.
- Characterization from execution only. The recipe now says it explicitly, and the summary's tests were regenerated by running the old helper from version history.
- No bulk approval. Comparison disagreements are grouped by cause, and each group needs its own decision before a switch. A surprise on the list blocks the switch until it's resolved.
- Hunt for other code-only rules. A prompt asked an agent to list conditions in the codebase that change user-visible behavior and aren't mentioned in any spec. It found nine, of which three were real rules; they went into specs.
Lesson. Before you replace code, find out what it knows. Record its behavior by running it, treat every surprise as a question for a person, and move rules that live only in code into a spec, a decision, and a test, where the next refactoring can't quietly remove them.
Architect's checklist#
Before and during structural change:
- ☐ Structural work and migrations have a standing share of time.
- ☐ Dependencies and models are updated in small steps, with evaluation before model changes.
- ☐ The debt register lists the highest-interest debt, with sizes and owners.
- ☐ Duplicated rules are found before fixing any one copy.
- ☐ Characterization tests are recorded by running the old code, and every surprise is resolved by a person.
- ☐ Changes are incremental: branch by abstraction or strangler fig, released continuously.
- ☐ Old and new are compared in production before each switch, with disagreements decided by cause.
- ☐ Data changes use expand, migrate, contract, with batched backfills and disagreement queries.
- ☐ Module boundaries are enforced in CI.
- ☐ Rules that live only in code are moved into specs, decisions, and tests.
- ☐ Diagrams, ADRs, and instruction files change in the same pull request as the code.
- ☐ A newcomer can make a real change from the repository alone.
- ☐ Project knowledge lives in plain files, so changing tools is cheap.
Exercises#
Beginner. In the Meal Planner, three features compute a shopping list's total: the list page, the weekly email, and the budget warning. One rounds each item's price, one rounds the total, and one ignores discounts. Write the inventory table (where, how, tests), decide the single rule, and list three characterization cases that would expose the differences.
Intermediate. The Trip Planner stores each itinerary item's time as local time without a time zone, and now needs to support trips across time zones. Write a migration plan with expand, migrate, contract steps, a batched backfill, and the disagreement query that must return zero before the old column is removed. Then write the ADR, including what it supersedes.
Advanced. The Shop Help Chat has grown three ways of deciding whether a customer may return an item: in the chat's tool, in the returns page, and in a nightly job that closes expired returns. Plan a branch-by-abstraction consolidation: the inventory, characterization tests generated from the running code, the order in which callers move, the comparison you'd run in production, and how you'd find return rules that live only in code. Finish with the instruction file lines and the CI check that keep a fourth implementation from appearing.
Lab, artifact, and review#
Lab. Take a deliberately tangled feature in your Learning Coach (or the supplied one: a progress calculation duplicated in three modules, one of which reads another's tables). Inventory the implementations, record characterization tests by running the current code, consolidate behind one interface using branch by abstraction, and enforce the boundary in CI. No user-visible behavior may change, except bugs you've written down and decided on.
Deliverable. Add to your portfolio:
- the inventory and the characterization tests, with the list of surprises and the decision on each;
- the refactoring plan, step by step, with how each step can be rolled back;
- the updated C4 diagram, before and after;
- ADR-015 (or your equivalent), with what it supersedes;
- the instruction file changes and the CI checks that enforce them.
Review. Give the repository to someone who has never seen it, and ask them to make a small change that touches your refactored code, using only what's in the repository. Write down every question they ask. Each is a missing line in a document.
Reflection#
- Which rule in your product is implemented in more than one place right now?
- What does your system do that no spec, test, or decision mentions?
- When did you last delete code, and what made it safe?
- If your main AI coding tool disappeared tomorrow, what knowledge would you lose?
- Could a newcomer understand why your system is the way it is from the repository alone?
Part IV ends here. The Learning Coach is deployed, secured, able to handle its peaks, visible from the inside, and still changeable after months of fast growth. Part V steps back from the project to the person. Chapter 19 is about the human architect: the judgment, communication, and leadership that every decision table in this book has quietly asked of you, from explaining a design to mediating a tradeoff to deciding when an agent's output isn't good enough.
References#
- AGENTS.md. AGENTS.md: A simple, open format for guiding coding agents.
- Anthropic. Model deprecations. Claude API documentation.
- Brown, S. The C4 model for visualising software architecture. c4model.com.
- Cunningham, W. (1992). The WyCash portfolio management system. OOPSLA '92 experience report.
- Feathers, M. (2016, August 8). Characterization testing. michaelfeathers.silvrback.com.
- Fowler, M. (2009, October 14). Technical debt quadrant. martinfowler.com.
- Fowler, M. (2014, January 7). Branch by abstraction. martinfowler.com.
- Fowler, M. (2024, August 22). Strangler fig. martinfowler.com.
- GitClear. (2025). AI copilot code quality: 2025 look back at 12 months of data. GitClear.
- Nygard, M. (2011, November 15). Documenting architecture decisions. Cognitect blog.
- Spolsky, J. (2000, April 6). Things you should never do, part I. Joel on Software.
Links checked October 2026.
Check your understanding
3 questions
1.The instructor dashboard calculates “this week” wrongly after a clock change. You ask an agent to fix it. What should happen first?
2.You need to refactor tangled code that nobody fully understands. What do you write before changing it?
3.Your team wants to switch to a different AI coding tool. What makes the switch cheap?
Was this chapter helpful?
Cite this chapter
Shi, T. (2026). Maintaining and Evolving AI-Built Systems. In Path to AI Architect: From vibe coding to designing, building, and leading AI-driven software systems (Online edition, Chapter 18). Exciton Research. Retrieved November 2, 2026, from https://excitonresearch.com/education/become-an-ai-architect/maintaining-and-evolving-ai-built-systems
Brief quotations are welcome with attribution and a link.
© 2026 Tian Shi. All rights reserved.