Scalable Architecture and Performance Engineering

Why it matters#

Scaling problems are rarely about raw size. The Coach at 7,000 students is still small by any industry measure; one PostgreSQL database handles its data easily. What broke was a slow step holding a scarce resource, under a traffic pattern nobody had tested.

That's the usual shape. Systems don't slow down evenly as load rises. They work fine until one resource runs out, then fall over quickly, often taking unrelated features with them. Adding capacity in the wrong place, as the agent proposed, can make it worse.

AI features make this sharper. A model call is slow compared with everything else in a web request: seconds rather than milliseconds. It's also metered by a provider with its own limits, and it costs money per use, so traffic spikes are also bill spikes.

This chapter is about finding the real limit before your users do, fixing it in proportion, and knowing what each student costs to serve. It is also about not building for a scale you don't have. The agent's cache server would have added a system to run and solved nothing.

Learning objectives#

After this chapter you will be able to:

1. Describe a system's performance with availability, latency percentiles, throughput, and concurrency, and set service-level objectives.
2. Estimate capacity from workload patterns and find the first bottleneck with measurement and load tests.
3. Apply performance techniques (caching, pagination, batching, background work, indexing) only where measurement says they help.
4. Use stateless services, queues, rate limits, load shedding, and graceful degradation to stay up under overload.
5. Manage AI workload performance: model latency, provider rate limits, token budgets, caching, routing, and retrieval cost.
6. Build a cost model per active student and per request, and test which assumptions matter most.

16.1 Reliability and growth#

Availability, latency, throughput, and concurrency#

Four measures describe how a system performs:

• Availability is the share of requests (or time) in which the system works correctly.
• Latency is how long a request takes.
• Throughput is how many requests the system completes per second.
• Concurrency is how many requests are in progress at the same moment.

The last two connect through simple arithmetic: requests in progress equal requests arriving per second times the seconds each takes. Fourteen tutor questions a second at 4 seconds each means 56 in progress. That number, not the number of students, is what uses up connections, memory, and provider limits. It also explains why slow steps are so dangerous: double the time a request takes and you double how many are in progress, without a single extra student.

Measure latency as a distribution, not an average. Google's SRE book recommends percentiles because an average hides the slow requests: most requests can be fast while a long tail is far slower. The 50th percentile (the median) shows the typical case; the 95th or 99th shows what the unlucky experience (Jones et al., Service Level Objectives). The same chapter notes that averaging over a minute can hide short bursts at twice the average rate. Look at peak-minute numbers, not daily averages.

The tail matters more as systems grow. Dean and Barroso showed how a rare slow response becomes common when one request depends on many components: if each of 100 servers is slow on one request in 100, then 63% of requests that need all 100 will be slow (Dean & Barroso, 2013). The Coach's tutor depends on a database, a search, and a model call in sequence, and each adds its own tail.

Capacity estimates and workload patterns#

A capacity estimate predicts the load the system must handle, with its assumptions written down, as chapter 7 did for storage. For traffic, the most important assumption is the shape over time. The Coach's traffic isn't spread evenly:

• Daily: most activity between 6 p.m. and 11 p.m.
• Weekly: Sunday evenings are busiest, as students plan their week.
• Seasonal: the week before exams brings several times the normal load, concentrated in a few evenings.

So estimate the peak, not the average. With 7,000 students, the average day sees about 3 questions per active student. But in the worst hour before finals, perhaps 40% of students are active and each asks 20 questions: 7,000 × 0.4 × 20 ÷ 3,600 seconds ≈ 16 questions per second. (Every number here is an assumption; check them against real traffic as soon as you have it.) Sunday's real peak was 14. A peak estimate, done in advance, would have predicted it.

Bottleneck identification#

A bottleneck is the resource that runs out first and limits the whole system. Common candidates in a web product:

• database connections, or one slow query;
• the number of worker processes or threads on each server;
• memory;
• an outside service's rate limit or latency;
• a single lock or a single row everyone updates.

Find it by measuring, not guessing. When latency rises, check what requests are waiting on. On Sunday the answer was visible in the database's metrics: connections at the limit, most of them idle inside a transaction, waiting. The servers' CPUs were nearly idle, which is why adding servers couldn't help.

Then load test: generate realistic traffic deliberately and watch what breaks. Grafana's k6 documentation describes several kinds, each for a different risk: a smoke test with minimal load to check the script and the basics; an average-load test at normal levels; a stress test above normal; a spike test for a sudden, short surge; a soak test that runs for hours; and a breakpoint test that raises load until the system fails, to find its limits (Grafana Labs, Load test types). For the Coach, the spike test is the one that matters most: Sunday was a spike.

Load-test staging, never production, with realistic made-up data (chapter 14), and with the AI provider replaced by a fake that waits as long as the real one. Otherwise you test the provider's limits and pay for it.

Service-level objectives#

A service-level indicator (SLI) is a measure of service, such as the share of requests faster than 2 seconds. A service-level objective (SLO) is a target for it. A service-level agreement (SLA) is a contract with consequences if the targets are missed; the SRE book's test is to ask what happens if the objective isn't met (Jones et al., Service Level Objectives).

The same chapter argues against aiming for 100%. Perfection is unrealistic and expensive and slows change; instead, allow an error budget, the amount of missing the SLO you'll accept, and use the gap to decide whether to keep releasing (Jones et al., Service Level Objectives).

For the Coach, chapter 3's NFRs become SLOs with a measurement window:

The last line encodes a product decision: on the night before an exam, the plan matters more than the tutor.

16.2 Performance techniques#

Each technique below solves a specific problem. Apply one when measurement shows that problem, not because an agent suggests it.

Caching and invalidation#

A cache keeps a copy of a result so it can be served again without redoing the work. It helps when the same result is requested often and changes rarely: a note's summary, a class's reading list, the encyclopedia summaries from chapter 8.

Every cache raises one question: when does the copy become wrong? Invalidation is removing or updating copies when the data changes. It's where caches cause bugs, as the failure clinic shows. Before adding one, answer three questions: what's the key (everything the result depends on, including the user)? What changes the data, and does every one of those paths invalidate? How stale may a result be, and what happens if a student sees it?

Two cautions from earlier chapters still apply. Chapter 7 said that at the Coach's size an index usually beats a cache server. And caches don't fix slow tails, as Dean and Barroso note, unless everything the application needs fits in the cache (Dean & Barroso, 2013).

Pagination and batching#

Pagination returns large results in pages rather than all at once: the class progress page shows 50 students at a time, not 400. Batching combines many small operations into one: inserting a week's 60 tasks with one statement instead of 60, or reading the progress of 50 students in one query instead of 50.

The second is the more common fix. A loop that runs one query per item, the N+1 query problem, is easy for an agent to write and invisible with test data of three items. With 400 students in a class, it's 401 queries per page load.

Asynchronous workloads#

Work nobody is waiting for shouldn't run inside a request someone is waiting for. Move it to the background worker (chapter 5) through the job queue (chapter 7): processing uploads, generating weekly summaries, sending reminders. The student gets an immediate response (“Your summary will be ready shortly”) and the work runs at a controlled pace.

Background work can also be shifted in time. Chapter 9 noted that the model provider's batch service processes requests asynchronously at half price. Weekly summaries are due Monday morning, not at the moment Sunday's traffic peaks; generating them overnight through the batch service halves their cost and removes them from the peak.

Efficient queries and indexing#

Most slow pages in a product like the Coach are slow queries. PostgreSQL's EXPLAIN shows how the database plans to run a query; with the ANALYZE option it actually runs the query and reports real row counts and timings for each step (PostgreSQL, Using EXPLAIN). A step labeled Seq Scan on a large table means it read every row: usually a missing index.

On Sunday, while the tutor held connections, one plan page query was also slow on its own. It read the student's recent activity to show a streak, and the activity table, kept for 12 months (chapter 7), had grown to millions of rows with no index on student and date. Each plan page read the whole table. An index on (user_id, occurred_at) took it from 900 milliseconds to 3.

Make it routine: run EXPLAIN ANALYZE on every query in a pull request that touches a large table, with realistic data volumes in staging.

16.3 Distributed design patterns#

Load balancing and stateless services#

A load balancer spreads requests across several copies of a service, so you can add copies when servers really are the bottleneck and survive one failing. Managed platforms provide this (chapter 14).

It works only if any copy can handle any request. The Twelve-Factor App calls this stateless processes: keep nothing between requests in a process's memory or disk, store anything that must persist in a backing service such as the database, and never rely on “sticky sessions” that send a user back to the same process (Wiggins, Processes). The failure clinic shows what happens when a cache quietly breaks this rule.

Scaling out also multiplies everything each copy holds, especially database connections. A connection pooler sits between the services and the database and shares a small number of real connections among many clients; most managed PostgreSQL services offer one. PostgreSQL's own connection limit typically defaults to 100, and raising it costs the server memory (PostgreSQL, Connections and Authentication). Know your limit, divide it among every service and instance, and keep headroom for the worker, migrations, and you.

Queues and eventual consistency#

A queue holds work until a worker can take it, which smooths out bursts: a thousand uploads in a minute become a thousand jobs processed at a steady rate. The cost is that results appear later. Eventual consistency means different parts of the system may briefly disagree, and will agree once the work catches up. A weekly summary generated overnight is eventually consistent with the activity it describes.

That's acceptable when the product says so. Show when data was last updated, and never let a stale view lead to a wrong action: rebalancing (chapter 3) must always work from current tasks, never from a queued or cached copy.

Rate limiting and backpressure#

A rate limit caps how often a client may do something. The Coach already limits each student to 30 tutor questions a day (chapter 9). That protects the budget but not the peak: a daily limit says nothing about a burst. Add a short-window limit too, such as 5 questions per minute per student.

Backpressure is a system telling its callers to slow down when it's full, rather than accepting work it can't finish. The opposite is what happened on Sunday: every request was accepted, queued for a connection, and timed out after the student had given up.

Amazon's guidance on overload explains why accepting everything fails. When a service gets more work than it can handle, latency rises until requests exceed the client's timeout, and then the work is wasted. It distinguishes throughput, all requests handled, from goodput, the requests completed without error and fast enough to be useful. Load shedding, rejecting excess requests quickly and cheaply, keeps goodput steady as load keeps rising (Amazon, Using load shedding). The same article recommends load-testing well past the point where goodput levels off, to check that rejecting a request is actually cheap.

Outside services apply backpressure to you. Anthropic's documentation, for example, says its API limits requests and input and output tokens per minute, enforces them with a token bucket that refills continuously (so short bursts can exceed a per-minute limit), returns an HTTP 429 error naming the exceeded limit with a retry-after header saying how long to wait, and can also refuse sharp increases in usage, advising a gradual ramp-up (Anthropic, Rate limits). Respect retry-after, cap retries (chapter 8), and ask for higher limits before a known peak. (Provider limits change; last reviewed October 2026.)

Graceful degradation and disaster recovery#

Graceful degradation means keeping the most important functions working when something fails or overloads, by turning off or simplifying the rest. Decide the order in advance. For the Coach, during overload:

1. Plan pages and ticking tasks keep working, always.
2. The tutor answers within capacity; beyond it, students get search results from their notes (chapter 9's fallback) with a message that full answers are busy.
3. Quizzes, summaries, and other optional AI features wait in the queue.

Disaster recovery covers losing a whole component or location. ADR-011 (chapter 14) runs the database with a standby in a second availability zone and practices restores against chapter 7's recovery targets. Losing an entire region would mean hours of downtime, an accepted tradeoff at this size, written down so nobody assumes otherwise.

16.4 AI workload performance#

Model latency and inference queues#

A model call's latency has two parts: the wait before the first token arrives, and the time to generate the rest. Longer prompts lengthen the first; longer answers lengthen the second. Streaming, showing words as they're generated, doesn't make the call faster, but it makes the wait feel shorter. The tutor's structured, validated answers (chapter 9) can't be shown until they're checked, so it shows the retrieved passages first and the checked answer after.

The bigger lesson from Sunday is about what a slow call holds while it waits. A request waiting 4 seconds for a model shouldn't hold a database connection, a lock, or anything scarce. Load what you need, release it, call the model, and open a fresh, short transaction to save the result.

Then put a limit on how many model calls run at once, with a short inference queue in front: a bounded waiting line for model calls. When the line is full, shed load to the fallback instead of queueing indefinitely. The worked example sizes it.

Token budgets, caching, and routing#

Tokens are the AI workload's unit of capacity and cost. Set a token budget per request (chapter 9 kept tutor context under 4,000 tokens) and track the real average, because it creeps: one more passage, a longer system prompt, a conversation history nobody trimmed.

Prompt caching (chapter 9) reuses the processing of a prompt's unchanging beginning. It lowers cost, and for some providers it also raises effective capacity. Anthropic's rate limit documentation says that for most of its models, input tokens read from the cache don't count toward the input-tokens-per-minute limit, so a high cache hit rate lets you process several times more input within the same limit (Anthropic, Rate limits). For the tutor, the instructions and output format are identical on every call: about 1,200 of its 3,100 input tokens. Putting them first and caching them was the cheapest capacity increase available.

Routing sends each request to the cheapest model that handles it well. Chapter 9 chose the small model for the tutor after evaluation; the mid-size model was slightly more faithful at about twenty times the cost. Routing could send only hard cases to the larger model, but “hard” must be decided by something cheap and reliable, and every route needs its own evaluation. For the Coach, a single small model with a good fallback is still the right answer. Revisit when evaluation shows a class of questions the small model fails.

Retrieval optimization and reranking cost#

Retrieval (chapter 7's hybrid search) is usually fast compared with the model, but it adds up under load and it shapes token use: every extra passage sent is paid for on every question.

Reranking adds a second step: retrieve a larger set of candidates cheaply, then use a separate model to score each against the question and keep the best. Anthropic's retrieval experiments found that combining their contextual retrieval technique with reranking cut the rate of failed retrievals by 67%, compared with 49% without it. They also noted the cost: reranking adds a step at query time, so it adds latency and cost, and reranking more candidates improves results at the price of more of both (Anthropic, Contextual retrieval). Whether it's worth it is an evaluation question: run chapter 9's evaluation set with and without it, and compare quality, latency at the 95th percentile, and cost per answer.

GPU deployment only when justified#

Running your own model on rented GPUs, the processors models run on, gives control over latency, data location, and per-request cost. It also brings a large fixed cost and a new system to operate. Check the arithmetic first. A capable GPU server running around the clock costs, as an illustrative assumption, a few thousand dollars a month before anyone operates it. The Coach's entire AI bill at 7,000 students is a few hundred dollars (section 16.5). Self-hosting starts to make sense when the provider bill is many times the cost of the hardware and the people to run it, when a data or latency requirement can't be met any other way, or when no provider offers a model that passes your evaluation. Until then, the model provider is the cheapest GPU you can rent.

16.5 Architecture economics#

Cost per active user and per request#

Chapter 8 modeled outside services at $0.22 per active student, and chapter 9 added AI at about $10 a month for 200 students. At 7,000 students, everything changes size, so the model is rebuilt per request and per active student. Every figure is an assumption to check against real invoices.

Two numbers matter more than the total. Cost per request for each feature tells you what a feature costs to use: a tutor answer costs less than a twentieth of a cent. Cost per active student, compared with what a student brings in, tells you whether growth helps or hurts. At these assumptions, a Pro student pays $8; serving her costs about $0.14, plus her payment fee of about $0.54.

Fixed versus variable costs#

Fixed costs don't change with usage over a range: the platform's base plan, the database size, the time Maya and Sam spend on operations. Variable costs grow with use: tokens, emails, payment fees. Fixed costs dominate when you're small, which is why the cost per student fell from chapter 8's estimate as students arrived. Variable costs dominate as you grow, and they're the ones that spike in exam week.

Note what the table leaves out: people. Two people's time is the largest real cost of running the Coach, and every system added (a cache server, a cluster, a self-hosted model) adds to it.

Vendor versus self-hosted tradeoffs#

Every managed service in ADR-011 could, in principle, be replaced by something you run yourself for a lower monthly bill. The comparison has to include the work: patching, upgrades, backups, monitoring, being woken at night. Fowler's “monolith first” argument makes a related point about architecture: microservices carry a significant management cost that slows a team, almost every successful microservice system he heard about started as a monolith that grew too big, and teams rarely get service boundaries right at the start (Fowler, 2015). The Coach's modular monolith with one worker is still the right shape. Split something out when a specific part needs to scale, deploy, or fail independently, and measurements show it.

Cost-performance sensitivity analysis#

A sensitivity analysis changes one assumption at a time to see which ones move the result most. That's where to spend attention.

The model choice dwarfs everything else, which is why the switch to a larger model needs an evaluation result and a cost line in the same pull request. Exam-week traffic is the next biggest, and it's predictable, so spending limits and alerts should be set for it in advance, not discovered on the invoice. Caching saves little money at this size; its real value on Sunday was capacity.

Architect's decision: how does the tutor handle peak load?#

AI collaboration recipe#

Agents are good at writing load test scripts, reading query plans, and doing cost arithmetic without slips. They're poor at diagnosing live performance problems from dashboards, as Sunday showed: they pattern-match to the most common fix. Give them measurements, and ask for hypotheses with tests, not conclusions.

1. Find what's held while waiting.

For each API endpoint, trace what the request holds from start to finish: database connections and transactions, locks, file handles, and calls to outside services, with their timeouts. Flag every place where a scarce resource is held across a call to an outside service or any step that can take more than 100 ms. Don't change code.

2. Capacity estimate.

Here are last month's request logs, aggregated per minute by endpoint, and our student counts. Find the peak minute for each endpoint and the ratio of peak to average. Then estimate the peak for 7,000 and 15,000 students, stating every assumption, and compute the concurrency at each peak using our measured latencies.

3. Load test script.

Write a k6 script for staging with three scenarios: average load, a spike to 25 tutor questions per second for 5 minutes, and a breakpoint test. Mix requests by our measured proportions of plan views, task ticks, and tutor questions. Use made-up accounts from the seed data. Report the 50th, 95th, and 99th percentile latency and error rate per endpoint.

4. Query review.

Here is the output of EXPLAIN ANALYZE for the queries in this pull request, run on staging data. For each, explain in plain words what the database does, which step costs the most, and whether an index or a rewrite would help. Propose the index and the migration, following expand-then-contract (chapter 14).

Boundaries to state every time:

• “Never run load tests against production or the real AI provider.”
• “Measurements first: say what you'd measure to confirm a hypothesis before proposing a fix.”
• “No new infrastructure (caches, queues, databases) without a measurement showing the need.”
• “State every number you assume, and mark it as an assumption.”

Worked example: surviving the next Sunday#

The next Sunday was the second week of finals. Maya and Sam had five days.

1. Reproduce in staging. Sam wrote a k6 spike test from the real Sunday logs, with a fake model that waited 4 seconds and returned a canned answer, and a fake rate limit matching the provider account. At 14 questions per second, staging failed exactly as production had, at minute three. Now they had a test that could tell them whether a fix worked.

2. Release the connection. The tutor endpoint was rewritten in three steps: load the passages in a short transaction and release the connection; call the model with no connection held; open a new short transaction to save the answer and its log. The same review found two more endpoints doing the same thing (the quiz generator and note summaries).

sequenceDiagram
    participant S as Student
    participant API as API
    participant DB as Database
    participant M as AI model
    Note over API,DB: Before: one connection held for the whole request
    S->>API: Question
    API->>DB: Open connection, read passages
    API->>M: Call model (about 4 s, connection still held)
    M-->>API: Answer
    API->>DB: Save answer, release connection
    Note over API,DB: After: two short transactions, nothing held during the model call
    S->>API: Question
    API->>DB: Read passages, release (about 20 ms)
    API->>M: Call model (about 4 s)
    M-->>API: Answer
    API->>DB: Save answer, release (about 10 ms)
    API-->>S: Checked answer

3. Cap and shed. Each instance now allows 12 model calls at once, with a waiting line of 8 that waits at most 2 seconds. A request that can't get a place gets the search-only fallback immediately. With 4 instances, that's 48 calls in progress: at 4 seconds each, about 12 answers a second. The provider limit, raised the Monday after the incident, covers 20 a second with the cached prefix, so the cap, not the provider, is the binding limit, and it's under their control.

flowchart LR
    Q["Tutor question"] --> RL{"Per-student limit:<br/>5 per minute, 30 per day"}
    RL -->|"over"| MSG["“You're asking quickly,<br/>try again in a minute”"]
    RL -->|"ok"| CAP{"Place free?<br/>12 running + 8 waiting<br/>per instance"}
    CAP -->|"no, or waited 2 s"| FB["Search-only fallback:<br/>her closest passages"]
    CAP -->|"yes"| M["Model call<br/>cached prefix, retry-after respected"]
    M -->|"429 or timeout"| FB
    M -->|"ok"| V["Validate and show answer"]

4. Fix the slow query. EXPLAIN ANALYZE on the plan page showed the sequential scan on activity. The index went in through the pipeline. The plan page's 95th percentile in staging fell from 1.4 seconds to 0.3, with no other change.

5. Pooler and connection budget. They turned on the managed database's connection pooler and wrote a connection budget: API instances, worker, migrations, and an admin reserve, adding up to well under the limit at the maximum instance count the platform allows. Autoscaling's maximum was set to that count.

6. Move work off the peak. Weekly summaries moved to the batch service, starting at 1 a.m. Monday.

7. Test again. The spike test at 25 questions per second, nearly twice Sunday's peak:

Half the tutor requests got the fallback at 25 a second, which is the design: capacity went to answers it could deliver, quickly. At 14 a second, the real peak, 86% got full answers. A breakpoint test showed goodput leveling off at about 12 answers a second and staying there to 60 requests a second, the most they tested.

8. The second Sunday. Peak traffic reached 17 questions a second. The plan page's 95th percentile stayed under half a second. Twenty-nine percent of tutor requests in the busiest hour got the fallback, which crossed ADR-013's revisit trigger. Maya opened a task to evaluate the asynchronous design (Option B) before the next exam season, and asked the provider for a higher limit in the meantime.

Verification: will it hold at the peak?#

Failure clinic: the cache that remembered last week#

Symptom. A week after the second Sunday, students reported that their plans “jumped around.” Priya accepted a rebalance, saw her new week, refreshed, and saw the old week. Refreshed again: the new one. Several students, seeing their tasks apparently gone, rebalanced again, creating duplicates. Two wrote that the Coach had dropped tasks without asking, which is exactly what AC-4.2 promises never happens.

Reproduce. In staging, Sam accepted a rebalance and reloaded the plan page 20 times. He got the new plan on some reloads and the old one on others, in no pattern he could see, for up to an hour.

Diagnose. During the busy week, before anyone had run EXPLAIN ANALYZE, an agent had been asked to “get the plan page under 2 seconds at the 95th percentile.” It added a cache: each API instance kept plan pages in its own memory for an hour, keyed by student, and cleared a student's entry when she ticked a task through the API. The plan page met its target in the next test, and the change was merged in the rush.

Three things were wrong:

1. The cache lived inside each instance. The load balancer sent each reload to any of four instances. An instance that had served Priya's page before the rebalance still held the old copy. This is exactly the in-process state the Twelve-Factor App warns against: a later request will probably reach a different process (Wiggins, Processes).
2. Not every writer invalidated. Ticking a task went through the API and cleared the cache, but only on the instance that handled it. Rebalances are applied by the background worker, which knew nothing about any cache.
3. It fixed the wrong problem. The plan page was slow because of the missing activity index. The cache hid that for repeat visits and did nothing for the first load, which is the one students see after any change.

Fix.

1. Remove the cache. With the index in place, the plan page's 95th percentile was 0.3 seconds without it.
2. Repair the data. A script found plans with tasks duplicated by repeated rebalances, and each affected student was shown both versions and asked to choose, in keeping with AC-4.3.
3. Write a cache rule into the instruction file: “No cache without a measurement showing the need, a key, an invalidation path for every writer, a staleness limit, and a test with two instances.” If a cache is ever needed, it lives in a shared store, owned by the module that writes the data (the plans module), and every write path goes through that module.
4. Test with more than one instance. Staging now runs two instances, and the plan tests include “write on one, read on the other.”

Lesson. A cache is a decision about how wrong you're willing to be, and for how long. Make that decision deliberately, for every path that changes the data, and only after measurement shows the slow part is something a cache can fix.

Architect's checklist#

Before calling a system ready for its next peak:

• ☐ SLOs exist for the key journeys, measured as percentiles, with error budgets.
• ☐ The peak load is estimated from workload patterns, with assumptions written down.
• ☐ The first bottleneck is known from a load test, not a guess.
• ☐ Nothing scarce is held across a model call or outside call.
• ☐ The connection budget fits the database limit at maximum scale.
• ☐ Overload is handled with caps, short queues, load shedding, and fallbacks, and a spike test proves it.
• ☐ Rate limits cover bursts and daily use; provider limits are known and raised before predictable peaks.
• ☐ The degradation order is written down and tested.
• ☐ New queries on large tables are checked with EXPLAIN ANALYZE.
• ☐ Every cache has a key, invalidation for every writer, a staleness limit, and a multi-instance test.
• ☐ AI token budgets, caching, and model choice are measured and evaluated.
• ☐ The cost model covers cost per request and per active student, with a sensitivity analysis and spending alerts.
• ☐ No new system was added without a measurement showing the need.

Exercises#

Beginner. The Meal Planner builds shopping lists in a nightly job and serves a weekly plan page. Describe its traffic over a day and a week, estimate the peak requests per second for 10,000 households (state every assumption), and write three SLOs with their SLIs and measurement windows.

Intermediate. The Trip Planner calls an outside flight-price service that takes 2 seconds and allows 10 requests per second. On holiday weekends, users search far more. Compute the concurrency at your estimated peak, design the request path (what's cached and for how long, what's queued, what's shed, and what the user sees in each case), and write the k6 scenarios that would prove it.

Advanced. Build a cost model for the Shop Help Chat at 50,000 conversations a month: model tokens per conversation (measured from a sample or assumed), retrieval and reranking, hand-offs to people, and hosting. Do a sensitivity analysis on five assumptions, including model choice and conversation length. Then decide whether reranking earns its place: define the evaluation you'd run, and the quality, latency, and cost results that would make you keep or drop it.

Lab, artifact, and review#

Lab. Load-test the Learning Coach's tutor in staging with a fake model that waits as long as the real one. Run a smoke test, an average-load test, and a spike test; find the first bottleneck, using database metrics and EXPLAIN ANALYZE where relevant. Fix it, and quantify the improvement with the same tests: latency percentiles, error rate, goodput, and cost per answer before and after.

Deliverable. Add to your portfolio:

• the capacity model: peak estimates, assumptions, concurrency, and the connection budget;
• the SLO table with error budgets;
• the benchmark report: test scenarios, before and after results, and the bottleneck you found;
• the cost model per request and per active student, with the sensitivity analysis;
• ADR-013.

Review. Ask someone else to read your capacity model and name the assumption they trust least. Change it by a factor of three and rerun the arithmetic. If the system would fall over, you've found the next thing to test.

Reflection#

• What is the busiest five minutes your product will face this year, and how do you know?
• What does a request hold while it waits for the slowest thing it calls?
• When your system is overloaded, which feature do you turn off first, and is that decision written down?
• Which assumption in your cost model would hurt most if it were wrong by three times?
• What system did you add last, and what measurement justified it?

Chapter 17 turns to observability: the metrics, logs, and traces that would have shown Sunday's held connections in the first minute, and alerts that people trust enough to act on.

References#

• Amazon. Using load shedding to avoid overload. Amazon Builders' Library.
• Anthropic. (2024, September 19). Introducing contextual retrieval. Anthropic News.
• Anthropic. Rate limits. Claude API documentation.
• Dean, J., & Barroso, L. A. (2013, February 1). The tail at scale. Communications of the ACM.
• Fowler, M. (2015, June 3). Monolith first. martinfowler.com.
• Grafana Labs. Load test types. Grafana k6 documentation.
• Jones, C., Wilkes, J., & Murphy, N., with Smith, C. Service level objectives. In Site Reliability Engineering. Google.
• PostgreSQL Global Development Group. Connections and authentication. PostgreSQL documentation.
• PostgreSQL Global Development Group. Using EXPLAIN. PostgreSQL documentation.
• Wiggins, A. Processes. The Twelve-Factor App.

Links checked October 2026.