Advanced: Reliability Patterns for Distributed AI Systems

Why it matters#

Chapter 8 taught the basic tools for calling another service safely: timeouts, retries at one layer with backoff and jitter, idempotency keys, circuit breakers, and graceful degradation. Chapter 16 added load shedding and capacity limits. Each was applied where it was needed, one integration at a time.

What the March incident showed is that reliability is a property of the whole system, not of each call. As the Learning Coach grew, it gained a gateway, a reranker, two model routes, background jobs for five campuses, and a dozen internal calls per tutor request. Each was reasonable alone. Together they could:

• multiply load when retries stacked across layers;
• spread a failure from one dependency to unrelated features through shared resources;
• keep failing after the cause was gone, because the system's own reaction sustained the overload.

This module is about those whole-system behaviors. It's for readers who will own systems with many moving parts, or review designs where a single slow dependency could take everything with it.

Learning objectives#

After this module you will be able to:

1. Explain cascading and metastable failures, and identify the feedback loop that sustains each.
2. Set deadlines that propagate through a request, and retry policies with budgets at a single layer.
3. Contain failures with circuit breakers, bulkheads, load shedding, and ordered fallbacks, including for agents and costs.
4. Design queue consumers that tolerate duplicate and poisoned messages.
5. Plan and run failure experiments with stated hypotheses.
6. Produce a dependency map with a policy for each dependency, a latency budget, and a game day report.

24.1 How distributed systems fail#

Partial failure, and why slow is worse than down#

In one program on one machine, a function either returns or crashes. Across a network, a call can also succeed slowly, fail after doing its work, or never answer. Amazon's guidance points out that many failures first show up as requests taking longer than usual, and that a waiting client holds resources (memory, threads, connections) the whole time; enough waiting clients exhaust them (Brooker).

That's why a slow dependency is often worse than a dead one. A dead service fails fast, and callers move on. A slow one holds every caller's resources until something gives out. In March, the reranker didn't fail; it got slow, and that was enough.

Cascading failures#

Google's SRE book defines a cascading failure as one that grows over time through positive feedback: part of the system fails, which raises the chance that other parts fail. Its example is a set of replicas where one fails from overload, shifting its load to the rest, which then fail too, like dominoes (Ulrich, 2016).

The Learning Coach's cascade crossed features rather than replicas. A slow reranker filled the shared worker pool; a full pool slowed the plan page; slow pages led students to reload, adding more requests. None of those components was broken. The way they were connected was.

Metastable failures#

The strangest part of March was that the outage outlived its cause. Bronson and colleagues gave this pattern a name: a metastable failure, where a trigger pushes a system into a bad state that persists even after the trigger is removed, because a sustaining effect, often work amplification from retries, keeps it there. They note that outages like this are usually blamed on the trigger, while the real root cause is the sustaining effect (Bronson et al., 2021).

Their paper's example could have been written about the Coach. A web application retries database queries that take more than a second. After a brief network outage, the surge of retried requests overloads the database, latency stays high, so clients keep retrying, doubling the load, and the database never recovers until load is cut or the retry policy changes (Bronson et al., 2021). For the Coach, the trigger was the provider's forty minutes. The sustaining effect was stacked retries, student re-asks, and the Monday summary jobs. Sam's fix, turning the tutor off, was the “strong corrective push” the paper describes.

They also observe that systems often run for long periods in a vulnerable state, healthy but one trigger away from a metastable failure, because that's where they're most efficient (Bronson et al., 2021). The Coach had been vulnerable since the gateway went in. Nothing showed it until midterms.

flowchart LR
    T["Trigger:<br/>provider errors<br/>for 40 minutes"] --> F["Tutor calls fail<br/>or slow down"]
    F --> R["Retries at 3 layers<br/>+ students re-ask"]
    R --> L["Call rate 3× normal"]
    L --> Q["Provider rate-limits<br/>the Coach"]
    Q --> F
    F --> P["Shared worker pool fills;<br/>plan page slows"]
    P --> RL["Students reload"]
    RL --> L

Tail latency and fan-out#

A last reason whole-system behavior differs from per-call behavior: when one request depends on many calls, it waits for the slowest. Dean and Barroso illustrated this with servers that usually answer in 10 milliseconds but take a full second one time in a hundred. A request served by one such server is slow 1% of the time. A request that must wait for a hundred of them in parallel is slow 63% of the time (Dean & Barroso, 2013).

A tutor request isn't a hundred-way fan-out, but it is a chain: authentication, permission lookup, two searches, reranking, the model, the citation check. Each step's slow tail adds to the total. Design to the 95th or 99th percentile of every step, not the average.

24.2 Deadlines, retries, and budgets#

Timeouts versus deadlines#

A timeout is how long one call may take. A deadline is the time by which the whole request must finish. Setting timeouts independently at each step goes wrong easily: if the tutor has 8 seconds in total and the reranker's timeout is 10, the reranker can use up the entire budget and the request will fail anyway, after wasting everyone's resources.

Google's SRE book recommends deadline propagation: set the deadline once, at the top of the request, and pass the remaining time down with every call. If the front end allows 30 seconds and has used 7 before calling the next service, that service gets 23, and so on down the chain (Ulrich, 2016). Each step uses the smaller of its own timeout and the time left, and skips work that can't finish in time.

For the Learning Coach, the tutor's budget became a table, owned by Sam and reviewed whenever a step was added:

Retry at one layer, and only where it's safe#

Chapter 8's rule was to retry at one layer. March showed how easily that rule erodes: the gateway added a layer, and a library default added another without anyone writing a line of code. Amazon's guidance gives the arithmetic: in a five-deep stack with three tries at each layer, a failing database sees 243 times the load (Brooker).

The fix is a decision, recorded once: which layer retries each kind of call. For the Coach, only the gateway retries model and reranker calls, and the client library's retries are explicitly set to zero in code, with a test that fails if the setting changes. The tutor code doesn't retry at all; it asks the gateway and accepts the answer or the fallback.

Retry only what's safe to repeat. Reads are safe. Writes and actions are safe only with idempotency (24.4).

Retry budgets#

Even one layer of retries adds load exactly when a dependency can least afford it. A retry budget caps retries as a share of normal traffic. The SRE book suggests a process-wide limit, such as allowing only a fixed number of retries per minute and simply failing the request once it's spent (Ulrich, 2016). Amazon describes a similar idea, limiting retries with a token bucket, which its SDK added in 2016 (Brooker).

The effect is that retries help with occasional errors, when the budget is full, and switch themselves off during an outage, when they'd only make things worse. The Coach's gateway allows retries up to 10% of the past minute's successful calls per dependency.

Backoff and jitter, briefly#

Chapter 8 covered both: wait longer after each failure (exponential backoff), with a cap, and add randomness (jitter) so clients that failed together don't retry together. Amazon adds advice that applies to the Monday summary jobs too: add jitter to every scheduled job and timer, because many machines doing work “at the top of the minute” or “at midnight” line up and create spikes of their own (Brooker). Five campuses' summaries no longer start at exactly 8:00 a.m.

24.3 Containing failure#

Circuit breakers#

A circuit breaker wraps calls to a dependency and counts failures. Once failures pass a threshold, it trips, and further calls fail immediately without being made. After a while it lets a trial call through; if that succeeds, normal calls resume (Fowler, 2014). Chapter 8 used one for the email service.

Two points matter more at system scale. First, a breaker is only half a design. Fowler stresses that callers must decide what to do when the breaker is open: fail, or fall back to something useful (Fowler, 2014). Second, breakers have critics. Amazon's guidance notes that they add modes that are hard to test and can lengthen recovery, which is why it often prefers retry budgets (Brooker). The Coach uses both, and tests the breaker's states in staging (24.5) rather than trusting them.

Bulkheads#

A ship's hull is divided into watertight compartments, so a breach floods one compartment instead of sinking the ship. Michael Nygard's Release It! lists bulkheads among its stability patterns for software: partition resources so that a failure in one part can't consume the resources another part needs (Nygard, 2018).

March's plan page outage was a missing bulkhead. Tutor and plan requests shared one pool of worker processes, so slow tutor calls starved plan requests. The fix separated them:

• The API runs as two services from the same image: one for the tutor (streamed, slow, dependent on outside providers) and one for everything else (fast, dependent only on the database).
• Each has its own database connection limit, so the tutor can't use the plan page's connections.
• Background jobs run on separate workers with their own limits, by job type, so summary emails can't delay document ingestion.

Bulkheads cost capacity: two half-full pools waste more than one full pool. They buy predictability, which is what an SLO needs.

Load shedding and ordered fallbacks#

Chapter 16's shedding sent excess tutor requests to the search-only answer. At system scale, write the fallbacks down as an ordered list per feature, from best to most basic, and make every step something you've tested:

1. Normal: hybrid search, reranker, primary model route.
2. Reranker unavailable or out of time: fused results without reranking.
3. Primary model route unavailable: a second route through the gateway, a managed model in the team's own cloud account set up the same way as Northfield's (ADR-018), already evaluated on the golden set.
4. Both model routes unavailable, or out of time: closest passages, no generated answer, with a clear message.
5. Search unavailable: the plan page and everything else still work; the tutor shows a short notice.

Each step is visible to students (“answers may be simpler right now”) and to the team (a dashboard panel shows which step each campus is on).

flowchart TD
    Q["Tutor question<br/>deadline: now + 8 s"] --> S{"Search returns<br/>in time?"}
    S -- "No" --> N["Short notice;<br/>plan page unaffected"]
    S -- "Yes" --> R{"Reranker breaker closed<br/>and ≥ 0.5 s left?"}
    R -- "Yes" --> RR["Rerank"]
    R -- "No" --> FU["Use fused list"]
    RR --> M1{"Primary model route<br/>available in time?"}
    FU --> M1
    M1 -- "Yes" --> A["Full answer"]
    M1 -- "No" --> M2{"Fallback route<br/>available in time?"}
    M2 -- "Yes" --> A2["Answer, labeled<br/>“may be simpler”"]
    M2 -- "No" --> P["Closest passages,<br/>no generated answer"]

Runaway work and runaway cost#

Some failures don't produce errors at all. An agent that loops, a job that resubmits itself, or a retry storm can run up a bill instead of a page. Treat cost as a reliability signal:

• Bound every loop. The Coach's rebalancing agent (chapter 9) already had a step limit and a cost limit per run. It now has a time limit too, and stops cleanly with a partial result when it hits any of them.
• Budget per tenant. The gateway tracks token spend per campus per hour and alerts at twice the normal rate for that hour, so a bug at one campus is noticed before the invoice arrives.
• Cap at the provider. Provider-side spending limits are the last line, set high enough not to cause outages in a normal exam week and low enough to stop a runaway weekend.

When cost does need to come down, work through the cheapest changes first: remove unnecessary model calls, use a smaller model where quality allows, send less context, cache what's safe to cache, batch background work, and only then compare providers or hardware. Each step is measured on the golden set before it ships.

24.4 Queues and the illusion of exactly once#

At-least-once delivery#

Background work in the Coach (document ingestion, weekly summaries, notifications) runs from a queue. A worker claims a job, does it, and marks it done. If the worker crashes before marking it done, the job must not be lost, so after a lease period it becomes available again.

Managed queues work the same way. Amazon SQS hides a received message for a visibility timeout (30 seconds by default) and makes it visible again if it isn't deleted in time. Its documentation states plainly that, because the service delivers at least once, a message can occasionally be delivered more than once even within that period (AWS, SQS visibility timeout).

So “exactly once” isn't something the queue gives you. You get at least once, and you make twice harmless.

Idempotent consumers#

A consumer is idempotent if processing the same job twice has the same effect as processing it once. Chapter 8 used idempotency keys for outgoing calls; Stripe's API, for example, saves the result of the first request made with a key and returns that result to any retry with the same key (Stripe, Idempotent requests). Consumers need the same idea from the inside:

• Give each job a key that identifies the effect, not the attempt: for a weekly summary, the student, the advisor, and the week.
• Before doing the effect, record the key in the same transaction as the work's result, or check it against a table of completed keys.
• If the key is already there, skip the effect and mark the job done.
• Re-check preconditions at the moment of the effect, not when the job was queued. A job queued at 8:00 to email an advisor must check, at the moment of sending, that the student still shares with that advisor.

The failure clinic in this module shows what happens without the last two.

Poison messages and dead-letter queues#

Some jobs fail every time: a corrupt file, a document in an unsupported format, a bug triggered by one input. Retried forever, a poison message wastes workers and can block the queue. A dead-letter queue collects jobs that have failed a set number of times so they stop circulating and someone can look at them. AWS's documentation describes the setting that controls this (the number of receives before a message is moved) and recommends setting it high enough to allow for ordinary retries (AWS, SQS dead-letter queues).

A dead-letter queue is only useful if someone owns it. The Coach's runbook names the owner per job type, alerts when anything lands there, and records each item's outcome: fixed and replayed, or discarded with a reason.

Lease length and long jobs#

If a job can run longer than its lease, a second worker may pick it up while the first is still working, which is exactly how duplicates happen without any crash. Set the lease longer than the job's realistic maximum, extend it while working (a heartbeat), or split long jobs into steps. AWS recommends the same: match the timeout to the longest processing time you expect, and extend it from the worker if a task runs long (AWS, SQS visibility timeout).

24.5 Testing failure on purpose#

From hope to hypothesis#

None of the March problems was exotic. Each could have been found by asking “what happens if the provider is slow?” and trying it. Netflix's engineers describe chaos engineering as experimenting on a distributed system to build confidence that it can withstand turbulent conditions, and set out principles for doing it: start from a hypothesis about steady-state behavior, vary real-world events, experiment in production, and automate experiments to run continuously (Basiri et al., 2016).

A team of three doesn't start in production. It starts in staging, with the principles intact:

• Steady state is defined by user-visible measures: plan page p95, tutor answer rate, error rate, queue age.
• Each experiment has a hypothesis: “If the reranker takes 5 seconds, tutor answers continue without reranking within the 8-second deadline, and the plan page is unaffected.”
• Faults are realistic: added latency, a share of errors, a dependency refusing connections, a worker killed mid-job, a duplicated message.
• There's an abort plan: who stops the experiment, how, and what “stop” means.

Game days#

A game day is a scheduled session where the team runs failure experiments together, watches what happens, and practices the response. It tests the system and the people: do the alerts fire, does the runbook work, does the on-call person know which flag to turn?

Run them before the moments that matter. For the Coach, that means the week before each term's midterms and finals, with every new dependency added since the last one.

Keeping experiments honest#

The value of an experiment is in the results that surprise you. Write the hypothesis before running it, record what actually happened (including the parts that went fine), and file a fix for every disproven hypothesis with an owner and a date. An experiment that always passes is either testing the wrong thing or isn't being run against the current system.

Architect's decision: how should the Coach handle dependency failures?#

AI collaboration recipe#

Agents can read a codebase faster than any person and find every place a call is made. They're less good at imagining how parts interact under stress. Use them to map and to check, and keep the hypotheses and the abort decisions with people.

1. Map the dependencies.

List every outbound call in this repository: what it calls, from which module, its timeout, whether and where it's retried (including library defaults), whether it's protected by a breaker, which resource pool it uses, and what happens on failure. Mark any value you inferred rather than read from code or configuration.

2. Find stacked retries.

For each external dependency, trace every path from a user request to that dependency and count the maximum number of attempts along each path, multiplying across layers. Show the paths with more than three attempts.

3. Draft experiments.

For each dependency in this map, propose one failure experiment for staging: the fault to inject, the hypothesis in terms of user-visible measures, the measurements to watch, and the abort condition. Don't run anything.

4. Review a consumer.

Review this job consumer for duplicate safety. If the same job is delivered twice, at the same time or an hour apart, what happens? Check whether preconditions are re-checked at the moment of the effect. Cite lines.

Boundaries to state every time:

• “Read and propose; don't inject faults or change configuration.”
• “Include library and framework defaults, not just our code.”
• “Mark inferences as inferences.”
• “Experiments run only in staging, with a named person able to abort.”

Worked example: redesigning for March, then breaking it on purpose#

Step 1: The dependency map. The agent's map listed 23 outbound calls. Lena checked each against the code and found three the agent had missed (calls made by a library on the agent's behalf, such as fetching provider credentials) and two where it had guessed a timeout that didn't exist. The corrected map showed the problems clearly: the reranker with no timeout, three retry layers on model calls, the provider library's default retries, and both API workloads sharing one worker pool.

Step 2: One policy per dependency, in one place. Instead of fixing each call, the team moved every model and reranker call behind the gateway's single client, configured from a table. Its core is short:

# backend/coach/gateway/call.py
async def call(dep: Dependency, request: Request, deadline: float) -> Response:
    """Call a dependency within the request's deadline, with at most one budgeted retry."""
    for attempt in (1, 2):
        remaining = deadline - time.monotonic()
        if remaining < dep.min_useful_seconds or dep.breaker.is_open():
            raise Unavailable(dep.name)
        try:
            response = await asyncio.wait_for(dep.send(request), min(dep.timeout, remaining))
            dep.breaker.record_success()
            return response
        except (asyncio.TimeoutError, TransientError):
            dep.breaker.record_failure()
            if attempt == 2 or not dep.retry_safe or not dep.retry_budget.try_spend():
                raise Unavailable(dep.name)
            await asyncio.sleep(random.uniform(0, dep.backoff_cap_seconds))

The tutor's code catches Unavailable at each step and moves down the fallback list. The provider library is created with its own retries set to zero, and a unit test asserts that setting.

Step 3: Bulkheads. The API was split into coach-tutor and coach-core services from the same image, each with its own scaling rule and database connection limit, in the infrastructure module from chapter 22. Background workers were split by job type.

Step 4: Idempotent jobs. Every job type got an effect key and a check at the moment of the effect. The summary job's fix is the subject of the failure clinic.

Step 5: The game day. Two weeks before finals, the team ran five experiments in staging, with production-like load from the load-test scripts. Maya held the abort switch.

The third row was the most useful result of the day. Nobody had looked at the fallback on a phone, which is how most students use the Coach.

Step 6: The real test. In late April, during finals, the primary provider had another incident: 25 minutes of elevated errors. The breaker opened, the managed fallback route carried the other campuses' tutor traffic, and the dashboards showed the fallback step per campus. The Coach was back on its primary route three minutes after the provider recovered. Nobody was paged for the plan page.

Verification: will it fail well?#

Failure clinic: the summary sent three times, once too many#

Symptom. On a Monday in February, an advisor at Riverside emailed the support address: she'd received the same student's weekly summary three times. Worse, the student had turned off sharing with her that morning, before the emails went out. The student found out when the advisor mentioned it.

Root cause. The summary job generated the summary (a model call, usually 5 to 15 seconds), then sent the email. The job queue's lease was 30 seconds. On Monday morning, with all campuses' summaries running and the model slow, some jobs took over 30 seconds. Their leases expired while they were still running, and another worker picked up each one and started again. One job ran three times in parallel.

The consent check happened when the job was queued, at 6 a.m. The student turned off sharing at 7:40. The emails went out after 8:00. The job never checked again.

Fix.

• Write to the student and the advisor, with the privacy office's guidance; the advisor deleted the summaries and confirmed it.
• Give each summary an effect key (student, advisor, ISO week) recorded in a sent_summaries table, inserted in the same transaction as the send is recorded, with a uniqueness constraint. A second attempt finds the key and stops.
• Check consent at the moment of sending, inside that transaction, and skip if it's no longer active.
• Set the lease longer than the job's realistic maximum and extend it with a heartbeat while the model call runs.
• Add jitter to the schedule, so five campuses' summaries spread across an hour.
• Add both cases to the game day: duplicate delivery, and consent revoked between queueing and sending.

Lesson. At-least-once delivery means a job will sometimes run twice, and later than you planned. Make the second run harmless, and check permission when you act, not when you planned to.

Architect's checklist#

• ☐ A dependency map lists every outbound call with its timeout, retry layer, breaker, resource pool, and fallback.
• ☐ Deadlines are set once per request and propagated; no step's timeout exceeds the time left.
• ☐ Each dependency is retried at exactly one layer, library defaults included, with a retry budget.
• ☐ Breakers protect slow or flaky dependencies, and their open state has a tested fallback.
• ☐ Workloads with different dependencies have separate resource pools.
• ☐ Each feature has an ordered list of fallbacks, all tested, all visible to users and the team.
• ☐ Loops, agents, and jobs have step, time, and token limits; spend is monitored per tenant.
• ☐ Every consumer is idempotent by effect key and re-checks permissions at the moment of the effect.
• ☐ Leases exceed realistic job durations or are extended while working.
• ☐ Poison jobs land in a dead-letter queue with a named owner and an alert.
• ☐ Scheduled jobs are jittered.
• ☐ Game days run before peak periods, with written hypotheses, results, and fixes.

Exercises#

Beginner. The Meal Planner calls a recipe service, a nutrition service, and a grocery-price service to build each week's plan. For each one, write what the user should see if it's down and if it's slow, and order the fallbacks from best to most basic. Then explain, in your own words, why a slow nutrition service might be worse than a dead one.

Intermediate. The Trip Planner rebuilds a day's plan when a flight is delayed: it calls a flight-status API, a hotel API, and a model, with retries in the client library, the API layer, and a background job. Draw the call paths, count the maximum attempts per user action, and redesign it with one retry layer, a deadline, and a retry budget. Write the latency budget table, as in 24.2.

Advanced. The Shop Help Chat starts returns, which creates a return label and emails the customer. Design the return job to be safe under duplicate delivery and delayed processing, including the case where the customer cancels the return between queueing and processing. Then write a game day plan with five experiments (fault, hypothesis, measurements, abort condition) and say which result would worry you most.

Lab, artifact, and review#

Lab. Build the dependency map for your Learning Coach or capstone, find every stacked retry and missing timeout, and fix the worst three. Then run a game day in staging with at least four experiments, one of which injects duplicate delivery into a background job. Record hypotheses before you start.

Deliverable. Add to your portfolio:

• the dependency map with a policy for each dependency;
• the latency budget for your most important request, with fallbacks;
• the game day report: hypotheses, results, surprises, and fixes with owners;
• one consumer's idempotency design, with the test that proves it.

Review. Give the dependency map to someone who didn't build the system and ask them to pick the dependency they'd break first. Run that experiment next.

Reflection#

• In your system, what's the trigger you fear most, and what would sustain the failure after the trigger is gone?
• Which retries in your code did you write, and which did a library add for you?
• What resource do your slowest and fastest features share?
• Which background job would do harm if it ran twice, or an hour late?
• When did you last break your system on purpose, and what surprised you?

The final advanced module steps back from systems to the role. By the end of the spring term the Learning Coach served five campuses under three kinds of contracts, and the state system asked Maya a question that wasn't technical at all: would she review the architecture of every AI product its campuses wanted to buy?

References#

• Amazon Web Services. Amazon SQS visibility timeout. Amazon Simple Queue Service Developer Guide.
• Amazon Web Services. Using dead-letter queues in Amazon SQS. Amazon Simple Queue Service Developer Guide.
• Basiri, A., Behnam, N., de Rooij, R., Hochstein, L., Kosewski, L., Reynolds, J., & Rosenthal, C. (2016). Chaos Engineering. IEEE Software, 33(3), 35–41.
• Bronson, N., Aghayev, A., Charapko, A., & Zhu, T. (2021). Metastable failures in distributed systems. Workshop on Hot Topics in Operating Systems (HotOS '21).
• Brooker, M. Timeouts, retries, and backoff with jitter. Amazon Builders' Library.
• Dean, J., & Barroso, L. A. (2013). The tail at scale. Communications of the ACM, 56(2), 74–80.
• Fowler, M. (2014, March 6). CircuitBreaker. martinfowler.com.
• Nygard, M. T. (2018). Release It! Design and Deploy Production-Ready Software (2nd ed.). Pragmatic Bookshelf.
• Stripe. Idempotent requests. Stripe API reference.
• Ulrich, M. (2016). Addressing Cascading Failures. In B. Beyer, C. Jones, J. Petoff, & N. R. Murphy (Eds.), Site Reliability Engineering: How Google Runs Production Systems, chapter 22. O'Reilly.

Links checked October 2026.