Observability, Reliability, and Production Feedback

Why it matters#

Chapter 16's Sunday night had the opposite problem: everything was visibly failing, and the dashboards couldn't say why. They showed CPU, which was fine, and not database connections waiting, which weren't. Between them, the two incidents cover the two questions monitoring has to answer: is something wrong? and what?

Observability is the ability to answer new questions about what a system is doing from the data it produces, without shipping new code to ask them. It goes beyond monitoring, which watches for problems you predicted. You'll never predict every failure, so you need data rich enough to investigate the ones you didn't.

AI features need this more than most. Chip Huyen observes that operational failures, such as timeouts and errors, are fairly easy to detect, while a model producing worse results is much harder, because you often don't know what the right output was; she concludes that machine learning systems often fail silently (Huyen, 2022). The extraction bug was exactly that kind: a silent failure in a pipeline feeding a model.

For a team directing AI agents, there's one more reason. Agents ship many changes, and each one is a chance for a quiet regression. Production feedback is how you find out which change, before an instructor has to tell you.

Learning objectives#

After this chapter you will be able to:

1. Explain logs, metrics, and traces, and instrument a request end to end with a trace ID that links them.
2. Design dashboards that answer specific questions and alerts that people act on.
3. Run reliability practices: SLOs with error budget policies, runbooks with owners, recovery drills, and post-incident learning.
4. Version and monitor AI behavior in production: prompt and model versions, evaluation trends, quality, latency, and cost, and user feedback collected with privacy protections.
5. Detect degradations, roll out changes safely, and prioritize improvements by impact.
6. Produce a dashboard, a runbook, a postmortem, and an AI evaluation trend report.

17.1 Observability fundamentals#

Logs, metrics, traces#

Systems produce three main kinds of telemetry, which OpenTelemetry, the open standard most tools now support, calls signals: a trace is the path of a request through the application, a metric is a measurement captured while it runs, and a log is a record of an event (OpenTelemetry, Signals). Each answers different questions:

• Metrics are numbers over time: requests per second, error rate, 95th percentile latency, connections in use. They're cheap to store and fast to query, so they power dashboards and alerts. They tell you that something changed.
• Logs are records of individual events, written as structured fields (chapter 13). They tell you what happened in a specific case.
• Traces show one request's journey as a tree of spans, each span a timed step (verify the session, retrieve passages, call the model) with attributes describing it. They tell you where the time went and which step failed.

You need all three, linked together. A metric shows the share of declined answers rising; traces of declined answers show what was retrieved; logs on those traces show which extractor version produced the text.

Correlation IDs and request journeys#

The link is a shared ID. Chapter 13 introduced the request ID; tracing formalizes it as a trace ID, created when a request arrives and passed to everything it causes. Between services, it travels in a standard HTTP header: the W3C Trace Context recommendation defines a traceparent header that carries a version, the trace ID, the ID of the calling span, and flags, in a fixed format any tracing tool can read (W3C, 2021). OpenTelemetry uses that header by default, and its libraries can stamp the trace and span IDs onto every log record, so logs and traces line up automatically (OpenTelemetry, Context propagation).

Two journeys need special care in the Learning Coach:

• Requests that start work in the background. When a student uploads a note, the API request ends quickly, and the worker extracts the text later. Store the trace context with the job, so the worker's spans join the same trace. Then the question “what happened to this upload?” has one answer.
• Data that outlives the request. The extraction bug lived in stored chunks, not in any one request. Record how derived data was made, as attributes on the data itself: each chunk stores the extractor version and the time it was made. That's what would have turned two days of searching into one query.

OpenTelemetry's guidance also covers privacy: don't put credentials or personal information in baggage, the key-value data that travels between services, and be careful what context you accept from or send to outside services (OpenTelemetry, Context propagation). Chapter 15's rule holds for every signal: IDs and metadata, never note text, names, or emails.

Dashboards and alerts#

Google's SRE book names four golden signals for any user-facing system: latency, traffic, errors, and saturation (how full the most constrained resource is), and says that if you can measure only four things, measure these (Ewaschuk, Monitoring Distributed Systems). On Sunday, the saturation signal that mattered was database connections in use, and the dashboard didn't have it.

A good dashboard answers a question someone will actually ask. Build a few, each for one audience:

• Service health: the four golden signals per key journey, against the SLOs from chapter 16.
• AI quality: what the AI features produced (section 17.3).
• Cost: spending per feature and per active student, against the cost model.

Mark every deploy, feature flag change, and dependency upgrade on the timelines. Most degradations start at a change, and seeing the line where the curve bends saves hours.

An alert asks a person to do something. The SRE book is strict about which alerts should wake someone: each page should be urgent, actionable, and about a problem users see or soon will; every page should demand intelligent action, not a rote response a script could do; and you should spend much more effort catching symptoms (what users experience) than causes (one of many things that could produce it) (Ewaschuk, Monitoring Distributed Systems).

Signal-to-noise in monitoring#

Everything else should be a ticket, work for the next working day, or a dashboard panel, or deleted. The SRE workbook's chapter on alerting defines what makes an alert good: precision (the share of alerts that were about something significant), recall (the share of significant events that caused an alert), detection time, and reset time, how long an alert keeps firing after the problem is fixed (Thurgood et al., Alerting on SLOs).

Low precision is how alert channels die. If most alerts don't matter, people learn to ignore all of them, and the one that matters is ignored too. The failure clinic is that story.

flowchart LR
    subgraph APP["Learning Coach services"]
        API["API<br/>spans, metrics, logs"]
        WK["Worker<br/>joins the trace from the job"]
    end
    API -->|"traceparent header,<br/>trace context stored with jobs"| WK
    API --> COL["Telemetry collector:<br/>drops personal data,<br/>samples traces"]
    WK --> COL
    COL --> MET[("Metrics")]
    COL --> TR[("Traces")]
    COL --> LOG[("Logs")]
    MET --> DASH["Dashboards:<br/>health, AI quality, cost"]
    MET --> AL{"Alert rules"}
    AL -->|"fast budget burn,<br/>broken promise"| PAGE["Page: act now"]
    AL -->|"slow burn,<br/>quality drift"| TICKET["Ticket: next working day"]
    TR --> DASH
    LOG --> DASH

17.2 Reliability practices#

SLOs, error budgets, and incidents#

Chapter 16 set SLOs for the plan page, rebalancing, the tutor, and overall availability. An SLO is only useful if something happens when you miss it. An error budget policy writes down what. Google's example policy says that if a service has used up its error budget over the previous four weeks, releases stop except for urgent fixes and security fixes until it's back within its SLO, and that any single incident using more than 20% of the budget gets a postmortem with at least one top-priority action. It also says plainly that the policy isn't a punishment; it gives the team permission to focus on reliability when the data says reliability matters more (Thurgood, 2018).

For the Coach, the policy fits on half a page: when the plan page's budget for the month is gone, Maya and Sam stop feature work and agent tickets on features until the cause is fixed.

Alert on the budget, not on every error. The SRE workbook recommends alerting on burn rate, how fast the budget is being used, and suggests as starting points paging when 2% of the budget is used in an hour or 5% in six hours, and opening a ticket when 10% is used in three days (Thurgood et al., Alerting on SLOs). A total outage pages within minutes; a slow leak becomes a ticket; a single failed request does nothing.

Runbooks and incident ownership#

Every alert links to a runbook (chapter 14): what the alert means, how to confirm it, what to do first, and whom to involve. An alert without a runbook isn't finished.

Every alert, dashboard, and runbook has an owner, a named person who keeps it accurate. With two people, ownership is simple but must still be explicit. Maya and Sam now alternate weeks on call, the person responsible for responding to pages; the other is the backup. During declared peak weeks (chapter 16), both are on call in the evenings.

Backups and recovery drills#

Chapter 14 set up quarterly restore drills, including the NFR-6 deletion replay. Treat drills as observability too: a drill is a test of whether the system does what you believe it does. Record each drill's timings and findings in the reliability log, and alert when a drill is overdue. The failure clinic shows a drill finding what the alerts missed.

Post-incident learning#

Chapter 14 introduced blameless postmortems. Two practices make them pay off over time:

• Track the actions to completion. A postmortem whose actions sit in a document has taught nothing. Each action becomes a ticket with an owner and a date, reviewed weekly until done.
• Read postmortems together. Every quarter, read the quarter's incidents side by side and look for patterns. The extraction bug, chapter 16's Sunday, and the failure clinic below share one: a change whose effect nobody was measuring. That pattern, not any single incident, is what this chapter's work fixes.

17.3 LLMOps and AI quality#

LLMOps is the practice of running AI model features in production: versioning what shapes their behavior, measuring their quality continuously, and changing them safely. It's operations, applied to a component whose output varies.

Prompt and model versioning#

Chapter 13 made prompts, model versions, retrieval settings, and tool definitions code. In production, record which versions produced each output. The tutor's span for each answer carries the full recipe:

• prompt version (tutor-v7) and model ID, pinned (chapter 9);
• retrieval settings version, and the number and IDs of passages retrieved;
• extractor and chunker versions of those passages;
• outcome (answered, insufficient evidence, validation failed, fallback) and, if validation failed, why.

With the recipe, every quality metric can be broken down by any part of it. The extraction bug becomes a chart: declines by extractor version, one line flat, one line climbing.

OpenTelemetry has published conventions for naming generative AI telemetry, such as model names and token counts, now maintained in their own repository (OpenTelemetry, Generative AI semantic conventions). Follow them where your tools support them, so dashboards and tools understand the same names. (These conventions are still evolving; last reviewed October 2026.)

Evaluation datasets and regression monitoring#

Chapter 13's golden set runs on every change. Production needs two more checks:

• Scheduled evaluation. Run the golden set every night against the production configuration, including the current versions of every dependency in the pipeline. This catches changes nobody made on purpose: a provider updating a model behind the same name, a library upgrade, a configuration drift. Add end-to-end cases that start from the original files, so extraction and chunking are tested, not only the tutor.
• Fresh samples. The golden set ages. Huyen describes data distribution shift: production inputs gradually differ from the data a system was tested on (Huyen, 2022). The university pilot brought nine new courses, with different notes and different questions. Each week, sample real questions from students who allow it, grade them with chapter 9's calibrated grader plus a human check on a few, and add the instructive ones to the golden set.

Report the results as a trend, not a snapshot. A weekly evaluation trend report shows each quality measure over the last eight weeks, with every change that might explain a movement.

Quality, latency, and cost dashboards#

The AI quality dashboard tracks what the feature produced, not just whether it responded:

Each line is broken down by prompt version, model, extractor version, and course. Alert on sustained movement against a baseline, such as the insufficient-evidence share rising more than five points above its four-week average for a day, as a ticket, not a page. Quality drift rarely needs someone at midnight; it needs someone this week.

User feedback with privacy protections#

Thumbs up and down, with an optional reason, are the cheapest quality signal you'll get. Three cautions:

• Feedback is biased. Few students press either button, and unhappy ones press more. Use it to find problems and examples, not to measure overall quality.
• Watch behavior too. Signals students give without meaning to: asking the same question again in different words, opening the cited passage, leaving the tutor right after an answer. Each is a weak signal; together they're informative.
• Collect only what you need. By default a feedback record holds the request ID, the rating, and the reason category. The question and answer text are stored only if the student ticks “share this example to help improve the tutor,” then kept for 90 days (an assumption to confirm with your privacy review) in a store only Maya and Sam can read. That follows chapter 15's logging rules and chapter 7's data inventory.

17.4 Continuous improvement#

Detecting degradations#

Most degradations show up as a change in a metric that already exists, at a moment that matches a change you made. Three habits find them:

• Baselines. Compare with the same hour last week, or a four-week average, not with a fixed number. Traffic before finals isn't comparable to a quiet Tuesday.
• Segments. A small change overall can be a large change in one place. The extraction bug moved the overall decline rate from 9% to 16%, but for notes uploaded after the upgrade it was 27%, and for notes heavy with formulas it was higher still.
• Change markers. Every deploy, flag change, dependency upgrade, prompt version, and provider announcement goes on the timeline.

Controlled experiments and feature flags#

A controlled experiment, often called an A/B test, randomly splits users between the current version and a new one and compares an agreed measure. Pete Hodgson's widely cited article on feature flags describes experiment toggles that consistently place each user in one group, alongside release toggles for unfinished work and ops toggles for switching features off in production. He also warns that flags are inventory with a carrying cost, and describes teams that give flags expiry dates or limit how many may exist (Hodgson, 2017).

Experiments need enough users to detect a difference. With 7,000 students, an experiment can detect a large effect on a common action, such as whether students open the tutor more, within a few weeks, but not a small effect on a rare one. For most AI changes, the order is: offline evaluation on the golden set, then a canary (chapter 14), then, only for changes where student behavior is the real question, an experiment with its measure and duration decided before it starts. Decide in advance; deciding after seeing the data finds patterns in noise.

Rollback and safe rollout#

Chapter 14's canary and rollback apply to AI changes, with one addition: a prompt or model change can be rolled out behind a flag, separately from code, and rolled back in seconds. For changes to stored data, like a new extractor, rollback isn't enough. The bad version's output stays in the database. Record which version produced each item (the recipe), so you can find and redo everything a bad version touched, as Maya did for 2,100 notes.

Prioritizing improvements by impact#

Production feedback produces more possible improvements than two people can make. Rank them with a simple, written formula: how many students are affected, how badly, how often, divided by the effort. The defect log (chapter 13), feedback reasons, evaluation failures, and postmortem actions all feed one list, reviewed weekly. Reliability work competes on the same list as features, and the error budget policy decides when it wins automatically.

flowchart LR
    P["Production signals:<br/>metrics, traces, feedback,<br/>instructor reviews, drills"] --> T["Weekly triage:<br/>rank by students affected<br/>× severity × frequency ÷ effort"]
    T --> G["New golden cases<br/>and regression tests"]
    T --> FIX["Fix or improvement,<br/>as a spec and task"]
    G --> E["Evaluation in CI<br/>and nightly"]
    FIX --> E
    E --> R["Release behind a flag:<br/>canary, then everyone"]
    R --> P

Architect's decision: how do we observe the Learning Coach in production?#

AI collaboration recipe#

Agents are good at adding instrumentation consistently, writing alert rules from SLOs, and summarizing traces and incident timelines. They tend to over-instrument and over-alert: give them a question to answer, not “add monitoring.”

1. Telemetry plan from questions.

Here are the questions we must be able to answer in production (attached), our SLOs, and the tutor's code. For each question, say which signal answers it (metric, trace attribute, or log field), the exact name and type, and where in the code to add it. Follow the OpenTelemetry naming conventions where they exist. No note text, names, emails, or tokens in any signal.

2. Instrument one journey.

Add OpenTelemetry tracing to the note upload journey: the API request, the job, and the worker's extract, chunk, and embed steps, as one trace. Store the trace context with the job. Add the extractor and chunker versions as attributes on the spans and as columns on note_chunks, with a migration that works with the previous code version. Don't change any behavior.

3. Alert review.

Here is every alert rule and the last 60 days of alert history with what was done after each. For each rule, report how often it fired, how often someone acted, and whether it tracks a symptom users see or a cause. Recommend keep, change to ticket, merge, or delete, with a reason. Propose burn-rate alerts for our SLOs using the SRE workbook's starting thresholds.

4. Investigation helper.

Here is a metric change (attached chart and timeline of changes). List hypotheses that could explain it, and for each, the query or trace search that would confirm or rule it out. Don't conclude; rank by how cheaply each can be checked.

Boundaries to state every time:

• “Telemetry holds IDs and metadata only; never note text, names, emails, tokens, or secrets.”
• “Every new alert needs an owner, a runbook link, and a page-or-ticket decision.”
• “Don't change application behavior while instrumenting.”
• “Hypotheses with checks, not conclusions.”

Worked example: instrumenting the tutor end to end#

Maya and Sam spent the week after the extraction bug making sure the next one would show up on its own.

1. Start with questions. They wrote down the questions they'd needed during the two incidents and couldn't answer: Which step of a slow tutor answer took the time? What is waiting for database connections? Are declines rising, and for which notes? Which versions produced this passage? What did this upload go through? Every signal they added had to answer one of these.

2. Trace the journeys. OpenTelemetry instrumentation went into the API and the worker. A tutor request now produces one trace:

The trace ID is returned with every error message, so a student's report leads straight to the trace. Uploads produce one trace across the API and the worker, with extraction quality recorded on the extract span: pages, characters, and the share of characters that are replacement symbols or blanks where text was expected.

3. Record the recipe on the data. note_chunks gained extractor_version and chunker_version, filled in for old chunks from the deploy history. The nightly golden set now includes 12 cases that start from original PDF files, including three pages of formulas.

4. Build three dashboards. Service health: the four golden signals per journey, with the saturation panels chapter 16 had lacked (connections in use, pool wait, inference queue depth, provider rate-limit responses). AI quality: the table from section 17.3, broken down by version and course. Cost: per feature and per active student, against chapter 16's model. Deploys, flags, and dependency upgrades are marked on every timeline.

5. Rewrite the alerts. Before: 23 rules, about 40 alerts a week. After: 9 rules.

Every rule has an owner and a runbook link. Everything else became a dashboard panel or was deleted.

6. Replay the incidents. In staging, Sam reinstalled the bad extractor version and uploaded the formula test notes alongside normal traffic. The damaged-characters ticket opened after 40 minutes; the insufficient-evidence ticket after 19 hours; the dashboard, split by extractor version, showed the cause in one view. Then he replayed chapter 16's Sunday with the old connection handling: the tutor SLO page fired in 4 minutes, and the pool wait panel pointed straight at the cause.

7. The first trend report. Eight weeks of the tutor's evaluation, rebuilt from the nightly runs and weekly samples:

The faithfulness column barely moved, which is the lesson: the tutor stayed faithful to the passages it got. The failure was upstream, visible only in the outcome mix. The report now goes to Maya, Sam, and Dr. Okafor every Monday.

Verification: can you see and fix what's happening?#

Failure clinic: the alert everyone muted#

Symptom. During the quarterly restore drill (chapter 14), the NFR-6 check, which confirms that deleted accounts stay deleted, failed in a way nobody expected. The restored database was fine. But object storage held 214 files whose owners' accounts had been deleted between 9 and 26 days earlier. NFR-6 promises deletion within 7 days.

Reproduce. Sam deleted a test account with a shared class note in staging. The database rows disappeared within a minute. The shared note's file stayed. The deletion job's log showed a “permission denied” error from object storage on the class library files, five retries, and then the job marked itself failed.

Diagnose. Chapter 15's hardening had tightened each service's storage permissions to match the permission matrix. The worker's row for the class library said “expire with class,” which the agent implementing it read as: the worker needs permission to run expiry rules, not to delete individual files. Account deletion of a student's shared notes was a path nobody had listed. The permission tests from chapter 15 checked every “never” cell, and passed; nothing tested that the worker could still do what it must.

The alert had fired. Every failed deletion job posted to the alert channel, 31 times over 26 days. But the channel received about 40 alerts a week: single failed requests, staging certificate warnings, a flaky uptime check, CPU spikes that resolved themselves. During finals week (chapter 16), Sam had muted it to concentrate on the real incident, and it stayed muted. Maya had stopped reading it months earlier.

What let it happen:

1. The alert described an event, not a broken promise. “Deletion job failed” could mean a brief network blip, retried successfully later. It said nothing about whether any student's deletion was overdue.
2. The channel had no precision. With most alerts meaningless, ignoring all of them was a rational response.
3. Nobody owned the alert, and it had no runbook.
4. Tests checked forbidden access, not required access. Least privilege needs both directions tested.

Fix.

1. Finish the deletions. The worker got delete permission on the class library prefix, and the 214 files were deleted, with the drill's check rerun to confirm. Maya asked her lawyer whether the delay needed to be reported or disclosed to the affected former students (chapter 15).
2. Alert on the promise. A page fires when the oldest open deletion request is more than 5 days old, two days before the promise is broken. A daily reconciliation job compares storage with the database and opens a ticket for any file without an owner.
3. Clean the channel. The alert review from the worked example cut 40 alerts a week to about 2. No more muting: an alert that's too noisy gets fixed or deleted the same week.
4. Test both directions. The permission matrix tests now include every “must be able to” cell, including the full account deletion path through every store, run in staging after any permission change.

Lesson. An ignored alert does more harm than a missing one, because it lets the team believe someone is watching. Alert on the promises you've made to users, make every alert one someone must act on, and treat a muted channel as an incident in itself.

Architect's checklist#

Before calling a system observable:

• ☐ The questions you need to answer in production are written down, and each maps to a signal.
• ☐ Each key journey produces one trace across services and background jobs, with the trace ID in logs and error messages.
• ☐ AI outputs and stored derived data record the recipe that produced them.
• ☐ Telemetry contains no personal data, and a check enforces it.
• ☐ Dashboards cover the golden signals, AI quality, and cost, with change markers.
• ☐ Pages are few, urgent, actionable, and based on symptoms or broken promises; everything else is a ticket or a panel.
• ☐ Every alert has an owner and a runbook.
• ☐ An error budget policy is written and agreed.
• ☐ The golden set runs nightly end to end, and fresh production samples are graded weekly.
• ☐ User feedback is collected with consent and minimal data.
• ☐ Postmortem actions are tracked to completion and reviewed for patterns.
• ☐ Improvements are ranked by impact on one shared list.

Exercises#

Beginner. The Meal Planner builds shopping lists in a nightly job. Write five questions you'd need to answer if users said their lists were wrong or missing. For each, name the signal (metric, trace attribute, or log field) that answers it, and decide which one deserves an alert, and whether it's a page or a ticket.

Intermediate. Design the trace for the Trip Planner when a flight delay arrives by webhook and the itinerary is adjusted (chapter 8's exercise). List the spans, their attributes, and how the trace context gets from the webhook handler to the background job. Then write two SLOs for the journey and burn-rate alerts for each, using the SRE workbook's starting thresholds.

Advanced. Build the AI quality dashboard for the Shop Help Chat: outcome mix (answered, handed to a person, declined), return requests started, feedback, and cost per conversation, each broken down by prompt version and model. Define the recipe recorded on each conversation. Then write a plan for a controlled experiment comparing two prompts: the measure, the groups, how long it runs, what you decide before it starts, and what result would make you switch.

Lab, artifact, and review#

Lab. Instrument the Learning Coach's tutor request and note upload end to end with OpenTelemetry, with a trace ID linking API, worker, and logs, and the recipe recorded on spans and chunks. Then debug a simulated production incident: your partner (or a script) introduces a silent fault, such as an extraction change, a slow query, or a prompt edit, without telling you what. Find it using only the dashboards, traces, and logs, and write down how long each step took.

Deliverable. Add to your portfolio:

• the three dashboards (screenshots or definitions) and the questions each answers;
• the alert table, with owners and runbook links;
• one runbook for a page you defined;
• a blameless postmortem of the simulated incident;
• an AI evaluation trend report covering at least four weeks or four runs;
• ADR-014.

Review. Give someone your dashboards and one trace ID from the simulated incident, and nothing else. If they can't find the cause within 30 minutes, note where they got stuck: that's the next signal to add.

Reflection#

• What could get worse in your product without any error, slowdown, or alert?
• When an alert fires, does someone act every time? If not, why does the alert exist?
• Could you tell, for any AI output or stored passage, exactly which versions produced it?
• Which promise to your users is protected by an alert on the promise itself?
• What did your last three incidents have in common?

The Learning Coach is now deployed, secured, able to handle its peaks, and visible from the inside. Chapter 18 closes Part IV with the longer view: maintaining and evolving a system that AI helped build, as its code, its dependencies, and its team change over the years.

References#

• Ewaschuk, R. Monitoring distributed systems. In Site Reliability Engineering. Google.
• Hodgson, P. (2017, October 9). Feature toggles (aka feature flags). martinfowler.com.
• Huyen, C. (2022, February 7). Data distribution shifts and monitoring. huyenchip.com.
• OpenTelemetry. Context propagation. OpenTelemetry documentation.
• OpenTelemetry. Generative AI semantic conventions. OpenTelemetry documentation.
• OpenTelemetry. Signals. OpenTelemetry documentation.
• Thurgood, S. (2018, February 19). Example error budget policy. In The Site Reliability Workbook. Google.
• Thurgood, S., Frame, J., Lenton, A., Quinito, C., Tolchanov, A., & Trdin, N. Alerting on SLOs. In The Site Reliability Workbook. Google.
• W3C. (2021, November 23). Trace Context. W3C Recommendation.

Links checked October 2026.