Deployment and Cloud Operations

Why it matters#

Until now, the book has treated deployment as a step at the end: the code is done, so put it online. In practice, deployment is where the product meets real users, at real times, with real data, and where mistakes stop being private.

AI makes this sharper in two ways. Agents produce changes faster, so more releases go out, and each one is a chance for something to go wrong in production that never went wrong in testing. And agents happily write deployment configuration, migrations, and scripts that look right and haven't met your production environment.

The goal of this chapter isn't to make you a cloud engineer. It's to give you enough understanding to choose an appropriate way to run the product, to make releases repeatable and reversible, and to know what to do when something breaks anyway. DORA, the long-running research program on software delivery, has found repeatedly that speed and stability aren't a tradeoff: teams that deliver well tend to do well on both (Harvey, 2026). Small, frequent, reversible releases are how a two-person team gets both.

Learning objectives#

After this chapter you will be able to:

1. Choose a deployment approach (static hosting, managed platform, serverless, containers, virtual machines, or orchestration) proportionate to the product.
2. Explain the cloud basics an architect needs: compute, networking, storage, identity, domains, TLS, regions, and configuration.
3. Build a pipeline that tests, builds once, and promotes the same artifact from staging to production.
4. Release progressively and roll back safely, including database changes.
5. Operate the deployed product: monitoring, migrations, backups with tested restores, and basic incident response.
6. Produce an infrastructure diagram, a deployment runbook, and a release checklist.

14.1 Deployment choices#

Deployment means putting a version of the product where users can reach it and keeping it running. The options differ mainly in how much of the running you do yourself.

Static hosting and managed backend platforms#

The Learning Coach's web app is React: after it's built, it's a set of files. Static hosting serves those files from many locations around the world, close to users, and handles certificates for you. There's nothing to patch and almost nothing to break.

A managed platform (often called a platform as a service) runs your server code. You give it your code or a container image and your settings; it starts the servers, restarts them when they crash, gives you logs, and scales the number of copies up and down. For a small team, this removes most operational work. The price is less control and, at larger scale, often more cost than running things yourself.

Serverless functions and managed containers#

Serverless functions run a piece of code per request or event, and you pay only while it runs. They suit occasional work, like resizing an uploaded image. They fit less well with long-running work or connections held open, such as the tutor's streamed answers or a database connection pool, and each function adds something to deploy and monitor.

A container packages your code with everything it needs to run (the language runtime, libraries, system files) into an image that runs the same way anywhere. Managed containers run your images for you without you managing the machines. Many managed platforms now run containers underneath, so the line between the two is blurring.

Virtual machines and orchestrated containers#

A virtual machine is a rented computer. You install and patch the operating system, configure the web server, set up restarts, and handle everything else. It gives full control and full responsibility.

Orchestration, most commonly Kubernetes, runs many containers across many machines, restarting, scaling, and connecting them according to configuration. It's powerful and widely used at scale, and it's a large system to learn and operate. For a product with one API and one worker, it's usually far more machinery than the problem needs.

Selecting complexity appropriate to scale#

Choose the least operational work that meets your real requirements, and write down what would make you change. For the Learning Coach, with a few thousand students, one API, one background worker, and two people:

• the web app on static hosting;
• the API and worker as two services on a managed platform, from one container image;
• managed PostgreSQL, with point-in-time recovery;
• managed object storage for uploaded notes, with versioning.

Agents tend to propose impressive architectures: clusters, service meshes, multiple regions. Ask what problem each piece solves for your users this year. If nobody can say, leave it out.

14.2 Cloud fundamentals for architects#

Compute, networking, storage, identity#

Every cloud setup combines four kinds of building block:

• Compute runs code: platform services, containers, functions, virtual machines.
• Networking connects things: public addresses, private networks between your services, and rules about who may connect to what. Your database should accept connections only from your own services, never from the whole internet.
• Storage keeps data: the managed database, object storage for files, and backups.
• Identity decides who and what may do things: people's accounts, and service identities for your code. Give each the least access it needs. The production API needs to read and write its database; it doesn't need to delete backups. Agents get no production access at all (ADR-008, chapter 11).

Keep staging and production in separate accounts or projects where your provider allows it, so a mistake in one can't touch the other. The 3 a.m. migration in chapter 11 reached a shared staging database because nothing kept it apart.

Domains, TLS, DNS, and reverse proxies#

DNS is the internet's directory: it turns learningcoach.app into the address of a server. TLS encrypts traffic between the browser and your server; it's what puts the padlock in the address bar, using a certificate that proves you control the domain. A reverse proxy sits in front of your servers, receives every request, handles TLS, and passes requests on; managed platforms provide one.

Certificates expire, and they're getting shorter-lived. Let's Encrypt, a widely used free certificate authority, announced in December 2025 that it would cut certificate lifetimes from 90 days to 45 by 2028, in line with industry-wide requirements, and recommended automated renewal with monitoring that alerts if a certificate isn't renewed on time (McPherrin, 2025). Let your platform renew certificates automatically, and add an expiry check to your monitoring anyway.

Region, availability, and data residency#

Cloud providers group data centers into regions (a geographic area) and, within a region, availability zones. AWS describes an availability zone as one or more separate data centers with their own power, networking, and connectivity, far enough from the others to avoid shared failures like floods or power cuts, but close enough for fast replication (AWS, Availability Zones). Other providers use similar ideas under similar names.

For the Learning Coach, a managed database with a standby copy in a second availability zone covers most hardware failures. Running in several regions is far more complex and rarely justified for a product this size.

Data residency is where data is physically stored, and some customers or laws require it to stay in a particular country or region. A school district may require student data to stay in its country. Choose the region with your users and contracts in mind, check that backups and outside services (the AI provider, email, logs) respect it too, and get legal advice on what applies (chapter 7 covered compliance by domain).

Configuration and secrets#

Chapter 4 introduced configuration: values that differ between environments, kept in environment variables rather than code. Secrets are configuration that grants access: database passwords, API keys, signing keys.

• Keep secrets in the platform's secret store or a secrets manager, never in the repository, the image, or a chat with an agent.
• Use different secrets in each environment, so a leaked staging key can't reach production.
• Know how to rotate each one, and practice it: replace the secret, deploy, confirm, revoke the old one.
• Keep a list of every setting each environment needs, checked automatically at startup. A service that's missing QUIZ_MODEL should refuse to start with a clear message, and the pipeline should catch the gap before a release goes out, not during an incident.

14.3 Reproducible builds and CI/CD#

Continuous integration (CI) runs checks on every change. Continuous delivery (CD) makes every change that passes deployable, through an automated pipeline. A reproducible build produces the same result from the same inputs, so what you tested is exactly what you ship.

flowchart LR
    PR["Pull request"] --> CI["CI: tests, gate checks<br/>(chapter 13)"]
    CI -->|merge| B["Build image once<br/>tagged with commit ID"]
    B --> REG[("Image registry")]
    REG --> STG["Deploy to staging<br/>migrations, smoke tests"]
    STG --> AP{"Release approved?<br/>gate report + human"}
    AP -->|no| X["Stop; fix forward<br/>in a new change"]
    AP -->|yes| CAN["Production canary:<br/>new version takes a small share"]
    CAN --> W{"Error rate and latency<br/>normal for 15 min?"}
    W -->|no| RB["Roll back to<br/>previous image"]
    W -->|yes| ALL["All traffic on<br/>new version"]

Automated tests before deployment#

The release gate from chapter 13 runs in the pipeline, not on someone's laptop. Nothing reaches staging without passing CI, and nothing reaches production without passing staging and the gate. Add smoke tests after each deploy: a handful of quick checks that the deployed system is alive (the home page loads, sign-in works, a plan can be read). They don't replace the test suite; they confirm the deployment itself worked.

Container images and dependency locking#

Build the image once, from a commit, and tag it with that commit's ID. Deploy that same image to staging and then to production. Rebuilding for production gives you an artifact you never tested.

Make the build itself repeatable:

• Lock dependencies. Commit the lockfile (chapter 11) and install exactly from it.
• Pin the base image. Docker's documentation points out that image tags can be changed by their publisher, so the same tag can give you a different image months later; pinning to a digest, a fingerprint of the exact image, guarantees the same one every time (Docker, Building best practices).
• Keep secrets and local files out. A .dockerignore file excludes files from the build, such as .env files and local data. The failure clinic shows why this matters.

Staging and production separation#

Staging is a copy of production used to try releases before users see them. It's only useful if it's like production. The Twelve-Factor App calls this dev/prod parity: keep the time, people, and tools gaps between development and production small, and use the same type and version of every backing service, because small differences cause code that passed in one place to fail in the other (Wiggins, Dev/prod parity).

For the Learning Coach, that means: the same image, the same PostgreSQL version, the same settings list (with different values), and its own database, filled with realistic made-up data. Never copy production data into staging without removing personal information, and treat staging access as carefully as you would treat the data in it.

Progressive releases and rollbacks#

A progressive release exposes a new version to some users before all of them. A canary release routes a small share of traffic to the new version first, watches it, and widens the share as confidence grows; rolling back means sending everyone back to the old version (Sato, 2014). Many managed platforms support this directly. If yours doesn't, a feature flag, a setting that turns a feature on for some users or none, gives you a similar control at the feature level.

The same article points out the catch that bit Maya on Tuesday: during a canary, two versions of the code run at once against one database, so the database has to support both (Sato, 2014). It names the technique parallel change; chapter 7's lab called it expand, migrate, contract. For the column rename:

1. Expand. Add due_on alongside due_date. The code writes both and reads due_date.
2. Migrate. Copy existing values. The code reads due_on, still writing both.
3. Contract. Once no running version reads due_date, remove it, in a later release.

Each step can be rolled back on its own, because at every point both the current and the previous version of the code work with the database. That's what “rollback ready” has to mean: the previous image can run against the current database. Check it in staging by actually rolling back.

14.4 Operating the deployed application#

Environment monitoring#

You need to know something is wrong before students tell you. Start with a few checks that cover most failures:

• Uptime checks from outside: is the site reachable, and does sign-in work?
• Error rate and latency for the API, compared with normal.
• Health of each service: running, restarting, out of memory.
• The jobs nobody watches: backups, the deletion worker, certificate renewal, the notifications queue. Each should report success, and silence should raise an alert.

That last point is the lesson of a well-known outage. In January 2017, an engineer at GitLab deleted the data directory on the primary production database, believing it was the secondary. When the team turned to backups, none of the usual methods produced a usable recent copy. The nightly backups had been failing for some time, because the backup tool's version didn't match the database's, and the failure emails were being rejected by the mail server, so nobody knew. GitLab recovered from a copy an engineer had made by hand about six hours earlier, and lost roughly six hours of database changes (GitLab, 2017). Monitor that your safety nets work, not just that they're switched on.

Chapter 17 covers observability in depth: metrics, logs, traces, and alerts that people trust.

Database migrations#

Migrations are the riskiest part of most releases, because code can be rolled back and data often can't.

• Every migration must work with the previous version of the code: expand, migrate, contract, as above.
• Run migrations from the pipeline, as their own step, never from a laptop or an agent session (chapter 11's staging incident).
• Set a lock timeout. Some changes lock a table while they run; on a large table at a busy hour, that's an outage. A timeout makes the migration fail quickly instead of blocking every request.
• Try migrations on realistic data in staging first, and time them.
• Review the migration line by line. It's a risky zone (chapter 4), and agents often write the one-step version.

Backup and recovery#

Chapter 7 set the targets: data loss of no more than minutes (the RPO) and recovery within a few hours (the RTO), with continuous archiving for point-in-time recovery and backups that expire after 14 days. Operations is where those targets become real:

• Turn on point-in-time recovery and check the setting, don't assume it.
• Practice restoring. Every quarter, restore to a scratch environment at a chosen moment, time it against the RTO, and check that the app works with the restored data.
• Write the restore runbook while practicing, not during an outage.
• Remember the deletion promise. NFR-6 says that after a restore, deletions recorded since the backup are applied again before the system goes live. The runbook includes that step, and the drill tests it.

Incident response basics#

An incident is an unplanned event that harms or threatens the service. Google's SRE book describes how incidents spiral when everyone focuses on the technical problem, nobody communicates, and people make changes on their own without coordinating. It recommends separate roles: an incident commander who holds the overall picture and assigns work, an operations lead who is the only one changing the system, and a communications lead who keeps people informed. It also advises declaring an incident early, for example when users can see the problem or it isn't solved after an hour, and keeping a live document of what's happening (Stribblehill, Managing Incidents).

A two-person team can't fill every role with a different person, but it can apply the idea: one person changes production, the other communicates, and both write things down. On Tuesday, both changed production and neither communicated.

Afterward, write a postmortem: what happened, the impact, what was done, the causes, and the actions that will prevent it happening again. The SRE book stresses that it should be blameless: it assumes people acted reasonably with the information they had and looks for what in the system let the mistake happen, because you can fix systems and processes but you can't fix people by blaming them (Lunney & Lueder, Postmortem Culture).

To see whether releases are getting safer, track a few of DORA's delivery measures: how often you deploy, how long a change takes to reach production, how many deployments need immediate intervention (the change fail rate), and how long it takes to recover from a failed deployment (Harvey, 2026). DORA warns against turning them into targets; use them to spot trends.

Architect's decision: how do we run the Learning Coach?#

AI collaboration recipe#

Agents are useful for drafting configuration, runbooks, and checklists, and for reviewing migrations and incident timelines. They don't get production credentials, and changes to infrastructure go through pull requests that a person applies.

1. Migration review.

Here is a migration and the code changes in the same pull request. Can the previous version of the code run correctly against the database after this migration? If not, rewrite the plan as expand, migrate, contract steps across separate releases, and say what each release contains. Also flag any statement that could lock a large table.

2. Environment comparison.

Here are the settings names (not values) for staging and production, and the list of settings the code reads at startup. List every setting missing from either environment, every one the code reads that neither defines, and every one defined that the code no longer reads.

3. Runbook draft.

From our infrastructure diagram, ADR-011, and the platform's documentation linked here, draft a runbook for: deploying a release, rolling back, restoring the database to a moment in time including the NFR-6 deletion replay, and rotating the AI provider key. Each step should say who does it, the exact action, how to confirm it worked, and what to do if it didn't. Mark anything you're unsure about as a question.

4. Incident timeline.

Here are the alerts, deploy log, chat messages, and notes from Tuesday's incident. Build a timeline with times, what was observed, what was done and by whom, and the effect. Don't assign blame; note where information was missing or actions overlapped.

Boundaries to state every time:

• “Never use, request, or print production credentials or secrets.”
• “Infrastructure and pipeline changes are pull requests; a person applies them.”
• “Every migration must work with the previous version of the code.”
• “Mark guesses about the platform's behavior as questions, with a link to the documentation you'd check.” (Platform features change; recheck the runbook against the documentation at least every six months, last reviewed October 2026.)

Worked example: a deployment you can reverse#

Maya and Sam spent the following week rebuilding how the Learning Coach was deployed.

1. The infrastructure diagram. First they drew what actually existed, then what ADR-011 called for:

flowchart TB
    U(["Students and instructors"]) -->|"HTTPS, learningcoach.app"| CDN["Static hosting<br/>web app (React build)"]
    U -->|"HTTPS, api.learningcoach.app"| PX["Platform reverse proxy<br/>TLS, auto-renewed certificates"]
    subgraph PROD["Production project (in-country region)"]
        PX --> API["API service<br/>image tagged with commit ID<br/>2+ instances"]
        W["Worker service<br/>same image"]
        DB[("Managed PostgreSQL<br/>standby in 2nd zone<br/>point-in-time recovery, 14-day retention")]
        OBJ[("Object storage<br/>uploaded notes, versioned")]
        SEC["Secret store"]
        API --> DB
        API --> OBJ
        W --> DB
        W --> OBJ
        SEC -.->|"injected at start"| API
        SEC -.-> W
    end
    API --> AI["AI model provider"]
    W --> MAIL["Email provider"]
    STG["Staging project<br/>same image and versions,<br/>own database, synthetic data"]
    CI["CI pipeline"] -->|"deploys"| STG
    CI -->|"promotes same image"| PX

Drawing it found two problems at once: the database accepted connections from any address, and the worker read the AI provider key from a plain environment variable set by hand. Both were fixed before anything else.

2. The environments table.

A startup check now compares the settings each service needs with what it receives, and the pipeline runs the same check against production's setting names before every release. It found QUIZ_MODEL and two others missing.

3. The pipeline. Merges to main build one image tagged with the commit ID. The pipeline deploys it to staging, runs migrations and smoke tests, and attaches the gate report. Maya or Sam approves. Production gets the same image as a canary taking 10% of traffic for 15 minutes; if errors or response times rise above normal, the platform shifts traffic back automatically. Deploys moved to mid-morning, outside the evening peak.

4. The rename, done properly. The due_on change went out as three releases over two weeks: add the column and write both; read the new one; drop the old one. Before each, they deployed the release in staging and rolled it back, checking that the previous version worked against the migrated database.

5. The drills. In staging, Sam deployed a build with a deliberate error on the plan page and rolled it back: 3 minutes from alert to recovery. Then Maya turned on point-in-time recovery in production (it had been off), waited a day, and restored production to a scratch environment as of a chosen minute. It took 52 minutes, inside the RTO. The NFR-6 deletion replay ran before the restored copy was opened, and the drill checked that three test accounts deleted after the restore point stayed deleted.

6. The runbook and incident plan. The runbook lives in the repository, beside the code it describes:

## Roll back a release
Who: whoever is operations lead for the incident (one person only)
1. Platform: services > api > deployments > previous image > "redeploy"
   (worker: same image ID)
2. Confirm: smoke tests pass; error rate back to normal within 5 min
3. If the database changed in this release: confirm the release notes
   say "previous version compatible"; if not, stop and fix forward
4. Post in #status: "Rolled back to <image>, investigating"

## During an incident
- First person to notice declares it in #incident and is commander
  until they hand over explicitly
- Commander names the operations lead; nobody else changes production
- The other person updates the status page within 10 minutes,
  then every 30 minutes
- Keep a timeline in the incident doc as you go

They wrote a blameless postmortem for Tuesday. Its actions were these changes, plus one more: the release report's “rollback ready” line now has to link to the staging rollback test for that release.

Verification: can you deploy and recover?#

Failure clinic: the key in the image#

Symptom. On the first of the month, the AI provider's bill was eleven times the usual amount. The provider's usage page showed millions of requests over three nights from addresses that weren't the Learning Coach's servers, all using the production API key.

Reproduce. Maya rotated the key at once, which stopped the requests, and then looked for how it had leaked. The key had never been committed to the repository; the secret scanner from chapter 13 would have caught that. She pulled the production container image from the registry and listed its files. In one of the image's layers, the build steps it's made from, was a file named .env, containing the production key.

Diagnose. Months earlier, an agent had written the first Dockerfile. Its copy step copied the whole project folder into the image. There was no .dockerignore, so the .env file on the build machine, which held production values for a one-off test, went into the image. A later step deleted the file, which made it disappear from the running container but not from the earlier layer, where anyone with the image could still read it. The registry had been set to public during a debugging session and never set back.

The problem was common enough to have a name in industry reports: GitGuardian's 2025 report found that 98% of the secrets it detected in DockerHub images were in image layers (GitGuardian, 2025).

What let it happen:

1. Secrets lived in a file on the build machine, instead of only in the secret store at runtime.
2. Nothing limited what went into the image. No .dockerignore, and nobody reviewed the Dockerfile as a risky file.
3. The secret scanner only looked at the repository, not at built images.
4. The registry's visibility was a setting nobody checked, and the AI provider account had no spending limit.

Fix.

1. Rotate and revoke. The leaked key was revoked; every other secret that had ever been in a .env file on that machine was rotated too.
2. Secrets only at runtime. Services read secrets from the secret store when they start. No .env file holds production values anywhere.
3. Limit the build. A .dockerignore excludes .env*, local data, and test files, and the Dockerfile copies only the folders it needs.
4. Scan images. The pipeline scans each built image for secrets and known vulnerabilities before pushing it, and the registry is private, with a monitoring check that alerts if that changes.
5. Cap the damage. The AI provider account has a monthly spending limit and an alert at 150% of normal daily usage.
6. Treat build files as risky. Dockerfile, pipeline configuration, and infrastructure settings joined the risky zones that need line-by-line review.

Lesson. Deleting a secret doesn't remove it from history, whether that history is a commit or an image layer. Keep secrets out of everything you build, give them only to running services, and make sure the bill can't grow without someone knowing.

Architect's checklist#

Before calling a deployment production-ready:

• ☐ The deployment approach is the least complex that meets real requirements, recorded in an ADR.
• ☐ The database is reachable only from your services; identities have least access; agents have none in production.
• ☐ TLS certificates renew automatically and expiry is monitored.
• ☐ The region and every outside service meet residency requirements, with legal advice where needed.
• ☐ Secrets live in a secret store, differ per environment, and can be rotated.
• ☐ Missing configuration is caught at startup and before release.
• ☐ One image per commit is promoted from staging to production; dependencies and base images are pinned.
• ☐ Staging matches production in image and service versions, with its own non-personal data.
• ☐ Releases are progressive, with automatic rollback on errors.
• ☐ Every migration works with the previous code; rollback is tested in staging.
• ☐ Monitoring covers uptime, errors, latency, and jobs that fail silently.
• ☐ Point-in-time recovery is on, and restores are practiced against the RTO.
• ☐ Incident roles, status updates, and blameless postmortems are agreed in advance.

Exercises#

Beginner. For the Meal Planner (a web app, an API, a database, and a nightly job that builds shopping lists), choose a deployment option for each part and justify it in one sentence. Then fill in an environments table for local, staging, and production: image, database, data, secrets, and who can deploy.

Intermediate. In the Trip Planner, a price field is stored as a decimal number, and you need to change it to an amount in cents plus a currency code. Write the expand, migrate, contract plan across releases. For each release, state what the code reads and writes, and how you'd confirm that rolling back to the previous release is safe. Then ask an agent to write the one-step migration, and use recipe prompt 1 to review it.

Advanced. Run a tabletop incident exercise for the Shop Help Chat: at 7 p.m. on a sale day, the assistant starts telling customers their returns were approved when they weren't. With two or three people, assign incident roles, decide what to communicate to customers and when, choose between rolling back the assistant, turning it off with a feature flag, or fixing forward, and write the blameless postmortem, including at least three system changes.

Lab, artifact, and review#

Lab. Deploy the Learning Coach to a managed platform, with separate staging and production. Release a change that includes a database migration using expand, migrate, contract. Then deploy a deliberately broken version to staging and perform a safe rollback, timing it. Finally, restore the staging database to a chosen moment.

Deliverable. Add to your portfolio:

• the infrastructure diagram, in Mermaid;
• the environments table;
• the deployment runbook, covering deploy, rollback, restore, and secret rotation;
• the release checklist, extending chapter 13's gate with deployment checks;
• the timings from your rollback and restore drills;
• ADR-011.

Review. Give someone who has never deployed your product only the runbook, and ask them to roll back a staging release while you watch without helping. Every question they ask is a missing line in the runbook.

Reflection#

• If your production database disappeared right now, how much data would you lose, and how long would recovery take? How do you know?
• What does your rollback button actually roll back?
• Which setting exists in one of your environments and not another?
• In your last incident, who was changing production, and who was telling users?

Chapter 15 turns to security, privacy, and responsible AI: threat modeling, protecting data, and the risks that come with putting AI in front of users.

References#

• AWS. Availability Zones. AWS Fault Isolation Boundaries whitepaper.
• Docker. Building best practices. Docker documentation.
• GitGuardian. (2025, March 11). The state of secrets sprawl 2025. GitGuardian Blog.
• GitLab. (2017, February 10). Postmortem of database outage of January 31. GitLab Blog.
• Harvey, N. (2026). DORA's software delivery performance metrics. DORA.
• Lunney, J., & Lueder, S. Postmortem culture: Learning from failure. In Site Reliability Engineering. Google.
• McPherrin, M. (2025, December 2). Decreasing certificate lifetimes to 45 days. Let's Encrypt.
• Sato, D. (2014, June 25). Canary release. martinfowler.com.
• Stribblehill, A. Managing incidents. In Site Reliability Engineering. Google.
• Wiggins, A. Dev/prod parity. The Twelve-Factor App.

Links checked October 2026.