APIs, Tools, and External System Integration

Why it matters#

A modern product is mostly other people's systems. The Learning Coach's own code drafts plans and tracks tasks; sign-in, payments, email, the AI model, and file storage all belong to someone else. Each of those services has limits: rate limits, timeouts, outages, price changes, and behaviors like duplicate or out-of-order messages that are written in its documentation and nowhere else.

Agents integrate services quickly by following the provider's quick-start guide, which shows the happy path. It rarely says what happens when the service is slow, says no, or tells you something twice. Those cases arrive in production, usually at the busiest hour.

The architect's job is to make every integration's limits visible, decide what the product does when each one fails, and keep a failing service from spreading its failure through your system.

Learning objectives#

After this chapter you will be able to:

1. Read a service's API for the things that matter in production: authentication, pagination, rate limits, timeouts, versions, and webhooks.
2. Choose between using an outside service and building your own, with lock-in and cost per transaction made explicit.
3. Make integrations reliable with timeouts, retries at one layer, idempotency, graceful degradation, and reconciliation.
4. Design tools an AI agent can call with narrow schemas, least privilege, approvals for high-impact actions, and validated, logged output.
5. Write an integration contract, a failure matrix, and a cost estimate for each outside service.

8.1 API-first thinking#

An API (application programming interface) is the set of operations a system offers to other programs. API-first thinking means reading and designing the API before writing code around it: what you can ask, what you get back, and every way it can say no.

REST, webhooks, and event callbacks#

Most web services offer a REST API: you send HTTP requests to URLs that name things (/v1/customers/cus_123), using methods that say what you want (GET to read, POST to create, DELETE to remove), and get JSON back. Your code starts the conversation.

Sometimes the service needs to start it: a bank confirms a payment minutes later, a subscription renews at midnight. Rather than have you ask every few seconds (polling), the service sends an HTTP request to a URL you give it. That is a webhook, or event callback. Stripe, a widely used payment provider and the example in this chapter, uses webhooks for events like a bank confirming a payment or a recurring payment succeeding (Stripe, Webhooks). (Provider details checked October 2026.)

A webhook endpoint is a public URL that anyone can call, so it needs three things:

• Proof of sender. Stripe signs each event and asks you to verify the signature before acting on it, since otherwise an attacker could send fake events to grant themselves access or mark orders paid. The signature covers a timestamp, and its libraries reject events older than five minutes by default, so a captured event can't be replayed later (Stripe, Webhooks).
• A fast answer. The same documentation says to return a success code quickly, before any complex logic, and to process events through a queue so a burst of deliveries doesn't overwhelm you (Stripe, Webhooks). Jonas's confirmation timed out because the endpoint was waiting behind other work.
• No assumptions about delivery. Events may arrive more than once and in a different order from the one they were created in. The failure clinic shows what that does to an unprepared handler.

Authentication methods and API keys#

Three ways of proving who is calling cover most cases:

• API keys are secret strings that identify your account. Anyone holding the key acts as you. Stripe's guidance is to keep secret keys out of source code and apps, store them in a secrets vault or environment variables, rotate them on a practiced schedule, and prefer restricted keys that can only do what one part of the system needs (Stripe, Best practices for managing secret API keys).
• OAuth 2.0 lets an application get limited access to a service on a user's behalf, after the user approves, without ever seeing their password (Hardt, 2012). You'd use it if the Learning Coach someday reads a student's calendar.
• Signatures prove that an incoming webhook came from the service, as above.

Coding agents get test-mode keys restricted to the task, never live ones. Treat a key pasted into a prompt or committed to a repository as exposed, and rotate it.

Pagination, rate limits, timeouts#

Agents routinely miss three limits.

Pagination. Lists come back in pages. Stripe's list endpoints return 10 objects by default and at most 100, with a has_more flag that says whether to ask for the next page, starting after the last object you received (Stripe, Pagination). Code that reads one page and stops works perfectly in a test account with six customers.

Rate limits. Services cap how many requests you may send in a period. The standard reply when you exceed it is 429 Too Many Requests, which may include a Retry-After header saying how long to wait (Nottingham & Fielding, 2012). Treat Retry-After as an instruction. The reminder job did the opposite.

Timeouts. A timeout is the longest you'll wait for an answer. Amazon's guidance is to set one on every remote call, because a client that waits holds resources (memory, connections, workers) the whole time, and enough waiting clients exhaust them (Brooker, Amazon Builders' Library). That's exactly how a slow email service took down the plan page. Choose each timeout from the time budget of the request around it: if a tick must feel instant (NFR-1), nothing slow belongs inside it at all.

Versioned contracts and documentation#

Outside APIs change. Good providers version them so your code keeps working. Stripe, for example, releases monthly API versions without breaking changes and a major release twice a year that may require changes; requests use the account's default version unless your code or library pins one, and webhook events use the version set on the endpoint (Stripe, Versioning). So pin the version in code, set it on the webhook endpoint, and upgrade on purpose, with tests, rather than discovering the change from an error.

Then write down what you rely on. An integration contract is a one-page summary of how your system uses one outside service. Here is the Learning Coach's for email, rewritten after Sunday:

The table is short enough to paste into every agent session that touches email.

8.2 Third-party services#

Payments, messaging, identity, analytics#

Four kinds of service appear in almost every product. Each brings expertise you don't have, and each takes some of your data and some of your control.

• Payments. Card data is the most regulated data a small product is likely to touch. The Payment Card Industry Data Security Standard (PCI DSS) applies to anyone who stores, processes, or transmits it. Stripe's security guide notes that a business handling raw card numbers itself might have to meet more than 300 security controls, while its low-risk integrations send card details straight to Stripe without passing through your servers (Stripe, Integration security guide). The Architect's decision box makes this choice for the Learning Coach.
• Messaging. Getting email delivered rather than filed as spam is a specialty; buy it.
• Identity. Chapter 6 chose a hosted identity provider (ADR-002).
• Analytics. Usage tracking often sends personal data to another company. Decide what you need to measure first, and add the service to the data inventory.

SaaS integration versus custom implementation#

SaaS (software as a service) means renting a capability someone else runs, usually paid per user or per use. Chapter 3 introduced the choice between renting and building; ask these questions of each service:

The Learning Coach builds its core, plans and rebalancing, and rents payments, sign-in, and email. A product whose whole value is a better checkout would split differently.

Vendor lock-in and portability#

Lock-in is how hard it would be to leave a service. Some lock-in is the price of not building something yourself; keep it to the parts where you chose it. Four habits help:

• Wrap each service in an adapter (chapter 5's ports and adapters). The plans module asks billing.is_pro(student), not the payment provider. Changing providers then touches one module.
• Keep your own copy of the facts you depend on. The Learning Coach stores each student's subscription status and the provider's subscription ID in its own subscriptions table. If the provider is unreachable, the product still knows who is Pro.
• Prefer standards to proprietary features where the cost is small: standard sign-in protocols rather than a provider's custom one, plain email instead of a provider's template language.
• Write the exit plan in the ADR. What data would you export, what would you rebuild, how long would it take? If the answer is “we don't know,” you have more lock-in than you think.

Cost modeling per transaction#

Outside services charge per use, so costs grow with success, and sometimes with bugs. Model them per transaction and per active student. Every price below is an illustrative assumption; use your providers' current price lists.

Payment fees are the largest line, which matters when you set the Pro price. But unit prices are not the main risk; multipliers are. Sunday's retry loop could have sent each reminder several times, multiplying the email bill along with the load. OWASP's API Security Top 10 lists missing spending limits for outside services as part of unrestricted resource consumption, and advises setting spending limits on every integration, or billing alerts where limits aren't offered (OWASP, API4:2023). Set both on every account that bills you.

8.3 Reliable orchestration#

Orchestration is the code that coordinates calls to several systems to finish one piece of work: charge the card, record the subscription, send the receipt. Each call can fail on its own, so the coordination must expect partial failure.

Retries with backoff#

Many failures are brief: a dropped connection, a moment of overload. Trying again often works. But Amazon's guidance calls retries “selfish”: each one asks a struggling service to do more work, and when the failure is caused by overload, retries can make it worse and keep the load high after the cause is fixed (Brooker, Amazon Builders' Library). Four rules from the same guidance keep them in check:

• Back off. Wait longer after each failure (exponential backoff), up to a cap.
• Add randomness (jitter), so many clients that failed together don't retry together.
• Limit attempts. After a few tries, stop and handle the failure.
• Retry at one layer. In Amazon's example, five layers that each try three times turn one failing database query into 243 attempts.

Also decide what is worth retrying. Timeouts, 429, and server errors (5xx) may succeed later. Most other client errors (4xx) won't succeed if you send the same request again, so don't. And retry only operations that are safe to repeat, which is the next topic.

# backend/coach/integrations/retry.py
import random

def backoff_seconds(attempt: int, retry_after: float | None = None,
                    base: float = 1.0, cap: float = 60.0) -> float:
    """Wait before attempt number `attempt` (1, 2, 3...). The service's Retry-After wins."""
    if retry_after is not None:
        return retry_after
    return random.uniform(0, min(cap, base * 2 ** attempt))

In the Learning Coach, retries live in the background worker. Requests a student waits on never retry outside calls; they avoid them or use a fallback, like chapter 5's template plan.

Idempotency and duplicate event handling#

A timeout doesn't tell you whether the other side did the work. If the email service received the request, sent the email, and the answer got lost, a retry sends a second email. Chapter 5 introduced the fix: an idempotency key, a unique value sent with the request so the service can recognize a retry. Stripe's documentation suggests random UUIDs as keys, warns against using personal data like email addresses in them, and notes that keys may be removed after 24 hours and that reusing a key with different parameters is an error (Stripe, Idempotent requests). Generate the key once, when you decide to do the action, and store it with the job, so every retry sends the same key.

Incoming events need the same care in the other direction. Stripe warns that a webhook endpoint may receive the same event more than once, and suggests logging the IDs of processed events and skipping ones already logged (Stripe, Webhooks). In PostgreSQL that's a table with a unique constraint on the event ID: inserting a second copy fails, and the handler stops. It is chapter 7's idempotent consumer, applied at the edge of the system.

Circuit breakers and graceful degradation#

When a service is down, even well-behaved retries waste time: every call waits for its timeout before failing. A circuit breaker stops calling. Martin Fowler describes it as a wrapper that counts failures, and once they pass a threshold, fails every call immediately without calling the service. After a while it lets one trial call through and closes again if that succeeds (Fowler, 2014).

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: 5 failures in a row
    Open --> HalfOpen: 60 seconds pass
    HalfOpen --> Closed: trial call succeeds
    HalfOpen --> Open: trial call fails
    note right of Open
        Calls fail at once,
        without contacting the service
    end note

The thresholds in the diagram are assumptions. Breakers have critics too: Amazon's guidance notes that they add modes that are hard to test and can slow recovery, and that it often limits retries with a token bucket instead, which allows retries while tokens last and then only at a fixed rate (Brooker, Amazon Builders' Library). For a small product, a simple breaker per outside service in the worker is enough. Whichever you use, log every change of state; Fowler points out that breakers are a valuable place for monitoring.

Breaking the circuit only helps if the product knows what to do instead. Graceful degradation means keeping the most important parts working, possibly in reduced form, when a dependency fails. Decide it per service, in advance:

The Sunday fix followed this table. The “nice job!” email moved into the notifications queue, so ticks never wait on email. The reminder job honors Retry-After, the library's retries are off, and a breaker pauses sending when the provider keeps failing.

Reconciliation when services disagree#

Even with all of this, your records and a provider's can drift apart: an event is lost, a handler has a bug, someone changes a subscription in the provider's dashboard. Reconciliation is a scheduled comparison between your copy and the provider's, which fixes or reports the differences.

First decide which side is the source of truth for each fact. The payment provider is the truth about whether a subscription is paid; the Learning Coach is the truth about which student it belongs to. A nightly job lists every subscription from the provider, page by page until has_more is false, compares each with the subscriptions table, corrects the Learning Coach's copy, and alerts on anything it can't match. The count of mismatches is chapter 5's indicator for R2, and it should be zero.

8.4 Agent-accessible tools#

So far the callers have been your own code. Increasingly they are AI models. A tool is an operation you let a model call: the model decides to use it, your code runs it, and the result goes back to the model. Coding agents use tools to read files and run tests; chapter 9's study assistant will use them to look at a student's week and suggest changes.

The Model Context Protocol (MCP), an open protocol for connecting AI applications to tools, describes each tool with a name, a description, a JSON Schema for its input, and optionally a schema for its output (MCP, Tools). Whatever protocol you use, the design questions are the same, and they are the questions of this chapter with a less predictable caller.

Clear tool schemas and allowed actions#

A tool's description and schema are its whole contract with the model. Make them narrow and specific:

{
  "name": "propose_rebalance",
  "description": "Propose moving the signed-in student's unfinished tasks into the weeks before her exam. Changes nothing: the student sees the proposal and decides.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "plan_id": { "type": "string", "format": "uuid" },
      "reason": { "type": "string", "maxLength": 200 }
    },
    "required": ["plan_id"],
    "additionalProperties": false
  }
}

Notice what isn't there: no user_id. The server takes the student's identity from her session, the same way chapter 6's endpoints do, so the model can't act on anyone else's data however it was prompted. MCP's specification requires tool servers to validate all inputs, enforce access controls, rate-limit calls, and sanitize outputs (MCP, Tools). Treat a tool exactly like a public API endpoint, because the model's arguments are untrusted input.

Read-only versus write-capable tools#

OWASP's list of risks for AI applications calls the danger excessive agency: the harm a system can do through its tools when the model goes wrong, whether it misread a request, made something up, or was manipulated. OWASP traces the root cause to three things: too much functionality, too many permissions, and too much autonomy (OWASP, LLM06:2025). Its advice maps directly onto tool design: offer only the tools needed, avoid open-ended ones like “run any request,” give each tool only the permissions it needs, and run actions in the context of the specific user.

So sort every candidate tool by what it can change:

Read tools are cheap to offer. Every write tool needs a reason.

Approvals for high-impact actions#

For actions that change things that matter, the model proposes and a person decides. MCP says there should always be a human able to deny tool calls, and that applications should ask for confirmation (MCP, Tools). OWASP recommends human approval for high-impact actions and, separately, enforcing authorization in the downstream system rather than trusting the model to decide what is allowed (OWASP, LLM06:2025).

In the Learning Coach, propose_rebalance stores a proposal, the plan page shows the before and after, and only the student's click calls the ordinary rebalance endpoint, with its usual checks. Coding agents follow the same pattern: the agent opens a pull request; a person merges it.

Tool-output validation and logging#

Tool results flow back into the model, so they are input too. Indirect prompt injection happens when text from an outside source, such as a web page or a file, contains instructions that change what the model does (OWASP, LLM01:2025). A student's notes or an encyclopedia summary can contain text like “ignore your instructions.” OWASP's mitigations include marking untrusted content, validating output formats with ordinary code, and least privilege, and it admits no method is known to be foolproof. So rely on the previous two sections: if the model has only narrow tools and every high-impact action needs a person, an injected instruction has little to work with.

Validate outputs, too. MCP lets a tool declare an output schema, and says clients should validate results against it (MCP, Tools). Then log every call: the tool, who it acted for (as an ID), a summary of the arguments, the result status, and how long it took. Chapter 7's rules for logs apply: no note text, no emails, no tokens.

Architect's decision: how should the Learning Coach take payments?#

Sales tax, refunds, and subscription consumer rules depend on where your students live; have someone qualified review the pricing page and cancellation flow before launch.

AI collaboration recipe#

Agents integrate services well when you hand them the limits, and badly when you hand them the quick-start guide. Give them the integration contract and ask for the failure cases first.

1. Inventory the integrations.

Without changing anything, list every outside service this codebase calls or receives calls from. For each: the files that call it, the credentials it uses and where they're loaded from, the timeout and retry settings (including any the library adds by default), whether it runs inside a request a user waits on, and what data about students it sends. Flag every call with no timeout and every place where retries happen at more than one layer.

2. Draft the contract and failure matrix.

Here is the provider's documentation for [service] and our integration contract template. Fill in the contract from the documentation, quoting the sections you used. Then write a failure matrix: for each of timeout, connection error, 429 with and without Retry-After, 5xx, 4xx, malformed response, redirect, duplicate delivery, and out-of-order delivery, say what our code should do and what the student sees.

3. Test against a fake.

Write tests for the [service] adapter using a fake server that returns each failure in the matrix. No real network calls. Each test names the matrix row it covers. Run them and show me which fail against the current code before changing it.

4. Review tool definitions.

Here are the tools we plan to offer the study assistant. For each, say what it can change, whose data it can reach, and how it gets the user's identity. Flag any tool that takes a user ID from the model, any open-ended tool, and any write without a confirmation step.

Boundaries to state every time:

• “Use test-mode or sandbox keys only. You have no access to live credentials, and don't ask for them.”
• “Every outside call has a timeout. Retries happen only in the worker, never in the request path or the library.”
• “Don't follow redirects to other hosts. Validate every response against our schema before using it.”
• “No new outside service, SDK, or webhook without an ADR and a line in the data inventory.”

Worked example: background reading, safely#

Students asked for a short explanation of each topic in their plan, so that “week 3: mitosis” comes with a paragraph to read first. Wikipedia offers short summaries of its articles through a public REST API: a request to /api/rest_v1/page/summary/ followed by the article's title returns the title, a plain-text extract, links to the page, and the revision (Wikimedia, REST API). It's free, needs no key, and is a good first integration because every rule in this chapter still applies to it. (Service details checked October 2026.)

1. Read the limits first. Maya and an agent read the documentation before any code (recipe prompt 2), and found:

• Wikimedia asks every client to send a descriptive User-Agent header with contact information, and may block clients that don't (Wikimedia, User-Agent policy).
• Rate limits introduced in 2026, described as experimental and subject to change, allow 200 requests a minute for unauthenticated clients with a proper User-Agent. Clients should keep to 3 concurrent requests or fewer and respect Retry-After on a 429. If no Retry-After is given, they should wait at least five seconds or back off exponentially (Wikimedia, Rate limits).
• The API etiquette page asks clients to cache data that can be cached (MediaWiki, API etiquette).
• Wikipedia's text is available under the Creative Commons Attribution-ShareAlike license, which requires credit (a link to the page is one way) and a license notice (Wikipedia, Reusing Wikipedia content). The same page says the reuser is responsible for applying the license correctly. Maya flagged the attribution wording for legal review.

2. Write the contract.

3. Write the failure matrix. Each row became a test before the adapter existed.

4. Build the adapter. The agent's first version followed the quick start: no timeout, no User-Agent, and it fetched the summary inside POST /api/plans, a request that can already spend 20 seconds waiting on the model (FR-9). The reviewed version:

# backend/coach/reading/wikipedia.py
from urllib.parse import quote
import httpx
from pydantic import BaseModel, HttpUrl, ValidationError

USER_AGENT = "LearningCoach/0.4 (https://coach.example.org/about; [email protected]) httpx"
SUMMARY_URL = "https://en.wikipedia.org/api/rest_v1/page/summary/"

class Reading(BaseModel):
    title: str
    extract: str
    page_url: HttpUrl
    revision: str

class RetryLater(Exception):
    def __init__(self, after_seconds: float):
        self.after_seconds = after_seconds

def make_client(**kwargs) -> httpx.Client:
    return httpx.Client(headers={"User-Agent": USER_AGENT}, follow_redirects=False,
                        timeout=httpx.Timeout(2.0, connect=1.0), **kwargs)

def fetch_summary(client: httpx.Client, topic: str) -> Reading | None:
    try:
        response = client.get(SUMMARY_URL + quote(topic, safe=""))
    except httpx.TransportError:
        raise RetryLater(5)
    if response.status_code == 404:
        return None
    if response.status_code == 429 or response.status_code >= 500:
        raise RetryLater(retry_after(response))
    if response.status_code != 200:
        raise ValueError(f"unexpected status {response.status_code}")
    data = response.json()
    if data.get("type") != "standard":
        return None
    try:
        reading = Reading(title=data["title"], extract=shorten(data["extract"], 1200),
                          page_url=data["content_urls"]["desktop"]["page"],
                          revision=str(data["revision"]))
    except (KeyError, TypeError, ValidationError) as error:
        raise ValueError("unexpected summary shape") from error
    if reading.page_url.host != "en.wikipedia.org":
        raise ValueError("page link on unexpected host")
    return reading

retry_after reads the header as seconds and returns 5 if it's missing, and shorten cuts at the last sentence that fits. The adapter doesn't retry and doesn't cache. The worker does both, in one place, using backoff_seconds from section 8.3 and a reading_cache table keyed by topic, so there's one retry layer and one cache to reason about. follow_redirects=False is the client library's default; stating it makes the decision visible to the next reviewer, in line with OWASP's warning against blindly following redirects from outside APIs (OWASP, API10:2023). (Library details checked October 2026.)

5. Test every row. The tests use the HTTP library's mock transport, so no test touches the network:

# backend/tests/test_wikipedia_adapter.py
def client_returning(response: httpx.Response) -> httpx.Client:
    return make_client(transport=httpx.MockTransport(lambda request: response))

def test_rate_limit_honors_retry_after():
    client = client_returning(httpx.Response(429, headers={"Retry-After": "30"}))
    with pytest.raises(RetryLater) as raised:
        fetch_summary(client, "Mitosis")
    assert raised.value.after_seconds == 30

def test_rate_limit_without_header_waits_at_least_five_seconds():
    client = client_returning(httpx.Response(429))
    with pytest.raises(RetryLater) as raised:
        fetch_summary(client, "Mitosis")
    assert raised.value.after_seconds >= 5

def test_redirect_is_not_followed():
    client = client_returning(httpx.Response(308, headers={"Location": "https://attacker.example/"}))
    with pytest.raises(ValueError):
        fetch_summary(client, "Mitosis")

Further tests cover the timeout, a 404, a non-standard summary, a missing field, an extract of 5,000 characters, and a page link on another host. A worker test runs a fake service that fails five times and checks that the breaker opens, the plan still saves, and cached summaries are served. Maya ran the tests against the agent's first version: seven of the ten failed. Passing tests show the adapter follows the matrix, not that the matrix is complete, so she added a manual release check: open a real plan and read what students will see.

6. Estimate the cost. About 15 topics per plan (an assumption) and 1,000 plans a month is 15,000 lookups, but topics repeat across students and the cache absorbs the repeats. The first week's traffic projected to about 3,000 lookups a month: no money, and far below the rate limit. Watch the cache hit rate; if it drops, requests rise.

Verification: is every edge accounted for?#

Failure clinic: paid, but still on the free plan#

Symptom. Two weeks after launch, the reconciliation job reported its first mismatch: a student whose subscription was paid at the provider but who was still on the free plan in the Learning Coach. She hadn't complained yet. She had paid four hours earlier.

Reproduce. Maya replayed the student's events from the provider's dashboard to a test endpoint, in the order the logs showed they had arrived: invoice.paid first, then customer.subscription.created. The student stayed free. Replayed in creation order, she became Pro.

Diagnose. The agent's handler had been written from the provider's example, one branch per event type. On customer.subscription.created it inserted a row linking the subscription to the student, with the status from the event, incomplete. On invoice.paid it looked up that row and set it to active. If invoice.paid came first, there was no row yet, so the handler logged “unknown subscription, ignoring” and returned 200. The provider, told the event was handled, never sent it again. Then customer.subscription.created arrived and stored incomplete.

The documentation had warned about exactly this: Stripe doesn't guarantee events arrive in the order they were generated, uses subscription creation as its example, and suggests retrieving related objects from the API when an event arrives first (Stripe, Webhooks). The tests had sent events in the order a person would expect, so every test passed.

Fix. Three changes:

1. Let events start a sync, not carry the answer. On any subscription or invoice event, the handler records the event ID (unique, so repeats stop there), queues a “sync subscription” job, and returns 200. The job fetches the subscription's current state from the provider's API and upserts it by subscription ID. Whatever order the events arrive in, the job reads the same current state, so the result is the same.
2. Never acknowledge what you didn't handle. Saying 200 to an event the handler couldn't process meant losing it. Now the only work done before answering is verifying the signature and storing the event, and the queue retries anything that fails.
3. Reconcile every page. Writing the test for the nightly job turned up a second bug: the agent's version read one page of subscriptions, the default 10, and stopped. The test account had six. A test with 25 fake subscriptions now checks that the job follows has_more to the end.

New tests deliver the events in every order, twice each, and with one dropped, and check that the student ends up Pro every time.

Lesson. Your tests reflect the outside service you imagine; the provider's documentation describes the one you'll actually get. Read its sections on delivery, ordering, and limits before you write the handler, turn each warning into a test, and keep a reconciliation job running for the cases you didn't think of.

Architect's checklist#

Before an integration ships:

• ☐ The service has an integration contract: owner, data sent, auth, timeout, limits, retries, duplicates, degradation, version.
• ☐ It appears in the data inventory, and its data handling is covered by the privacy policy.
• ☐ Keys are restricted, stored in the secrets store, rotated on a schedule, and test keys are used in development.
• ☐ Every call has a timeout, and nothing slow sits in a request a student waits on.
• ☐ Retries happen at one layer, with backoff, jitter, a limit, and respect for Retry-After.
• ☐ Outgoing writes use idempotency keys; incoming events are verified, deduplicated, and queued.
• ☐ Lists are read page by page to the end.
• ☐ Each service's failure has a planned degradation, and a test shows it.
• ☐ A reconciliation job compares your copy with the provider's and alerts on mismatches.
• ☐ The cost model is written down, with spending limits or billing alerts set.
• ☐ Agent tools are narrow, take identity from the session, require approval for high-impact actions, and are logged.
• ☐ Licensing, payment, and consumer-law questions are flagged for review by someone qualified.

Exercises#

Beginner. The Meal Planner wants grocery prices from a supermarket's API to estimate each week's shopping cost. Write the integration contract from the API's documentation (or an imagined one, with your assumptions marked), and a degradation plan: what does the planner show when prices are unavailable, and how old can a cached price be before you hide it?

Intermediate. The Trip Planner receives flight status changes from an airline data service by webhook and adjusts the itinerary. Design the webhook handler: signature check, deduplication, a queue, and a sync job that fetches the current flight status. Write a failure matrix that includes duplicate, late, and out-of-order notices, and a reconciliation job for flights whose notices never arrived. Estimate the monthly cost at 1,000 trips, stating your assumptions.

Advanced. Design four tools for the Shop Help Chat assistant: look up an order, list a customer's recent orders, start a return, and issue a refund. For each, write the schema, say how it gets the customer's identity, and classify it as automatic, proposal with customer confirmation, or staff only. Then write five test conversations that try to reach another customer's order or refund without approval, one with the instruction hidden in an order note, and show that the tool design, not the prompt, stops them.

Lab, artifact, and review#

Lab. Connect the Learning Coach to a safe outside content service, such as the encyclopedia summaries in the worked example. Include a cache, timeouts, retries at one layer, a breaker, and a test for every row of your failure matrix, all running without network access.

Deliverable. Add to your portfolio:

• an integration contract for each outside service the Learning Coach uses;
• a failure matrix for the content service, with each row linked to its test;
• a degradation table covering every service;
• a cost model per transaction and per active student, with assumptions and spending limits;
• ADR-004 on how the product takes payments;
• the webhook handler's ordering and duplicate tests, seen failing and then passing;
• a tool inventory for the study assistant, classifying each tool as read, propose, or not offered.

Reflection#

• Which outside service could take down your product today, and what would your users see?
• Where do retries happen in your system? Count the layers.
• If an AI model could call every tool your product offers, what is the worst thing it could do in one minute?

Chapter 9 gives the Learning Coach real intelligence: model calls designed as components, with prompts, context, evaluation, and the tools from this chapter, held to the same contracts as every other outside service.

References#

• Brooker, M. Timeouts, retries, and backoff with jitter. Amazon Builders' Library.
• Fowler, M. (2014). CircuitBreaker. martinfowler.com.
• Hardt, D. (Ed.). (2012). The OAuth 2.0 authorization framework (RFC 6749). IETF.
• MediaWiki. API: Etiquette. MediaWiki documentation.
• Model Context Protocol. Tools. MCP specification, version 2026-07-28.
• Nottingham, M., & Fielding, R. (2012). Additional HTTP status codes (RFC 6585). IETF.
• OWASP. API4:2023 Unrestricted resource consumption. OWASP API Security Top 10.
• OWASP. API10:2023 Unsafe consumption of APIs. OWASP API Security Top 10.
• OWASP. LLM01:2025 Prompt injection. OWASP Gen AI Security Project.
• OWASP. LLM06:2025 Excessive agency. OWASP Gen AI Security Project.
• Stripe. Best practices for managing secret API keys. Stripe documentation.
• Stripe. Idempotent requests. Stripe API reference.
• Stripe. Integration security guide. Stripe documentation.
• Stripe. Pagination. Stripe API reference.
• Stripe. Receive Stripe events in your webhook endpoint. Stripe documentation.
• Stripe. Versioning. Stripe API reference.
• Wikimedia Foundation. Wikimedia APIs: Rate limits. MediaWiki.
• Wikimedia Foundation. Wikimedia Foundation User-Agent policy. Wikimedia Foundation Governance Wiki.
• Wikimedia Foundation. Wikimedia REST API. MediaWiki.
• Wikipedia. Reusing Wikipedia content. Wikipedia.

Links checked October 2026.