Lesson 260·M23 Backend & Distributed Systems

Backend Architecture for AI SaaS (synthesis)

Open

Learning Objectives

By the end of this lesson you should be able to:

Assemble L233–L259 into one coherent AI SaaS backend
Draw the full flow: gateway → auth → limits → cache → services → events → stream
Explain each part's placement by its boundary — the front door, the fast layer, the engine room
Describe the failure behavior of the whole — the breaker, the bulkhead, the degradation
Defend the architecture in an interview: the parts, the boundaries, the trade-offs (L259)

One-Line Definition

Backend architecture for an AI SaaS is the module's synthesis — the front door (the gateway, L236, with the auth L237 and the limits L242), the fast layer (the cache and Redis, L243–244), the engine room (the queues and the workers, L245–249), the streaming (L251), the services and their seams (L252–254), and the resilience (the idempotency L255, the retries L256, the breakers L257, the degradation L258) — one coherent backend, assembled and defended (L260).

The one-sentence interview answer: "The AI SaaS backend is the module in one architecture (L260). The front door: the gateway (L236) — the auth (L237), the authorization (L238), the rate limits (L242), the budgets (L149). The fast layer: the cache (L244) in Redis (L243) — the sessions (L237), the responses (L171). The engine room: the queues (L245) and the workers (L249) — the model calls (L145), the workflows (L217), the agents (L200), with the DLQ (L232). The streaming: the SSE (L251) out of the chat and the generation services (L233). The services: the modular monolith (L253) with the seams (L253) — the gateway, the chat, the generation, the data (L233) — split when the scale pays (L252). The events (L248) at the seams: the job completions, the billing (L332). And the resilience: the idempotency (L255), the retries (L256), the breakers (L257), and the degradation (L258) — the whole platform designed for the partial failures (L259). Assemble it, draw it, and defend it — that's M23 (L260)."

Mental Model

Think of the AI SaaS backend as a well-run building — the front desk, the mailroom, the engine room, and the emergency plan. The front desk (the gateway, L236): every visitor checked in (the auth, L237), paced (the limits, L242), and sent to the right office (the routing, L233). The mailroom (the cache, L244): the hot items at hand, fast (L243). The engine room (the queues, L245): the slow work processed off the day's floor (L249). The offices (the services, L252): the chat, the generation, the data (L233) — the departments with their walls (L253). And the emergency plan (the resilience, L257): the breakers trip (L257), the bulkheads contain (L257), and the building serves what it can (the degradation, L258). The building works because the front desk guards, the mailroom speeds, the engine room works, and the emergency plan holds (L260).

   the building (the AI SaaS backend, L260)
   ┌────────────────────────────────────────────────────────┐
   │ the front desk (the gateway, L236) · the mailroom      │
   │ (the cache, L244) · the engine room (the queues, L245) │
   │ the offices (the services, L252) · the emergency plan  │
   │ (the resilience, L257)                                 │
   └────────────────────────────────────────────────────────┘

The mental model is the building: the guarded desk, the fast mailroom, the hard-working engine room, and the emergency plan (L260).

Visual Flow — The Whole Backend, One Diagram

   the client
        │
        ▼
   ┌────────────────────── THE FRONT DOOR (L236) ──────────────────────┐
   │  auth (L237) → authorization (L238) → rate limit (L242)          │
   │  → the budget (L149) → the trace starts (L213)                   │
   └──────────────────────────┬───────────────────────────────────────┘
                              ▼
   ┌────────────────────── THE FAST LAYER (L243) ─────────────────────┐
   │  the cache (L244): the sessions (L237), the responses (L171)     │
   │  the counters (L242) · the dedupe (L255)                         │
   └──────────────────────────┬───────────────────────────────────────┘
                              ▼
   ┌────────────────────── THE SERVICES (L252) ───────────────────────┐
   │  the chat (L233) + the generation (L145) + the data (L189)       │
   │  — the modular monolith (L253), split by the seams (L252)        │
   │  the streaming out (L251) · the events at the seams (L248)       │
   └──────────────────────────┬───────────────────────────────────────┘
                              ▼
   ┌────────────────────── THE ENGINE ROOM (L245) ────────────────────┐
   │  the queues (L245) + the workers (L249) + the DLQ (L232)         │
   │  — the model calls (L145), the workflows (L217), the agents (L200)│
   └──────────────────────────────────────────────────────────────────┘
      THE RESILIENCE (L257): the idempotency (L255), the retries (L256),
      the breakers (L257), the degradation (L258) — the whole platform

The flow is the module in one diagram: front door → fast layer → services → engine room, wrapped by the resilience (L260).

How It Works — The Assembly, Part by Part

The front door (L236). The gateway (L236): the auth (L237), the authorization (L238), the rate limits (L242), the budgets (L149), and the trace's start (L213) — the L172 baseline, operational (L236).
The fast layer (L243). The cache (L244) in Redis (L243): the sessions (L237), the responses (L171), the counters (L242), and the dedupe store (L255) — the sub-millisecond layer (L151).
The services (L252). The modular monolith (L253) with the seams (L253): the chat (L233), the generation (L145), the data (L189) — the streaming out (L251), the events at the seams (L248), split when the scale pays (L252).
The engine room (L245). The queues (L245) and the workers (L249): the model calls (L145), the workflows (L217), the agents (L200) — with the DLQ (L232) and the idempotency (L255).
The resilience (L257). The idempotency (L255), the retries (L256), the breakers (L257), and the degradation (L258) — the whole platform designed for the partial failures (L259).

Note

The assembly rule: every part is placed by a boundary (L260). The front door (L236) is where the auth (L237) and the limits (L242) live — the L172 baseline (L172). The fast layer (L243) is where the hot data lives (L244). The engine room (L245) is where the slow work lives (L222). The services (L252) are split by their seams (L253). And the resilience (L257) wraps everything — the idempotency (L255) at the mutating operations, the retries (L256) at the calls, the breakers (L257) at the providers, and the degradation (L258) at the user. An architect who can name the boundary for each part can defend the whole assembly (L260) — the L259 vocabulary in one architecture (L260).

Real Project Usage

A production AI SaaS (L357). The full assembly: the gateway (L236), the auth (L237), the limits (L242), the cache (L244), the queues (L245), the streaming (L251), and the resilience (L257) — the L260 shape (L260).
A multi-tenant platform (L357). The per-tenant limits (L242) and the isolation (L320) at the front door (L236) — the L260 backend, tenanted (L260).
A serverless AI stack (L283). The API Gateway (L267), the Lambda (L266), and the SQS (L246) — the L260 shape, AWS-shaped (L283).
A chatbot product (L162). The streaming chat (L251), the cache (L171), and the queue (L222) — the L260 core (L260).
Anything "production AI" (L260). The pattern is the shape: the front door, the fast layer, the services, the engine room, and the resilience (L260).

The through-line: the floor plan is the module's output — every AI SaaS backend is this assembly, and M23's milestone is building it and defending it (L260).

Interview Explanation

Say it in four moves:

The assembly. "The front door (L236), the fast layer (L243), the services (L252), the engine room (L245), and the resilience (L257)."
The flow. "The request enters the gateway, hits the cache, reaches the services, streams back — and the heavy work rides the queue."
The boundaries. "The auth and the limits at the door (L236); the hot data in the fast layer (L244); the slow work in the engine room (L245)."
The resilience. "The idempotency (L255), the retries (L256), the breakers (L257), and the degradation (L258) — the platform survives (L259)."

Senior-Level Insights

The architecture is the sum of its boundaries (L260). A senior review of an AI backend checks the boundaries first (L260): where's the auth (L236), where's the fast layer (L244), where's the engine room (L245), where's the breaker (L257)? Naming each is the review (L260).
The front door is the L172 baseline (L236). The gateway (L236) operationalizes the security baseline (L172): the auth (L237), the limits (L242), the budgets (L149) — the client never trusted (L172).
The engine room is the economics (L222). The queues (L245) keep the request path fast (L222) — the model calls (L145) and the workflows (L217) processed off the path (L249), the cost (L150) bounded (L256).
The resilience is the production difference (L257). The idempotency (L255), the retries (L256), the breakers (L257), and the degradation (L258) — the demo works when the provider works; this backend works when it doesn't (L258).
The observability is the whole's record (L213). The trace (L213) across the front door, the services, and the engine room (L330) — the debugging (L211), the audit (L322), and the cost (L332) read the same record (L260).

Common Mistakes

The front door missing (L236). The services called directly (L172) — no auth (L237), no limits (L242), no budgets (L149).
The slow path unlayered (L222). The model call in the request (L222) — the engine room (L245) skipped, the user waits (L151).
The hot data in the database (L151). Every read hitting Postgres (L151) — the fast layer (L244) missing.
The services as a ball of mud (L253). No seams (L253) — the split (L252) a rewrite (L253).
The resilience missing (L257). No idempotency (L255), no breakers (L257) — the provider outage (L168) is a hard crash (L258).
The observability bolted on (L213). The trace after shipping (L341) — the debugging (L211) and the audit (L322) starved (L260).

Best Practices

Draw the floor plan first (L260) — the front door, the fast layer, the services, the engine room (L260).
Place every part by its boundary (L260) — the auth at the door (L236), the hot data in the fast layer (L244), the slow work in the engine room (L245).
Start modular (L253) — the seams in the code (L253), the split when the scale pays (L252).
Wrap the resilience (L257) — the idempotency (L255), the retries (L256), the breakers (L257), the degradation (L258).
Trace the whole (L213) — the front door to the engine room (L330).
Build the milestone (L260) — assemble it, and defend it with the L259 vocabulary (L260).

Interview Questions

Q: Walk me through an AI SaaS backend.

A: Five parts (L260). The front door — the gateway (L236): the auth (L237), the authorization (L238), the rate limits (L242), the budgets (L149). The fast layer — the cache (L244) in Redis (L243): the sessions (L237), the responses (L171), the counters (L242). The services — the modular monolith (L253) with the seams (L253): the chat (L233), the generation (L145), the data (L189), with the streaming out (L251). The engine room — the queues (L245) and the workers (L249) with the DLQ (L232). And the resilience — the idempotency (L255), the retries (L256), the breakers (L257), and the degradation (L258).

Q: Why is the gateway the front door?

A: Because it's where the cross-cutting concerns live (L236). Every request passes through it (L233) — the auth (L237), the limits (L242), the budgets (L149), and the trace's start (L213) are enforced once, at the door (L236). It's the L172 baseline — the key server-side (L275), the client untrusted (L172) — operational (L236). The services trust the gateway's verdict (L236).

Q: Where does the AI work run?

A: The engine room (L245). The model calls (L145), the workflows (L217), and the agents (L200) are enqueued (L222) and processed by the workers (L249) — off the request path (L222). The queue (L245) brings the retries (L256), the DLQ (L232), and the idempotency (L255). The request enqueues and returns fast (L151); the engine room does the slow work (L260).

Q: What makes it production and not a demo?

A: The resilience and the observability (L260). The idempotency (L255) makes the retries safe (L256); the breakers (L257) stop the persistent failures; the bulkheads (L257) contain the blast; and the degradation (L258) serves the user through it all. And the trace (L213) spans the whole — the front door to the engine room (L330) — serving the debugging (L211), the audit (L322), and the cost (L332). A demo works when the provider works; this backend works when it doesn't (L258).

Follow-Up Questions

What are the five parts (L260)?
Why is the gateway the front door (L236)?
Where does the AI work run (L245)?
How does the resilience compose (L257)?
How do you defend the whole (L259)?

Comparison Table — Demo vs the Production Backend

Layer	Demo	Production (this lesson)
Front door (L236)	the service called directly	the gateway: auth, limits, budgets (L237, L242)
Fast layer (L244)	the database reads (L151)	the Redis cache (L243)
AI work (L222)	in the request	the engine room (L245, L249)
Services (L253)	one folder	the modular seams (L253)
Resilience (L257)	none	idempotency, retries, breakers, degradation (L258)
Observability (L213)	logs	the trace across the whole (L330)

The senior read: the table is the milestone — M23's claim is building the right column and defending it with the left column's failures in mind (L260).

Code Example — The Assembly in One Shape

The AI SaaS backend (L260) — the floor plan as folders:

  gateway/                 THE FRONT DOOR (L236)
    auth.ts                session · token · key (L237)
    authorize.ts           scopes + quotas (L238, L149)
    rate-limit.ts          the token bucket (L242)

  fast-layer/              THE MAILROOM (L243)
    cache.ts               cache-aside + TTL (L244)
    counters.ts            the limits and the quotas (L242)
    dedupe.ts              the idempotency store (L255)

  services/                THE OFFICES (L252)
    chat/                  the chat module (L233) — the streaming (L251)
    generation/            the generation module (L145)
    data/                  the data module (L189)
    contracts.ts           the versioned APIs (L254, L341)

  engine-room/             THE ENGINE ROOM (L245)
    queues.ts              the producers and the consumers (L245)
    workers.ts             the worker pool (L249)
    dlq.ts                 the poison catcher (L232)

  events/                  THE SEAMS (L248)
    topics.ts              the pub/sub (L247)
    schemas.ts             the event contracts (L248)

  resilience/              THE EMERGENCY PLAN (L257)
    retry.ts               the backoff + jitter (L256)
    breaker.ts             the three states (L257)
    degrade.ts             the fallback tiers (L258)

  The front door guards, the mailroom speeds, the offices serve,
  the engine room works, and the emergency plan holds.

What the reader must SEE — the boundaries as folders:

  gateway/   the auth + limits at the door (L236)
  fast-layer/ the hot data (L244)
  services/  the seams (L253) + the streaming (L251)
  engine-room/ the slow work (L245) + the DLQ (L232)
  resilience/ the retries, breakers, degradation (L257, L258)

  Every folder is a boundary; every boundary is a lesson.

Tip

The folder shape is the architecture: gateway, fast-layer, services, engine-room, resilience — each a boundary, each a lesson (L260). If the auth isn't at the door (L236) or the slow work isn't in the engine room (L245), the floor plan is missing its walls — that's M23's milestone in a directory tree (L260).

Performance Notes

The front door is the latency budget (L151). The auth (L237) and the limits (L242) in Redis (L243) — the checks sub-millisecond (L236).
The fast layer is the request's speed (L151). The cache hits (L244) — the database and the model skipped (L150).
The engine room is the async throughput (L222). The workers (L249) — the model calls (L145) processed in parallel (L222), the request path fast (L151).
The resilience is the cost's bound (L150). The retries (L256) bounded (L256), the breakers (L257) stopping the failing provider's bill (L150), the degradation (L258) serving the cheap fallbacks (L171, L157).

Debugging Scenarios

Symptom	First check (L260)	The lever
The request is slow	The fast layer (L244)	The cache (L243)
The user waits on the model	The engine room (L222)	The queue (L245)
The provider outage crashes	The resilience (L257)	The breaker (L257), the degradation (L258)
The double charges	The idempotency (L255)	The dedupe key (L255)
The failure is opaque	The trace (L213)	The distributed trace (L330)

Quick Revision Notes

The AI SaaS backend = the five parts (L260): the front door, the fast layer, the services, the engine room, the resilience.
The front door: the gateway (L236) — the auth (L237), the limits (L242).
The fast layer: the Redis cache (L243, L244).
The services: the modular monolith (L253) with the seams (L252).
The engine room: the queues (L245) and the workers (L249).
The resilience: the idempotency (L255), the retries (L256), the breakers (L257), the degradation (L258).

Cheat Sheet

BACKEND ARCHITECTURE FOR AI SAAS = the module's synthesis

THE FIVE PARTS (L260)
  front door  the gateway (L236) — the auth (L237), the
              authorization (L238), the limits (L242), the budgets (L149)
  fast layer  the Redis cache (L243, L244) — the sessions (L237),
              the responses (L171), the counters (L242), the dedupe (L255)
  services    the modular monolith (L253) — the chat (L233), the
              generation (L145), the data (L189), with the streaming (L251)
  engine room the queues (L245) and the workers (L249) — the model
              calls (L145), the workflows (L217), the agents (L200), the DLQ (L232)
  resilience  the idempotency (L255) · the retries (L256) · the
              breakers (L257) · the degradation (L258)

THE BOUNDARIES (L260)
  the auth and the limits at the door (L236)
  the hot data in the fast layer (L244)
  the slow work in the engine room (L245)
  the events at the seams (L248) · the trace across the whole (L213, L330)

THE MILESTONE (M23)
  assemble the gateway, auth, queues, caching, streaming — one backend
  and defend it with the L259 vocabulary (L260)

INTERVIEW, 4 MOVES
  1 assembly "front door, fast layer, services, engine room, resilience"
  2 flow     "the request: gateway → cache → services → stream; the work: queue"
  3 boundaries "auth at the door, hot data fast, slow work queued"
  4 resilience "idempotency, retries, breakers, degradation (L258)"

Key Takeaways

Recap

The AI SaaS backend is the module's synthesis (L260): the front door (L236), the fast layer (L243), the services (L252), the engine room (L245), and the resilience (L257)
The front door (L236) is the gateway — the auth (L237), the authorization (L238), the rate limits (L242), and the budgets (L149) — the L172 baseline, operational (L236)
The fast layer (L243) is the Redis cache (L244) — the sessions (L237), the responses (L171), the counters (L242), and the dedupe store (L255)
The services (L252) are the modular monolith (L253) with the seams (L253) — the chat (L233), the generation (L145), and the data (L189) — split when the scale pays (L252)
The engine room (L245) is the queues and the workers (L249) — the model calls (L145), the workflows (L217), and the agents (L200), with the DLQ (L232)
The resilience wraps the whole (L257): the idempotency (L255), the retries (L256), the breakers (L257), and the degradation (L258) — the platform designed for the partial failures (L259), traced across the whole (L213, L330)

Check your understanding

Answer each one out loud. Then rate yourself honestly.

0/8

What are the five parts (L260)?

Why is the gateway the front door (L236)?

What's in the fast layer (L243)?

What's the service shape (L253)?

Where does the AI work run (L245)?

How does the resilience compose (L257)?

What's the trace's span (L213)?

What is M23's milestone (L260)?

A Closing Note — The Building, Assembled

That was the last lesson of the Backend & Distributed Systems module — and the one you'll ship. L233–L259 gave you the parts; this lesson gave you the floor plan: the front door, the fast layer, the services, the engine room, and the resilience — each placed by its boundary, wrapped by the trace, and defended with the L259 vocabulary. When you can draw it, build it, and defend it — naming the auth at the door (L236), the hot data in the fast layer (L244), the slow work in the engine room (L245), and the breakers at the providers (L257) — you have claimed Milestone M23.

The next module turns the backend into infrastructure: Cloud & AWS for AI (L261–L287) — the regions and IAM (L261–262), the compute (L264, L266), the storage (L265), the gateway (L267), the databases (L268), the queues (L270), and the Bedrock (L278–281) that hosts the models — the L260 backend, deployed on AWS (L287). You've built the platform; now you'll build the cloud it stands on.