High-Scale AI System

The high-scale AI system is the module's synthesis — millions of requests, designed (L358): the scale (the requests L358, the tokens L332, the concurrency L266, L358), the caching (the fast layer L171: the responses L171, the retrieval L189, the sessions L237, L358), the batching (the engine room L282: the queue L270, the workers L266, the batch L282, L358), and the cost (the economics L285: the tokens L332, the provisioned L278, the caching's hit rate L269, L358) — the L347 protocol (L347), run at the millions (L358).

The one-sentence interview answer: "The high-scale AI system is the module in one design (L358). The scale (L358): the millions of requests (L358) per day (L358) — the tokens (L332) per request (L358) — the concurrency (L266) at the peak (L358). The caching (L171): the fast layer (L171) — the response cache (L171): the repeated prompts (L358) served from the Redis (L269); the retrieval cache (L189): the repeated queries (L358) — the vector search (L189) skipped; and the sessions (L237). The batching (L282): the engine room (L282) — the queue (L270): the heavy work (L222) enqueued (L358); the workers (L266) — the model calls (L278) batched (L282) and processed (L358). The cost (L285): the tokens (L332) — the biggest line (L150); the provisioned throughput (L278) — the steady load (L358) at the committed price (L285); and the cache's (L269) hit rate (L358) — the model calls (L278) avoided (L358). The AI shape (L173): the L357 SaaS (L357) at the scale (L358) — the caching (L171), the batching (L282), and the cost (L285) — the L347 protocol (L347), run at the millions (L358)."

Think of the high-scale system as the city's rush-hour transit. The city (the system, L358) moves the millions (L358). The subway's express lines (the caching, L171): the frequent stops (the repeated requests, L358) served instantly (L171) — the passengers (the requests, L358) don't wait for the slow buses (the model, L278). The freight trains (the batching, L282): the bulk goods (the heavy work, L222) — the model calls (L278) — carried in the batches (L282), the yards (the queue, L270) absorbing (L358). And the fares (the cost, L285): the tokens (L332) — the biggest (L150); the monthly passes (the provisioned, L278) — the commuters (the steady load, L358) at the discount (L285). The city works because the express lines serve the frequent, the freight carries the bulk, and the passes price the steady (L358).

   the rush hour (the high scale, L358)
   ┌────────────────────────────────────────────────────────┐
   │ the express lines (the caching, L171) — the frequent   │
   │ served instantly (L358)                                │
   │ the freight (the batching, L282) — the bulk in the     │
   │ batches (L358)                                         │
   │ the fares (the cost, L285) — the tokens (L332), the    │
   │ passes (L278)                                          │
   └────────────────────────────────────────────────────────┘

The mental model is the rush hour: the express, the freight, and the fares (L358).

   the millions of requests (L358)
        │
        ▼
   ┌────────────────────── THE FRONT DOOR (L267) ───────────────────────┐
   │  the gateway (L267): the auth (L319), the rate limits (L318),     │
   │  the quotas (L149)                                                │
   └──────────────────────────┬───────────────────────────────────────┘
                              ▼
   ┌────────────────────── THE CACHE (L171) ────────────────────────────┐
   │  the response cache (L171): the repeated prompts (L358)           │
   │  the retrieval cache (L189): the repeated queries (L358)          │
   │  the sessions (L237) — the Redis (L269)                           │
   └──────────────────────────┬───────────────────────────────────────┘
                              ▼
   ┌────────────────────── THE ENGINE ROOM (L282) ──────────────────────┐
   │  the queue (L270) · the workers (L266) · the batch (L282)         │
   │  the model calls (L278) batched and processed (L358)              │
   └──────────────────────────┬───────────────────────────────────────┘
                              ▼
   ┌────────────────────── THE COST (L285) ─────────────────────────────┐
   │  the tokens (L332) · the provisioned (L278) · the hit rate (L269) │
   └──────────────────────────────────────────────────────────────────┘

The flow is the module in one diagram: door → cache → engine room → cost (L358).

• The scale (L358). The millions of requests (L358) per day (L358) — the tokens (L332) per request (L358) — the concurrency (L266) at the peak (L358).
• The caching (L171). The fast layer (L171): the response cache (L171), the retrieval cache (L189), the sessions (L237) — in the Redis (L269).
• The batching (L282). The engine room (L282): the queue (L270), the workers (L266), the batch (L282).
• The cost (L285). The tokens (L332), the provisioned (L278), the hit rate (L269).

• A chat at the scale (L348). The cache (L171) on the repeated prompts (L358), the streaming (L251) — the millions (L358).
• A RAG at the scale (L349). The retrieval cache (L189) — the repeated queries (L358) — the index (L183) sharded (L358).
• A SaaS at the scale (L357). The per-tenant (L320) quotas (L149) and the meters (L332) — the isolation (L320).
• A batch pipeline (L282). The queue (L270) and the workers (L266) — the model calls (L278) batched (L358).
• Anything at the scale (L358). The levers (L358) — the caching (L171), the batching (L282), the cost (L285).

The through-line: the levers are the scale's — the cache, the batch, and the cost (L358).

Say it in four moves:

1. The scale. "The millions of requests — the tokens, the concurrency (L358)."
2. The caching. "The response cache (L171), the retrieval cache (L189), the sessions (L237)."
3. The batching. "The queue (L270), the workers (L266), the batch (L282)."
4. The cost. "The tokens (L332), the provisioned (L278), the hit rate (L269)."

• The hit rate is the cost's lever (L358). The cache's (L269) hit rate (L358) — the model calls (L278) avoided (L358) — the tokens (L332) and the cost (L285) down (L358).
• The batch is the throughput's (L282). The queue (L270) and the workers (L266) — the model calls (L278) batched (L282) — the throughput (L358) up, the cost (L285) down (L358).
• The provisioned is the steady's (L278). The steady load (L358) on the provisioned (L278) — the committed price (L285) — the L285 lever (L285), scale-shaped (L358).
• The sharding is the index's (L358). The vectors (L183) sharded (L358) — the per-tenant (L320) and the hash (L358) — the retrieval (L189) at the scale (L358).
• The observability is the scale's (L346). The tokens (L332), the latency (L333), and the cost (L334) — the L346 standard (L346), scale-shaped (L358).

• The cache-less scale (L358). The model (L278) on every request (L358) — the tokens (L332) explode (L358) — the cache (L171) is the lever (L358).
• The sync heavy work (L222). The model call (L278) in the request (L222) — the user waits (L151) — the queue (L270) is the engine room (L358).
• The on-demand at the scale (L278). The pay-per-token (L150) on the steady (L358) — the provisioned (L278) is the price (L285).
• The un-sharded index (L183). The one vector index (L183) — the retrieval (L189) at the millions (L358) — the shard (L358) is the scale (L358).
• The cost-blind scale (L285). The tokens (L332) un-measured (L346) — the bill (L334) at the millions (L358).

• Cache the repeats (L171) — the responses (L171), the retrieval (L189).
• Batch the heavy (L282) — the queue (L270), the workers (L266).
• Provision the steady (L278) — the committed price (L285).
• Shard the index (L183) — the per-tenant (L320) and the hash (L358).
• Watch the cost (L346) — the tokens (L332), the hit rate (L269).

Q: Walk me through the high-scale AI system.

A: The module in one design (L358). The scale — the millions of requests, the tokens, the concurrency (L358). The caching — the responses (L171), the retrieval (L189), the sessions (L237). The batching — the queue (L270), the workers (L266), the batch (L282). And the cost — the tokens (L332), the provisioned (L278), the hit rate (L269).

Q: How do you cache at the scale?

A: Three layers (L358): the response cache (L171) — the repeated prompts (L358) served from the Redis (L269); the retrieval cache (L189) — the repeated queries (L358) — the vector search (L189) skipped; and the sessions (L237) — the state (L348) in the Redis (L269). The hit rate (L269) is the cost's (L285) lever (L358).

Q: How do you batch the model calls?

A: The engine room (L282): the heavy work (L222) enqueued (L270) — the workers (L266) process the batches (L282) — the model calls (L278) batched (L358) — the throughput (L358) up and the cost (L285) down (L358). The request path (L151) enqueues and returns fast (L358).

Q: What's the cost of the scale?

A: The three levers (L358): the tokens (L332) — the biggest line (L150) — the cache (L171) and the routing (L155) cut it (L358); the provisioned (L278) — the steady load (L358) at the committed price (L285); and the sharding (L358) — the index (L183) at the scale (L358). The L285 economics (L285), scale-shaped (L358).

• What's the scale (L358)?
• How do you cache at the scale (L171)?
• How do you batch the model calls (L282)?
• What's the cost of the scale (L285)?
• What's the hit rate (L269)?

The senior read: the right column is the scale — the levers applied (L358).

The high-scale AI system (L358) — the floor plan as folders:

  front-door/                THE DOOR (L267)
    gateway.ts               the auth (L319) + the limits (L318) + the quotas (L149)

  fast-layer/                THE CACHE (L171)
    response-cache.ts        the repeated prompts (L358) — the hit rate (L269)
    retrieval-cache.ts       the repeated queries (L358) — the search skipped (L189)
    sessions.ts              the state (L237) — the Redis (L269)

  engine-room/               THE BATCH (L282)
    queue.ts                 the heavy work (L222) — the SQS (L270)
    workers.ts               the batched model calls (L278, L282)
    batch.ts                 the throughput's (L358) lever (L358)

  model/                     THE MODEL (L278)
    routing.ts               the small for the simple (L155, L157)
    provisioned.ts           the steady at the committed price (L278, L285)

  data/                      THE INDEX (L183)
    shards.ts                the per-tenant (L320) and the hash shards (L358)

  watch/                     THE OBSERVABILITY (L346)
    tokens.ts                the L332 metering (L332) · the cost (L334)

  The door guards, the cache serves, the engine room works,
  and the watch meters the cost (L358).

What the reader must SEE — the boundaries as folders:

  front-door/   the auth, the limits, the quotas (L267)
  fast-layer/   the responses, the retrieval, the sessions (L171)
  engine-room/  the queue, the workers, the batch (L282)
  model/        the routing, the provisioned (L155, L278)
  data/         the shards (L358)
  watch/        the tokens and the cost (L332, L334)

  Every folder is a lever; every lever is a lesson (L358).

• The cache is the latency's lever (L358). The hit (L171) — the sub-millisecond (L151) — the model (L278) skipped (L358).
• The batch is the throughput's (L282). The workers (L266) — the batched (L282) calls — the concurrency (L266) up (L358).
• The provisioned is the latency's control (L278). The steady (L358) — the predictable (L333) at the committed (L285).
• The shard is the index's (L358). The per-tenant (L320) and the hash (L358) — the retrieval (L189) at the scale (L358).

• The high-scale AI system = the module's synthesis (L358): the scale, the caching, the batching, the cost.
• The scale: the millions of requests (L358) — the tokens (L332), the concurrency (L266).
• The caching: the responses (L171), the retrieval (L189), the sessions (L237).
• The batching: the queue (L270), the workers (L266), the batch (L282).
• The cost: the tokens (L332), the provisioned (L278), the hit rate (L269).

HIGH-SCALE AI SYSTEM = millions of requests, designed

THE SCALE (L358)
  the millions of requests (L358) per day (L358)
  the tokens (L332) per request (L358)
  the concurrency (L266) at the peak (L358)

THE CACHING (L171)
  the response cache (L171) — the repeated prompts (L358)
  the retrieval cache (L189) — the repeated queries (L358)
  the sessions (L237) — the Redis (L269)
  the hit rate (L269) — the cost's (L285) lever (L358)

THE BATCHING (L282)
  the queue (L270) — the heavy work (L222) enqueued (L358)
  the workers (L266) — the model calls (L278) batched (L282)
  the throughput's (L358) lever (L358)

THE COST (L285)
  the tokens (L332) — the biggest line (L150)
  the provisioned (L278) — the steady (L358) at the committed (L285)
  the shards (L358) — the index (L183) at the scale (L358)

INTERVIEW, 4 MOVES
  1 scale   "the millions of requests (L358)"
  2 caching "the responses, the retrieval, the sessions (L171)"
  3 batching "the queue, the workers, the batch (L282)"
  4 cost    "the tokens, the provisioned, the hit rate (L285)"

That was the last lesson of the AI System Design module — and the one you'll design with. L347–L357 gave you the shapes; this lesson gave you the levers: the caching, the batching, and the cost — the L347 protocol, run at the millions. When you can run the four phases (L347), assemble the shapes (L348–357), and pull the scale's levers (L358), you have claimed Milestone M28.

The next module turns the designed system into the enterprise system: Enterprise AI Solutions Architecture (L359–L380) — the requirements (L359), the stakeholders (L360), the ADRs (L361), the technology and the vendor selection (L362–366), the trade-offs (L367), the cost and the capacity (L368–369), the scalability (L370), the compliance (L371), the governance (L372–373), the DR (L374), the integrations (L375–376), the multi-cloud (L377), the platform (L378), the case study (L379), and the architect's toolkit (L380). You've designed the city; now you'll run the enterprise.