Fault Tolerance & Graceful Degradation

Fault tolerance and graceful degradation are the fallback story (L258) — when a part fails (L257), the system keeps serving through the degradation tiers (L258): the full response when everything works, the degraded response when a fallback fires (the cache, L171, the smaller model, L157), and the honest message when nothing can (L258) — the user served through the failure, never a hard crash (L258).

The one-sentence interview answer: "Graceful degradation is the fallback story (L258). When a part fails (L257), the app degrades in tiers instead of failing hard (L258). The first tier — the full response: everything works, the full pipeline (L173). The second tier — the degraded response: a fallback fires — the cache (L171) serves the repeat, the smaller or the faster model (L157) answers, the retrieval-only answer (L174) serves — the response is worse but present (L258). The third tier — the honest message: nothing can serve — the app tells the user, honestly, with the retry (L256) or the status (L258). The detection: the breaker (L257) and the errors (L168) signal the degradation (L258). The fault tolerance is the system's ability to keep serving through the failures (L258) — the user never sees a hard crash (L258)."

Think of graceful degradation as the restaurant's response when the chef is out. The full menu (the full response, L258): the chef is in, everything's cooked fresh (the full pipeline, L173). The chef is out (the provider fails, L168) — the restaurant degrades in tiers (L258): the first tier, the pre-made dishes (the cache, L171) — the repeat orders served fast (L258); the second tier, the sous-chef's simplified menu (the smaller model, L157) — the answer's there, just simpler (L258); the third tier, the honest sign (L258): "the kitchen's closed, come back in an hour" (the honest message with the retry, L256). The restaurant never slams the door (the hard crash, L258) — it serves what it can, tier by tier (L258).

   the restaurant's tiers (L258)
   ┌────────────────────────────────────────────────────────┐
   │ tier 1: the full menu — everything works (L258)        │
   │ tier 2: the pre-made + the sous-chef — the fallbacks   │
   │         (the cache, L171 · the smaller model, L157)    │
   │ tier 3: the honest sign — "come back" (L258)           │
   └────────────────────────────────────────────────────────┘

The mental model is the restaurant's tiers: the full, the degraded, and the honest — never the slammed door (L258).

   a request arrives, the provider fails (L168)
        │
        ▼
   ┌──────────────────────────────────────────────────────────┐
   │ 1 · THE SIGNAL (L257)                                    │
   │     the breaker (L257) or the error (L168) — the         │
   │     degradation triggered (L258)                         │
   └──────────────────┬───────────────────────────────────────┘
                      ▼
   ┌──────────────────────────────────────────────────────────┐
   │ 2 · TIER 1 — THE CACHE (L171)                            │
   │     the repeat? → the cached response (L171, L258)       │
   └──────────────────┬───────────────────────────────────────┘
                      ▼ no repeat
   ┌──────────────────────────────────────────────────────────┐
   │ 3 · TIER 2 — THE SMALLER MODEL (L157)                    │
   │     the fast model (L157) or the retrieval-only (L174)   │
   │     — the answer present, simpler (L258)                 │
   └──────────────────┬───────────────────────────────────────┘
                      ▼
   ┌──────────────────────────────────────────────────────────┐
   │ 4 · TIER 3 — THE HONEST MESSAGE (L258)                   │
   │     "the service is temporarily unavailable" (L258)      │
   │     with the retry (L256) and the status (L258)          │
   └──────────────────────────────────────────────────────────┘

The flow is the tiers: signal → cache → smaller model → honest message (L258).

• The tiers (L258). The full response when everything works (L258); the degraded response when a fallback fires (L258); the honest message when nothing can (L258). The tiers are the degradation's design (L258).
• Tier 1 — the cache (L171). The repeat requests served from the cache (L171) — the exact-repeat responses (L171) available during the outage (L258).
• Tier 2 — the smaller model (L157). The degraded generation (L258): the smaller or the faster model (L157) — the answer present, simpler (L258); or the retrieval-only answer (L174) — the RAG without the synthesis model (L258).
• Tier 3 — the honest message (L258). The transparent failure (L258): "the service is temporarily unavailable" (L258) — with the retry-after (L256) and the status page (L258). The honest message is a designed UX, not a crash (L258).
• The detection (L257, L168). The breaker (L257) and the errors (L168) signal the degradation (L258) — the fail-fast (L257) into the tier (L258).

• The provider outage (L168). The model provider fails (L168) — the breaker (L257) trips, and the app serves: the cache (L171) for the repeats (L258), the smaller model (L157) for the new (L258), the honest message (L258) for the rest (L258).
• The service overload (L252). The generation service (L145) overloaded (L252) — the bulkhead (L257) contains it, and the chat degrades to the retrieval-only (L174) answers (L258).
• The third-party failure (L227). The external API (L227) down (L168) — the integration service (L252) degrades to the cached data (L244, L258).
• The launch day (L357). The high load (L252) — the degradation tiers (L258) keep the app serving (L258).
• Anything failing (L260). The degradation (L258) is the L260 platform's fallback story (L260) — the tiers, designed (L258).

The through-line: the restaurant's tiers — the full, the degraded, and the honest — the user served through the failures (L258).

Say it in four moves:

1. The tiers. "Full → degraded → honest (L258)."
2. The fallbacks. "The cache (L171), the smaller model (L157), the retrieval-only (L174)."
3. The honest message. "The designed third tier (L258) — the truth, the retry (L256)."
4. The detection. "The breaker (L257) and the errors (L168) signal the degradation (L258)."

• The tiers are the design (L258). The senior answer designs the degradation in advance (L258) — the demo's crash (L258) is the design's absence (L258).
• The fallbacks are the L171 and L157 levers (L171, L157). The cache (L171) and the smaller model (L157) — the existing levers, deployed as the degradation (L258).
• The honest message is the trust (L258). The transparent failure (L258) — the user's trust survives the outage (L258).
• The detection is the breaker's signal (L257). The breaker (L257) and the errors (L168) — the degradation triggered deliberately (L258).
• The observability is the degradation's record (L213). The degraded responses logged (L213), the outage's impact measured (L332) — the degradation observed (L258).

• The hard crash (L258). The raw error to the user (L258) — the degradation (L258) missing (L258).
• The cache without the freshness (L171). The stale cached responses served during the outage (L171) — the L140 contract ignored (L258).
• The smaller model without the quality (L157). The degraded answer with no quality check (L195) — the wrong answers shipped (L258).
• The dishonest message (L258). "Everything's fine" during the outage (L258) — the user misled (L258).
• No detection (L257). The degradation never triggered (L257) — the failures raw (L258).
• The degradation unobserved (L213). The tiers' usage unmeasured (L332) — the outage's impact invisible (L258).

• Design the tiers in advance (L258) — the full, the degraded, the honest (L258).
• Deploy the fallbacks (L258) — the cache (L171), the smaller model (L157), the retrieval-only (L174).
• Design the honest message (L258) — the truth, the retry (L256), the status (L258).
• Trigger from the signals (L257) — the breaker (L257) and the errors (L168).
• Check the degraded quality (L195) — the smaller model's answers verified (L258).
• Observe the degradation (L213) — the tiers' usage, the outage's impact (L332).

Q: What is graceful degradation?

A: The fallback story (L258). When a part fails (L257), the app degrades in tiers instead of failing hard (L258): the full response when everything works (L258); the degraded response when a fallback fires — the cache (L171) or the smaller model (L157) (L258); and the honest message when nothing can (L258). The user never sees a hard crash (L258).

Q: What are the fallback tiers?

A: Three (L258). The cache (L171) — the repeat requests served during the outage (L258). The smaller or the faster model (L157) — the answer present, simpler (L258); or the retrieval-only (L174) — the RAG without the synthesis model (L258). And the honest message (L258) — "temporarily unavailable" with the retry (L256) and the status (L258). Each tier worse but present (L258).

Q: Why is the honest message a tier, not a failure?

A: Because it's designed (L258). The demo's crash — the raw error, the dead page — is not degradation (L258). The senior third tier is designed: the transparent message (L258) — what happened, when to retry (L256), where the status lives (L258) — with the right status code (L234) and the log (L213). The user's trust survives the outage (L258) because the app told the truth gracefully (L258).

Q: How does the degradation trigger?

A: The signals (L257): the breaker (L257) trips on the persistent failures (L257), and the errors (L168) signal the transient ones (L258). The fail-fast (L257) into the tier (L258): the breaker's open state routes to the cache (L171), the smaller model (L157), or the honest message (L258) — the degradation triggered deliberately, not accidentally (L258).

• What are the tiers (L258)?
• What are the fallbacks (L258)?
• Why is the honest message designed (L258)?
• How does the degradation trigger (L257)?
• What do you observe (L213)?

The senior read: the tiers are the ladder — down gracefully, never off the cliff (L258).

// Graceful degradation: the tiers (L258).
async function answerWithDegradation(req) {
  try {
    // TIER 0 — the FULL response (L258): everything works.
    return await fullPipeline(req);                       // L173
  } catch (e) {
    log({ event: 'degraded', reason: e.message });        // L213
    if (breaker.isOpen()) {                              // the signal (L257)
      // TIER 1 — the CACHE (L171, L258).
      const cached = await cache.get(cacheKey(req));
      if (cached) return cached;                         // the repeat served (L171)

      // TIER 2 — the SMALLER MODEL (L157, L258).
      const simple = await streamText({ model: 'fast-small', … });  // L157
      if (simple) return simple;                         // the present answer (L258)
    }

    // TIER 3 — the HONEST MESSAGE (L258).
    return error(503, {
      code: 'service_unavailable',
      message: 'The AI service is temporarily unavailable. Please retry shortly.',
      retryAfter: 60,                                    // the retry (L256)
    });
  }
}

What the reader must SEE — the restaurant's tiers:

  fullPipeline()        → tier 0, everything works (L173)
  breaker.isOpen()      → the signal (L257)
  cache.get()           → tier 1, the repeat (L171)
  model: 'fast-small'   → tier 2, the present answer (L157)
  error(503, …)         → tier 3, the honest message (L258)

  The full, the degraded, and the honest — never the crash.

• The fail-fast is the latency win (L151). The breaker's fast failure (L257) into the tier (L258) — the hang avoided (L257).
• The cache is the outage's cost lever (L171). The repeats served without the model (L150) — the degraded tier's economics (L258).
• The smaller model is the degraded cost (L150). The fast model (L157) — the cheaper generation (L150) during the outage (L258).
• The observation is the impact's record (L213). The degraded responses logged (L213), the outage's impact measured (L332) — the degradation observed (L258).

• Graceful degradation = the fallback story (L258) — the tiers (L258).
• The tiers: the full, the degraded, the honest (L258).
• The fallbacks: the cache (L171), the smaller model (L157), the retrieval-only (L174).
• The honest message: the designed third tier (L258) — the truth, the retry (L256).
• The detection: the breaker (L257) and the errors (L168).
• The observation: the tiers' usage, the impact (L213, L332).

FAULT TOLERANCE & GRACEFUL DEGRADATION = the fallback story

THE TIERS (L258)
  tier 0  the FULL response — everything works (L173)
  tier 1  the DEGRADED — the cache (L171): the repeats served (L258)
  tier 2  the DEGRADED — the smaller model (L157), the
          retrieval-only (L174): the present, simpler answer (L258)
  tier 3  the HONEST message (L258): the truth, the retry (L256),
          the status (L258)

THE DETECTION (L257, L168)
  the breaker (L257) trips on the persistent (L257)
  the errors (L168) signal the transient (L168)
  the fail-fast (L257) into the tier (L258)

THE RULES
  the honest message is a tier, not a failure (L258)
  the crash is the design's absence (L258)
  the degraded answers are quality-checked (L195)
  the degradation is observed (L213, L332)

INTERVIEW, 4 MOVES
  1 tiers   "full → degraded → honest (L258)"
  2 fallbacks "the cache (L171), the smaller model (L157)"
  3 honest  "the designed third tier (L258)"
  4 trigger "the breaker (L257) and the errors (L168)"

You now hold the fallback story: the full, the degraded, and the honest tiers — the cache and the smaller model as the fallbacks, the transparent message as the last resort, and the breaker's signal as the trigger. The platform now serves through the failures — gracefully, tier by tier (L258).

Next: the vocabulary — distributed systems concepts (L259), CAP, consistency, and the design-round words.