Latency & TTFT Monitoring

The latency & TTFT monitoring treats the time-to-first-token as the product metric it is (L333) — the TTFT (the time to the first token: the streaming L251 response's first chunk, L145), the total (the time to the full response: the TTFT plus the generation L333), and the monitoring (the histograms L331: the p50, the p95, the p99 L333, and the alarms L274) — the chat's (L162) feel is the first token's (L333).

The one-sentence interview answer: "The TTFT is the product metric (L333). The TTFT (L145): the time to the first token (L333) — the streaming (L251) response's first chunk (L145) — what the user sees first (L162). The total (L333): the time to the full response (L333) — the TTFT (L145) plus the generation (L333) — what the user waits for (L333). The monitoring (L333): the histograms (L331) of both (L333) — the p50, the p95, and the p99 (L333) — the p99 (L333) as the slowest user's (L162) experience (L333); and the alarms (L274) — the TTFT over the budget (L151) → the page (L274). The optimization (L151): the latency budget (L151) — the gateway (L267), the retrieval (L189), and the model (L278) within it (L333) — the trace (L330) showing where the time goes (L333). The AI shape (L173): the chat (L162) — the TTFT (L145) as the UX (L333): the streaming (L251) delivering the first token fast (L145), the monitoring (L333) watching the p99 (L333), and the alarms (L274) catching the regression (L333)."

Think of the TTFT as the restaurant's first-bite time. The diner (the user, L162) orders (the request, L328); the kitchen (the backend, L260) cooks (the generation, L333). The first bite (the TTFT, L145) — how fast the first dish (the first token, L145) arrives — is what the diner feels (L162); the full meal (the total, L333) — how long the whole dinner (the full response, L333) takes. The maître d' (the monitor, L333) times both (L333): the usual tables (the p50, L333), the slow nights (the p95, L333), and the worst (the p99, L333) — and the alarm (L274) rings when the first bite (L145) exceeds the promise (the budget, L151). The restaurant works because the first bite is fast, and the slow nights are watched (L333).

   the first bite (the TTFT, L145)
   ┌────────────────────────────────────────────────────────┐
   │ the first dish (the first token, L145) — the TTFT      │
   │ (L333)                                                 │
   │ the full meal (the total, L333) — the response (L333)  │
   │ the timings (the histograms, L331) — the p50, the p95, │
   │ the p99 (L333) · the alarm (L274)                      │
   └────────────────────────────────────────────────────────┘

The mental model is the first bite: the first dish, the full meal, and the timings (L333).

   the request (L328)
        │
        ▼
   ┌────────────────────── THE TTFT (L145) ─────────────────────────────┐
   │  the gateway (L267) → the retrieval (L189) → the model's first    │
   │  token (L278, L145)                                               │
   │  ─── 120ms ─── 80ms ─── 900ms ───                                 │
   │  the first token at 1.1s (L333)                                   │
   └──────────────────────────┬───────────────────────────────────────┘
                              ▼
   ┌────────────────────── THE TOTAL (L333) ────────────────────────────┐
   │  the generation: the remaining tokens (L333)                      │
   │  the full response at 3.4s (L333)                                 │
   └──────────────────────────┬───────────────────────────────────────┘
                              ▼
   ┌────────────────────── THE MONITORING (L333) ───────────────────────┐
   │  the histograms (L331): the TTFT's p99 (L333)                     │
   │  the alarm (L274): the TTFT over 2s → the page (L274)             │
   └──────────────────────────────────────────────────────────────────┘

The flow is the timeline: TTFT → total → monitoring (L333).

• The TTFT (L145). The time to the first token (L333): the streaming (L251) response's first chunk (L145) — what the user sees first (L162).
• The total (L333). The time to the full response (L333): the TTFT (L145) plus the generation (L333).
• The monitoring (L333). The histograms (L331) of both (L333): the p50, the p95, and the p99 (L333); the alarms (L274) on the thresholds (L333).
• The optimization (L151). The latency budget (L151): the gateway (L267), the retrieval (L189), and the model (L278) within it (L333) — the trace (L330) showing where the time goes (L333).

• A chat product (L162). The TTFT (L145) as the UX (L333) — the streaming (L251), the p99 (L333) watched.
• A RAG platform (L280). The retrieval's (L189) contribution to the TTFT (L333) — the trace (L330) showing it.
• An agent product (L279). The tool calls (L315) in the total (L333) — the trajectory's (L340) latency.
• A coding assistant (L354). The hardest latency budget (L151) — the TTFT (L145) for the autocomplete (L354).
• Anything streaming (L251). The TTFT (L145) — the product metric (L333).

The through-line: the metric is the first token's — the TTFT, the total, and the p99 (L333).

Say it in four moves:

1. The TTFT. "The first token — what the user sees first (L145)."
2. The total. "The full response — the TTFT plus the generation (L333)."
3. The monitoring. "The histograms (L331) — the p50, the p95, the p99 (L333)."
4. The optimization. "The latency budget (L151) — the trace (L330) shows where the time goes (L333)."

• The TTFT is the perceived latency (L162). The first token (L145) — what the user feels (L162) — the streaming (L251) and the cache (L171) optimize it (L333).
• The p99 is the slowest user (L333). The histogram's (L331) p99 (L333) — the worst (L333) experience (L162) — the budget's (L151) truth (L333).
• The total is the completion (L333). The generation (L333) — the model's speed (L333) and the tokens (L332) — the batch's (L282) metric (L333).
• The trace is the time's map (L330). The spans (L330) — the gateway (L267), the retrieval (L189), the model (L278) — the slow part (L333) located (L330).
• The budget is the design's (L151). The latency budget (L151) — the parts (L333) within it (L151) — the design (L333) by the budget (L151).

• The average only (L333). The mean (L333) hiding the spikes (L333) — the p99 (L333) is the truth (L333).
• The total only (L333). The full response (L333) without the TTFT (L145) — the UX (L162) unmeasured (L333).
• The model blamed (L333). The slow response (L333) assumed the model (L278) — the trace (L330) shows the retrieval (L189) or the gateway (L267) (L333).
• The budget-less design (L151). The parts (L333) un-budgeted (L151) — the TTFT (L145) over (L333).
• The alarm-less watch (L274). The p99 (L333) without the alarm (L274) — the regression (L333) undetected (L333).

• Watch the TTFT and the total (L333) — both (L333).
• Watch the p99 (L333) — the histogram (L331) of the latency (L333).
• Trace the time (L330) — the spans (L330) of the parts (L333).
• Budget the latency (L151) — the design (L333) by the budget (L151).
• Alarm the thresholds (L274) — the TTFT's budget (L151) → the page (L274).

Q: Walk me through the latency and TTFT monitoring.

A: The TTFT as the product metric (L333). The TTFT — the first token, what the user sees first (L145). The total — the full response (L333). The monitoring — the histograms (L331): the p50, the p95, the p99 (L333). And the optimization — the latency budget (L151).

Q: Why the TTFT and not the total?

A: The UX (L162): the TTFT (L145) is what the user feels (L162) — the typing (L251) starts (L333); the total (L333) is the completion (L333). The chat (L162) optimizes the TTFT (L145) — the perceived latency (L162); the batch (L282) optimizes the total (L333). The metric (L333) follows the product (L333).

Q: What's the p99 for?

A: The slowest user (L333): the histogram (L331) shows the distribution (L331) — the p50 (L333) is the usual (L333), the p99 (L333) is the worst (L333). The average (L333) hides the spikes (L333); the p99 (L333) is the experience (L162) that complains (L333).

Q: How do you find where the time goes?

A: The trace (L330): the spans (L330) — the gateway (L267), the retrieval (L189), the model (L278) — each with its duration (L333). The TTFT (L145) is the sum of the pre-generation spans (L333); the slow span (L330) is the optimization's (L151) target (L333).

• What's the TTFT (L145)?
• What's the total (L333)?
• Why the TTFT and not the total (L333)?
• What's the p99 (L333)?
• How do you find where the time goes (L330)?

The senior read: the TTFT for the interactive, the total for the completion (L333).

// The latency monitoring (L333) — the TTFT and the total (L333).
// 1 · THE INSTRUMENTS (L331) — the histograms (L333).
const ttftHist   = histogram('ai.ttft_ms', [200, 500, 1000, 2000, 4000]);   // L145
const totalHist  = histogram('ai.total_ms', [500, 1000, 2000, 5000, 10000]); // L333

// 2 · THE MEASURE (L333) — the TTFT and the total (L333).
async function streamedCall(req) {
  const started = performance.now();
  let ttftMs = null;

  const stream = await model.stream(req.prompt);   // the streaming (L251)
  for await (const chunk of stream) {
    if (ttftMs === null) {
      ttftMs = performance.now() - started;        // the TTFT (L145)
      ttftHist.observe(ttftMs);                    // the histogram (L333)
    }
    // the chunk → the client (L162)
  }

  const totalMs = performance.now() - started;     // the total (L333)
  totalHist.observe(totalMs);                      // the histogram (L333)
}

// 3 · THE ALARM (L274) — the TTFT's budget (L151).
//   ai.ttft_ms p99 > 2000ms for 5m → the page (L274)

// 4 · THE TRACE (L330) — the spans show where the time goes (L333).

What the reader must SEE — the metric, monitored:

  ttftHist + totalHist        → the histograms (L331)
  first chunk → ttftMs        → the TTFT (L145)
  stream end → totalMs        → the total (L333)
  p99 > 2000ms → the page     → the alarm (L274)
  the spans                   → the time's map (L330)

  The first token, the full response, the p99 (L333).

• The TTFT is the UX's lever (L162). The first token (L145) — the streaming (L251), the cache (L171), and the provisioned (L278) — the perceived latency (L162).
• The total is the generation's (L333). The model's speed (L333) and the tokens (L332) — the output's length (L333).
• The p99 is the budget's truth (L333). The worst (L333) — the SLO (L333) and the alarm (L274).
• The histogram is the storage's cost (L331). The buckets (L331) — the bounded (L333).

• The latency & TTFT = the product metric (L333): the TTFT, the total, the monitoring.
• The TTFT: the first token (L145) — the perceived latency (L162).
• The total: the full response (L333) — the TTFT plus the generation (L333).
• The monitoring: the histograms (L331) — the p50, the p95, the p99 (L333).
• The optimization: the latency budget (L151) — the trace (L330).

LATENCY & TTFT MONITORING = the time-to-first-token as the metric

THE TTFT (L145)
  the time to the first token (L333)
  the streaming (L251) response's first chunk (L145)
  what the user sees first (L162) — the perceived latency (L162)

THE TOTAL (L333)
  the time to the full response (L333)
  the TTFT (L145) plus the generation (L333)
  what the user waits for (L333)

THE MONITORING (L333)
  the histograms (L331) — the p50, the p95, the p99 (L333)
  the p99 (L333) — the slowest user's (L162) experience (L333)
  the alarms (L274) — the TTFT over the budget (L151) → the page (L274)

THE OPTIMIZATION (L151)
  the latency budget (L151) — the parts within it (L333)
  the trace (L330) — where the time goes (L333)

INTERVIEW, 4 MOVES
  1 TTFT   "the first token (L145)"
  2 total  "the full response (L333)"
  3 monitor "the p50, the p95, the p99 (L333)"
  4 optimize "the latency budget (L151), the trace (L330)"

You now hold the metric: the TTFT, the total, and the monitoring — with the p99 watched and the budget set. The first bite is fast — and the slow nights are paged (L333).

Next: attributing the model spend to the features, the tenants, and the users — Cost Tracking (L334).