Latency Optimization

Latency optimization is engineering an LLM product's speed — splitting response time into time-to-first-token and generation throughput, then applying the levers (caching, tiering, input size, output length, streaming, batching) so the product feels fast within its cost and quality budget.

The one-sentence interview answer: "Latency has two halves: time-to-first-token — dominated by input processing, attention over the whole prompt (L136, L138) — and generation throughput, one forward pass per output token (L135). I optimise the first with caching (L171), tiering (L148), and input size (L138); the second with model tier, output length (L143), and streaming (L145). Then I budget: 'feels fast' is a target per feature, and I trade latency against cost (L150) deliberately."

Think of an AI response as a train journey with two legs: the wait at the platform (TTFT) and the speed of the train (tokens/sec). Streaming (L145) is the trick that makes the platform wait feel like travel — but the engineering is making the platform wait short and the train fast.

   total response time
   ├───────────────────────────────────────────────────┤
   │ TTFT                │  generation                 │
   │ (wait at platform)  │  (the train, token by token)│
   │                     │                             │
   │ input processing    │  one forward pass           │
   │ = attention over    │  per output token (L135)    │
   │   the whole prompt  │  → output length drives it  │
   │   (L136, L138)      │                             │
   │                     │                             │
   │ levers:             │  levers:                    │
   │  cache (L171)       │   tier (L148)               │
   │  tier (L148)        │   output length (L143)      │
   │  input size (L138)  │   stream (L145)             │
   │  network            │   batch                     │
   └─────────────────────┴─────────────────────────────┘

The mental model: two numbers, two lever sets. TTFT = "how long until the train starts moving"; generation = "how fast it moves". Streaming makes the wait feel shorter; caching and tiering make it actually shorter.

   user clicks "send"
        │
        ▼
   ┌───────────────────────────────────────────────┐
   │ network to your server        ~20-100ms       │
   └──────────────────┬────────────────────────────┘
                      ▼
   ┌───────────────────────────────────────────────┐
   │ YOUR stack (gateway, auth, retrieval)         │
   │  retrieval (L189)  ~50-200ms                  │
   └──────────────────┬────────────────────────────┘
                      ▼
   ┌───────────────────────────────────────────────┐
   │ PROVIDER                                          │
   │  prompt cache hit?   ~30-80ms  (L171)        │
   │  else full input pass ~0.5-5s (L138, O(n²))  │
   │  ──── TTFT ────                              │
   │  generation: N tokens × per-token forward    │
   │             pass   ~20-80ms/token (L135)     │
   └──────────────────┬────────────────────────────┘
                      ▼
   ┌───────────────────────────────────────────────┐
   │ stream back (L145) — user sees token 1 fast,  │
   │ then the rest pour in                         │
   └───────────────────────────────────────────────┘

The picture is the toolbox: every hop is a lever. The network, the retrieval (L189), the cache (L171), the tier (L148), the input size (L138), the output length (L143), and the stream (L145) — each one adds or removes milliseconds, and the architect's job is to know which ones dominate for this feature.

• TTFT is dominated by input processing. The whole prompt must be attended over before the first output token — attention is O(n²) in input length (L136, L138). So TTFT scales with input size, and the levers are: shrink the input (L138), cache the prefix (L171, repeated input ≈ free), and pick a faster tier (L148).
• Generation throughput is one forward pass per token (L135). A 300-token answer is 300 sequential forward passes. So the generation leg scales with output length, and the levers are: shorter outputs (L143), a faster tier (L148), and streaming (L145) to make the wait visible instead of silent.
• Caching (L171) is the asymmetric win. A cached prefix skips the input processing entirely — TTFT drops to ~10% of the uncached number. For chat (same system prompt + summarised history, L166) and RAG (stable doc prefix), caching is the biggest single lever.
• Tiering (L148) is the second win. The small model is faster on both legs — lower TTFT and higher tokens/sec. The flagship's latency is a feature you pay for, routed to the hard tail only (L157).
• Batching helps throughput, hurts TTFT. Batching many requests together is efficient for the provider but queues your request — good for background jobs (L222), wrong for interactive chat.

• Chat products — the TTFT fight. The perceived quality of ChatGPT/Claude/Gemini is mostly TTFT + streaming (L145). Caching the system prompt (L171) and streaming immediately are the two non-negotiables.
• Voice and assistant UIs. TTFT is the whole experience — a 2-second first token breaks a voice loop. Voice routes to the fastest tier (L148) and caches aggressively.
• RAG products. TTFT includes retrieval (L189): a slow vector search (L182) delays the first token. Caching the retrieved prefix on repeated queries (L171) is the RAG-specific lever.
• Agents (L200). Every tool round trip (L144) adds a TTFT and a generation leg. Agent latency = sum of its steps; the levers are fewer steps (better planning, L202) and shorter per-step outputs.
• Batch / background (L222). When nothing is watching, latency is irrelevant — optimize cost (L150) instead, and batch for throughput. Latency levers are only for the interactive path.

The through-line: latency levers apply where a human waits; cost levers apply where a machine works. The two halves of the same optimization.

Say it in four moves:

1. The split. "Latency has two halves: TTFT — dominated by input processing, attention over the whole prompt (L136, L138) — and generation, one forward pass per output token (L135)."
2. The levers. "I attack TTFT with caching (L171), tiering (L148), and input size (L138); generation with tier, output length (L143), and streaming (L145)."
3. The asymmetric wins. "Caching is the biggest: a cached prefix skips input processing entirely — TTFT drops to ~10%. Tiering is second: the small model is faster on both legs."
4. The budget. "Then I set a target per feature — 'feels fast' is a number — and trade latency against cost (L150) and quality deliberately. Latency optimization is never free."

• TTFT is the number users actually feel (L145). Total latency barely matters when the first token lands in 300ms and the rest streams in. Optimizing TTFT is optimizing the product's perceived intelligence.
• Caching is the senior move (L171). It's the rare lever that cuts cost and latency with no quality loss — the only cost is cache stability (L142). A senior answer leads with it.
• Input size is a retrieval decision (L138, L189). TTFT scales with input; so does cost (L150). Tight retrieval is a latency lever, a cost lever, and a quality lever at once — the best kind.
• Output length is the hidden latency driver (L135, L143). Most people optimize TTFT and ignore that a 500-token answer is 500 sequential forward passes. Structured, short outputs (L143) attack the slowest leg.
• Latency budget per feature, not per system. A chat answer has a 2-second budget; a background summarisation has none. The architect sets the budget by who's watching, then applies the levers.

• Optimizing total time instead of TTFT. Cutting generation time by 50% while the user waits 4 seconds for the first token is optimizing the wrong number (L145).
• Ignoring input size (L138). A 100K-token RAG context makes TTFT seconds — regardless of how fast the model is. The input is the TTFT lever.
• No caching (L171). Every request paying full input cost when the prefix is identical — the biggest missed win in most systems.
• Flagship for everything (L148). The flagship is slower and more expensive; tiering cuts both (L157).
• Unbounded outputs. Letting the model write 800 tokens when 150 would do — the slowest, most expensive leg, ignored (L135, L150).
• Streaming without cache. Streaming hides the wait but doesn't shorten it; the two compose, they don't replace each other (L145).

• Measure TTFT and tokens/sec separately — they're different problems with different levers (L333).
• Cache the prefix (L171) — freeze the system prompt, keep the doc prefix stable.
• Tier the interactive path (L157) — small model first, flagship for the hard tail.
• Keep inputs tight (L138, L189) — retrieval quality is a latency lever.
• Design short structured outputs (L143) — the generation leg is per-token.
• Stream everything a human reads (L145) — the felt-quality layer on top of the real ones.

Q: What determines an AI product's latency?

A: Two halves. Time-to-first-token, dominated by input processing — attention over the whole prompt (L136, L138), so it scales with input size. Then generation, one forward pass per output token (L135), so it scales with output length. Different halves, different levers.

Q: How do you reduce TTFT?

A: Cache the prefix (L171) — a cache hit skips input processing, dropping TTFT to ~10%. Use a faster tier (L148). And shrink the input (L138): tight retrieval (L189), summarised history (L166), a lean system prompt (L142). Then stream (L145) so the short TTFT is visible.

Q: Why is output length a latency driver?

A: Because generation is sequential — one forward pass per token (L135). A 500-token answer is 500 passes. Short, structured outputs (L143) and the right tier cut that leg directly; streaming (L145) makes it feel faster regardless.

Q: How do you trade latency against cost?

A: The same levers move both (L150). Caching cuts both with no quality loss. Tiering cuts both but trades quality — so I route the interactive path to the small model and the hard tail to the flagship (L157). Shorter outputs cut both but trade completeness. I name the trade per lever and set a latency budget per feature.

• How does prompt caching (L171) work, and what makes a prefix cacheable?
• Why does a RAG request have a latency problem that plain chat doesn't (L174)?
• When is batching the right latency trade (L222)?
• How does agent latency differ from chat latency (L200)?
• What's the relationship between TTFT and perceived quality (L145)?

The senior read: the table is a budget tool — pick the levers by who's watching (human → stream + cache + tier; machine → batch + cost), and name the trade of each.

// Latency optimization in code: cache the prefix, tier the path, stream the answer.
// 1 · The system prompt is frozen — it's the cache key (L142, L171).
const SYSTEM = 'You are a concise analyst.';

// 2 · The tier is config, not code (L148, L157) — interactive path = small model.
const TIER = { model: 'gpt-4o-mini', temperature: 0.3 };

// 3 · Stream (L145) so TTFT is visible, and the request is cache-friendly.
export async function POST(req) {
  const { question } = await req.json();
  const stream = await openai.chat.completions.create({
    ...TIER,
    messages: [
      { role: 'system', content: SYSTEM },          // ← byte-stable → cache hit
      { role: 'user', content: question },
    ],
    max_tokens: 200,                                // ← short output, short generation leg
    stream: true,
  });
  return toSSE(stream);                            // ← TTFT visible, tokens pour in
}

// 4 · The real latency levers, named:
//    cache (L171)   → the frozen SYSTEM prefix hits the cache, TTFT ≈ 10%
//    tier (L148)    → small model, fast both legs
//    output (L143)  → 200 max tokens, not 800
//    stream (L145)  → the user sees token 1 fast, then the rest

What the reader must SEE — the levers are the request:

  frozen system prompt  → cache hit (L171)
  small model tier      → fast both legs (L148)
  max_tokens 200        → short generation leg (L135, L143)
  stream: true          → TTFT visible (L145)

  Four levers, one request — no latency magic.

• TTFT and generation are separate budgets. A 300ms TTFT + 20ms/token × 300 tokens = ~6.3s total — but streamed, it feels like 300ms. Measure both (L333), optimize both.
• Caching is the biggest single lever (L171). ~10× on TTFT for repeated prefixes — bigger than any model swap.
• Input size is O(n²) in attention (L136). Halving a 100K input doesn't halve TTFT linearly — it cuts the dominant term. Retrieval tightness (L189) is a first-class latency lever.
• Tiering is the both-legs lever (L148). Small models are faster and cheaper; the flagship's latency is a routed exception (L157), not a default.
• Batching is a background lever (L222). It improves provider throughput at the cost of per-request TTFT — wrong for chat, right for jobs.

• Latency = TTFT + generation: input processing (L136, L138) then one forward pass per output token (L135).
• TTFT levers: cache (L171), tier (L148), input size (L138).
• Generation levers: tier (L148), output length (L143), streaming (L145).
• Caching is the asymmetric win — ~10× TTFT, no quality loss.
• Latency and cost are the same tokens (L150) — the levers overlap.
• Latency budget per feature, by who's watching — human → stream+cache+tier; machine → batch+cost (L222).

LATENCY = TTFT + generation

  TTFT         input processing (attention, O(n²) in input — L136/L138)
  generation   one forward pass per output token (L135)

LEVERS
  cache (L171)    TTFT ~10×   trades stability
  tier (L148)     both legs   trades quality (route, L157)
  input size      TTFT        trades recall (L189)
  output length   generation  trades completeness (L143)
  stream (L145)   felt        trades complexity
  batch (L222)    throughput  trades per-request TTFT

THE BASELINE REQUEST
  frozen system prompt  → cache hit (L171)
  small tier            → fast both legs (L148)
  tight max_tokens      → short generation leg (L143)
  stream: true          → TTFT visible (L145)

RULES
  measure TTFT and tokens/sec separately (L333)
  budget per feature, by who's watching
  latency & cost are the same tokens (L150)

INTERVIEW, 4 MOVES
  1 split   "TTFT + generation, different levers"
  2 TTFT    "cache, tier, input size"
  3 gen     "tier, output length, stream"
  4 budget  "target per feature, trade deliberately"

You now hold the full latency toolbox: TTFT and generation, two halves with two lever sets, and the baseline request — frozen prefix, small tier, tight output, stream. Combined with cost (L150), you can now answer the two questions every AI architecture faces — "how much?" and "how fast?" — with the same ledger, the same levers, and the same honesty about trade-offs.

Next: the providers themselves — the OpenAI API (L152), Anthropic (L153), Gemini (L154), abstraction (L155), and the comparison that ties the module together (L156–L157).