Metrics

The metrics are the numbers the AI platform runs on (L331) — the types (the counters: the monotonic counts — the requests, the errors; the gauges: the current values — the queue depth L270, the concurrency L266; the histograms: the distributions — the latency L333, the tokens L332, L331), the AI metrics (the tokens L332, the latency L333, the cost L334, and the quality L341), and the alarms (the thresholds L274: the error rate over 1% → the page L274) — the platform's (L260) numbers (L331).

The one-sentence interview answer: "The metrics are the numbers the platform runs on (L331). The types (L331): the counters (L331) — the monotonic counts (L331): the requests (L331), the errors (L331), the tokens (L332) — the totals (L331); the gauges (L331) — the current values (L331): the queue depth (L270), the concurrency (L266), the quota's usage (L149); and the histograms (L331) — the distributions (L331): the latency (L333) — the p50 and the p99 (L333) — and the tokens per call (L332). The AI metrics (L331): the tokens (L332) — the usage and the cost (L334); the latency (L333) — the TTFT (L145) and the total (L333); and the quality (L341) — the evals' scores (L341). The alarms (L274): the thresholds (L274) — the error rate over 1% for 5 minutes (L274) → the page (L274); the cost over the budget (L334) → the alert (L334). The AI shape (L173): the platform (L260) — the metrics (L331): the counters (L331), the gauges (L331), and the histograms (L331) — the numbers (L331) the alarms (L274) and the scaling (L271) run on (L331)."

Think of the metrics as the factory's instrument panel. The panel (the metrics, L331) has the instruments (L331): the odometers (the counters, L331) — the miles (the requests, L331) and the breakdowns (the errors, L331), always increasing (L331); the fuel gauges (the gauges, L331) — the current level (the queue depth, L270) and the engines running (the concurrency, L266); and the speed charts (the histograms, L331) — the distribution of the speeds (the latency, L333) — the usual (the p50, L333) and the rare spikes (the p99, L333). The control room (the dashboards, L274) shows the panel (L331), and the alarms (L274) ring when the needle crosses the red line (L274). The factory works because the panel is complete, the alarms are set, and the engineers read the trends (L331).

   the panel (the metrics, L331)
   ┌────────────────────────────────────────────────────────┐
   │ the odometers (the counters, L331) — the totals (L331) │
   │ the fuel gauges (the gauges, L331) — the current       │
   │ values (L331)                                          │
   │ the speed charts (the histograms, L331) — the          │
   │ distributions (L331) · the alarms (L274)               │
   └────────────────────────────────────────────────────────┘

The mental model is the panel: the odometers, the fuel gauges, and the speed charts (L331).

   the call (L328)
        │
        ▼
   ┌────────────────────── THE INSTRUMENTS (L331) ──────────────────────┐
   │  the counter: requests_total++ (L331)                             │
   │  the counter: errors_total++ on the failure (L331)                │
   │  the gauge: queue_depth = sqs.usage (L270)                        │
   │  the histogram: latency_ms.observe(duration) (L333)               │
   └──────────────────────────┬───────────────────────────────────────┘
                              ▼
   ┌────────────────────── THE ALARMS (L274) ───────────────────────────┐
   │  the error rate over 1% for 5m (L274) → the page (L274)           │
   │  the cost over the budget (L334) → the alert (L334)               │
   └──────────────────────────┬───────────────────────────────────────┘
                              ▼
   ┌────────────────────── THE SCALE (L271) ────────────────────────────┐
   │  the queue depth (L270) → the workers (L271)                      │
   └──────────────────────────────────────────────────────────────────┘

The flow is the metric: observe → alarm → scale (L331).

• The counters (L331). The monotonic counts (L331): the requests (L331), the errors (L331), the tokens (L332) — the totals (L331) and the rates (L331).
• The gauges (L331). The current values (L331): the queue depth (L270), the concurrency (L266), the quota's usage (L149).
• The histograms (L331). The distributions (L331): the latency (L333) — the p50 and the p99 (L333) — and the tokens per call (L332).
• The AI metrics (L331). The tokens (L332), the latency (L333), the cost (L334), and the quality (L341) — the AI's numbers (L331).

• A production AI SaaS (L357). The metrics (L331): the requests (L331), the tokens (L332), the latency (L333), the cost (L334) — per tenant (L320).
• A chat product (L162). The TTFT's histogram (L145, L331) — the p95 (L333) watched (L331).
• A RAG platform (L280). The retrieval's latency (L333) and the tokens (L332) — the cost (L334) per query (L331).
• An agent product (L279). The tool success (L339) and the trajectory's length (L340) — the agent's metrics (L331).
• Anything AI (L328). The numbers' layer (L331) — the counters, the gauges, the histograms (L331).

The through-line: the numbers are the platform's — the counters, the gauges, and the histograms (L331).

Say it in four moves:

1. The types. "The counters, the gauges, the histograms (L331)."
2. The AI metrics. "The tokens (L332), the latency (L333), the cost (L334)."
3. The alarms. "The thresholds (L274) — the error rate, the cost (L334)."
4. The scale. "The queue depth (L270) → the workers (L271)."

• The type is the question (L331). The counter (L331) for the totals, the gauge (L331) for the current, the histogram (L331) for the distribution (L331).
• The p99 is the user's experience (L333). The histogram's (L331) p99 (L333) — the slowest 1% (L333) — the UX's (L162) truth (L331).
• The tokens are the cost's counter (L332). The tokens (L332) as the counter (L331) — the cost (L334) derived (L334) — the bill (L334) attributed (L331).
• The queue is the scale's gauge (L270). The queue depth (L270) — the workers (L271) scaled (L271) — the L271 autoscaling (L271), metric-fed (L331).
• The alarm is the threshold's (L274). The error rate (L274) and the cost (L334) — the thresholds (L274) — the on-call (L274) paged (L331).

• The log-only watch (L329). The records (L329) without the numbers (L331) — the trends (L331) invisible (L331).
• The type mismatch (L331). The gauge (L331) for the total (L331) — the counter (L331) is the totals' (L331).
• The p50 only (L333). The average (L333) hiding the spikes (L333) — the p99 (L333) is the truth (L331).
• The tokens unmeasured (L332). The usage (L332) absent (L331) — the cost (L334) un-attributable (L331).
• The alarm-less metrics (L274). The numbers (L331) without the thresholds (L274) — the incidents (L304) undetected (L331).

• Pick the type by the question (L331) — the counter, the gauge, the histogram (L331).
• Watch the p99 (L333) — the histogram (L331) of the latency (L333).
• Count the tokens (L332) — the cost's (L334) counter (L331).
• Alarm the thresholds (L274) — the error rate, the cost (L334).
• Feed the scaling (L271) — the queue depth (L270) to the workers (L271).

Q: Walk me through the metrics.

A: The numbers the platform runs on (L331). The types — the counters (the totals), the gauges (the current), the histograms (the distributions) (L331). The AI metrics — the tokens (L332), the latency (L333), the cost (L334). And the alarms — the thresholds (L274).

Q: What's the difference between the counter and the gauge?

A: The question (L331): the counter (L331) is the monotonic total — the requests, the errors, the tokens (L331) — and its rate (L331); the gauge (L331) is the current value — the queue depth (L270), the concurrency (L266) — it goes up and down (L331). The counter (L331) for "how many", the gauge (L331) for "how much right now" (L331).

Q: Why the histogram?

A: The distribution (L331): the latency (L333) is not one number (L333) — the p50 (L333), the p95 (L333), the p99 (L333) — the histogram (L331) shows the shape (L331). The average (L333) hides the spikes (L333); the p99 (L333) is the user's (L162) experience (L331).

Q: How do the metrics drive the platform?

A: Two ways (L331): the alarms (L274) — the error rate over 1% (L274) and the cost over the budget (L334) page the on-call (L274); and the scaling (L271) — the queue depth (L270) scales the workers (L271). The numbers (L331) are the platform's (L260) controls (L331).

• What are the types (L331)?
• What's the counter vs the gauge (L331)?
• Why the histogram (L331)?
• What are the AI metrics (L331)?
• How do the metrics drive the platform (L331)?

The senior read: the type is the question — the totals, the current, the distribution (L331).

// The metrics (L331) — the instruments of the call (L331).
// 1 · THE COUNTERS (L331) — the totals (L331).
const requestsTotal = counter('ai.requests_total');   // L331
const errorsTotal   = counter('ai.errors_total');     // L331
const tokensTotal   = counter('ai.tokens_total');     // L332

// 2 · THE HISTOGRAMS (L331) — the distributions (L331).
const latencyHist = histogram('ai.latency_ms', [10, 50, 100, 500, 1000, 5000]);  // L333
const ttftHist    = histogram('ai.ttft_ms', [100, 300, 500, 1000, 2000]);        // L145

// 3 · THE GAUGES (L331) — the current values (L331).
const queueGauge = gauge('ai.queue_depth');           // L270

async function observed(req) {
  requestsTotal.inc();                                // the counter (L331)
  const started = performance.now();
  try {
    const out = await model.invoke(req.prompt);
    tokensTotal.inc(out.usage.total);                 // the tokens (L332)
    latencyHist.observe(performance.now() - started); // the histogram (L333)
    ttftHist.observe(await measureTTFT());            // the TTFT (L145)
    return out;
  } catch (e) {
    errorsTotal.inc();                                // the errors (L331)
    throw e;
  }
}

// 4 · THE ALARM (L274): the error rate over 1% → the page (L274).
// 5 · THE SCALE (L271): the queue depth (L270) → the workers (L271).

What the reader must SEE — the numbers, observed:

  requestsTotal + errorsTotal → the counters (L331)
  tokensTotal                 → the usage (L332)
  latencyHist + ttftHist      → the distributions (L333, L145)
  queueGauge                  → the current (L270)
  the alarm + the scale       → the controls (L274, L271)

  The counters, the gauges, the histograms (L331).

• The observe is the request's latency (L331). The counters (L331) — the negligible (L331) cost (L331).
• The histogram is the storage's cost (L331). The buckets (L331) — the bounded (L331) storage (L285).
• The tokens are the cost's driver (L332). The counter (L332) — the cost (L334) derived (L334).
• The alarm is the incident's speed (L274). The threshold (L274) — the detection (L274) fast (L331).

• The metrics = the numbers' layer (L331): the types, the AI metrics, the alarms.
• The counters: the totals — the requests, the errors, the tokens (L332).
• The gauges: the current — the queue depth (L270), the concurrency (L266).
• The histograms: the distributions — the latency (L333), the TTFT (L145).
• The alarms: the thresholds (L274) — the error rate, the cost (L334).

METRICS = the counters, the gauges, the histograms

THE TYPES (L331)
  the counters (L331) — the monotonic totals (L331)
    the requests (L331), the errors (L331), the tokens (L332)
  the gauges (L331) — the current values (L331)
    the queue depth (L270), the concurrency (L266)
  the histograms (L331) — the distributions (L331)
    the latency (L333) — the p50, the p99 (L333)
    the TTFT (L145) · the tokens per call (L332)

THE AI METRICS (L331)
  the tokens (L332) — the usage and the cost (L334)
  the latency (L333) — the TTFT (L145) and the total (L333)
  the quality (L341) — the evals' scores (L341)

THE ALARMS (L274)
  the error rate over 1% (L274) → the page (L274)
  the cost over the budget (L334) → the alert (L334)

THE SCALE (L271)
  the queue depth (L270) → the workers (L271)

INTERVIEW, 4 MOVES
  1 types    "the counters, the gauges, the histograms (L331)"
  2 AI       "the tokens, the latency, the cost (L331)"
  3 alarms   "the thresholds (L274)"
  4 scale    "the queue → the workers (L271)"

You now hold the numbers: the counters, the gauges, the histograms, and the alarms — with the p99 watched and the queue driving the scale. The instrument panel is lit — and the red lines are set (L331).

Next: the metering your pricing needs — Token Usage Tracking (L332).