Rate Limiting & Abuse Prevention

The rate limiting & abuse prevention is the control that stops the abuse at the door (L318) — the limits (the per-key L319, the per-IP, and the per-tenant L320 caps, L318), the algorithms (the token bucket L242: the rate and the burst, L318), and the quotas (the budgets: the tokens per day L149 and the cost per month L334, L318) — placed at the gateway (L267) — the L242 discipline (L242), security-shaped (L318).

The one-sentence interview answer: "The rate limiting is the abuse's door control (L318). The limits (L318): the per-key (L319) — the customer's API key (L320) capped (L318); the per-IP (L318) — the source's address (L318) capped (L318); the per-tenant (L320) — the tenant's usage (L320) capped (L318). The algorithms (L242): the token bucket (L242) — the rate (L242) and the burst (L242): 10 tokens per second, 20 burst (L318) — the traffic (L318) shaped (L242). The quotas (L149): the budgets (L149) — the tokens per day (L332) and the cost per month (L334) — the burning (L317) bounded (L318). The placement (L318): at the gateway (L267) — the limits (L318) enforced before the model (L278) — the abuse (L317) never reaches the compute (L318). The AI shape (L173): the model endpoint (L278) — the L242 discipline (L242) applied (L318): the per-key limits (L318), the quota (L149), and the 429 (L318) — the abuse (L317) stopped at the door (L318)."

Think of the rate limiting as the nightclub's velvet rope. The rope (the rate limit, L318) at the door (the gateway, L267): the bouncer (the limiter, L318) counts the guests (the requests, L318) — the regulars (the per-key, L319) get the known allowance (L318), the strangers (the per-IP, L318) get the smaller (L318); the club's capacity (the token bucket, L242) — the guests per minute (the rate, L242) with the short burst (the burst, L242); and the VIP quotas (the budgets, L149) — the bottles per night (the tokens, L332). The rude guests (the abusers, L317) — the line-cutters (the burst, L318) — hit the rope (the 429, L318). The club works because the rope is at the door, the counts are set, and the capacity is watched (L318).

   the velvet rope (the rate limit, L318)
   ┌────────────────────────────────────────────────────────┐
   │ the door (the gateway, L267) — the limit placed (L318) │
   │ the counts (L318) — the per-key (L319), the per-IP     │
   │ (L318), the per-tenant (L320)                          │
   │ the capacity (the token bucket, L242) · the quotas     │
   │ (L149)                                                 │
   └────────────────────────────────────────────────────────┘

The mental model is the rope: the door, the counts, and the capacity (L318).

   the request (L318)
        │
        ▼
   ┌────────────────────── THE GATEWAY (L267) ───────────────────────────┐
   │  the key's bucket (L242): the rate + the burst (L318)              │
   │  the tokens left (L318)? → the allow (L318)                        │
   │  the empty bucket (L318) → the 429 (L318)                          │
   └──────────────────────────┬───────────────────────────────────────┘
                              ▼
   ┌────────────────────── THE QUOTA (L149) ────────────────────────────┐
   │  the tokens per day (L332) · the cost per month (L334)            │
   │  the quota exhausted (L149) → the 429 (L318)                      │
   └──────────────────────────┬───────────────────────────────────────┘
                              ▼
   ┌────────────────────── THE MODEL (L278) ────────────────────────────┐
   │  the admitted request (L318) reaches the model (L278)             │
   └──────────────────────────────────────────────────────────────────┘

The flow is the limit: bucket → quota → model (L318).

• The limits (L318). The per-key (L319), the per-IP (L318), and the per-tenant (L320) caps (L318) — the abuse (L317) bounded by the identity (L318).
• The algorithms (L242). The token bucket (L242): the rate (L242) and the burst (L242) — the traffic (L318) shaped (L242).
• The quotas (L149). The budgets (L149): the tokens per day (L332) and the cost per month (L334) — the burning (L317) bounded (L318).
• The placement (L318). At the gateway (L267) — the limits (L318) enforced before the model (L278) — the abuse (L317) never reaches the compute (L318).

• A production AI SaaS (L357). The gateway (L267) — the per-key (L319) and the per-tenant (L320) limits (L318).
• A model API (L278). The per-customer quota (L149) — the tokens (L332) and the cost (L334) bounded (L318).
• A chatbot (L162). The session limits (L318) — the junk calls (L317) capped (L318).
• A multi-tenant SaaS (L357). The per-tenant limits (L320) — the one tenant's abuse (L317) doesn't starve the others (L320).
• Anything with a model (L317). The door's control (L318) — the L242 discipline (L242), security-shaped (L318).

The through-line: the control is the door's — the limits, the quotas, and the 429 (L318).

Say it in four moves:

1. The limits. "The per-key (L319), the per-IP, the per-tenant (L320)."
2. The algorithm. "The token bucket — the rate and the burst (L242)."
3. The quotas. "The tokens per day, the cost per month (L149)."
4. The placement. "At the gateway (L267) — before the model (L278)."

• The gateway is the limit's home (L318). The limits (L318) at the gateway (L267) — the abuse (L317) stopped before the compute (L318).
• The identity is the limit's key (L318). The per-key (L319) and the per-tenant (L320) — the abuse (L317) bounded by the identity (L318).
• The bucket is the shape (L242). The rate and the burst (L242) — the traffic (L318) shaped (L242) — the burst (L317) absorbed (L318).
• The quota is the budget (L149). The tokens (L332) and the cost (L334) — the burning (L317) bounded (L318).
• The 429 is the contract (L318). The response (L318) — the Retry-After (L318) — the client (L318) backs off (L318).

• The no limits (L318). The open door (L317) — the abuse (L317) free (L318).
• The shared limit (L318). The one bucket for everyone (L318) — the one abuser (L317) starves the rest (L320); the per-key (L319) is the fix (L318).
• The bucket mis-sized (L242). The tiny rate (L242) — the legitimate burst (L318) blocked (L318); the sizing (L242) by the traffic (L318).
• The quota missing (L149). The burst stopped (L318), the burning (L317) un-bounded (L318) — the quota (L149) is the budget's (L318).
• The 429 without the guidance (L318). The client (L318) retrying hard (L318) — the Retry-After (L318) and the backoff (L256) are the contract (L318).

• Limit per identity (L318) — the key (L319), the IP (L318), the tenant (L320).
• Shape with the bucket (L242) — the rate and the burst (L242).
• Bound with the quota (L149) — the tokens (L332) and the cost (L334).
• Place at the gateway (L267) — before the model (L278).
• Return the Retry-After (L318) — the backoff (L256) contract (L318).

Q: Walk me through the rate limiting.

A: The abuse's door control (L318). The limits — the per-key (L319), the per-IP, the per-tenant (L320). The algorithm — the token bucket (L242): the rate and the burst (L318). The quotas — the tokens per day (L332) and the cost per month (L334). And the placement — at the gateway (L267).

Q: Why the token bucket?

A: The shape (L242): the rate (L242) — the sustained traffic (L318) — and the burst (L242) — the short spike (L318) absorbed (L242). The bucket (L242) refills at the rate (L242): the empty bucket (L318) → the 429 (L318), the backoff (L256) → the refill (L242).

Q: How do you stop the abuse?

A: The limits and the quotas (L318): the per-key (L319) and the per-tenant (L320) limits (L318) stop the burst (L317); the quota (L149) — the tokens (L332) and the cost (L334) per customer (L320) — stops the burning (L317). The abuse (L317) hits the 429 (L318) at both (L318).

Q: Where do the limits live?

A: At the gateway (L267): the limits (L318) enforced before the model (L278) — the abuse (L317) never reaches the compute (L318). The Redis (L243) holds the counters (L242) — the per-key buckets (L318) at the door (L267).

• What are the limits (L318)?
• Why the token bucket (L242)?
• What's the quota (L149)?
• Where do the limits live (L267)?
• What's the 429 (L318)?

The senior read: the limit shapes the traffic; the quota bounds the budget (L318).

// The rate limiting (L318) — the door's control (L318).
// 1 · THE TOKEN BUCKET (L242) — the per-key shape (L318).
async function checkRate(key) {
  const bucket = await redis.get(`rl:${key}`);      // the counter (L243)
  const tokens = bucket ?? RATE;                    // the refill (L242)
  if (tokens <= 0) {
    return { allowed: false, retryAfter: retryAfterOf(bucket) };  // L318
  }
  await redis.decr(`rl:${key}`);                    // the token spent (L242)
  return { allowed: true };
}

// 2 · THE QUOTA (L149) — the budget (L334).
async function checkQuota(customerId) {
  const used = await quota.usage(customerId, 'month');   // the tokens (L332)
  if (used >= QUOTA_MONTHLY) return { allowed: false };  // L149
  return { allowed: true };
}

// 3 · THE GATEWAY (L267) — the limits before the model (L278).
async function gateway(req) {
  const rate = await checkRate(req.apiKey);          // L318
  if (!rate.allowed) return error(429, { retryAfter: rate.retryAfter });
  const quota = await checkQuota(req.customerId);    // L149
  if (!quota.allowed) return error(429, { reset: 'month' });
  return invokeModel(req);                           // L278
}

What the reader must SEE — the control, applied:

  redis rl:{key} bucket     → the token bucket (L242, L243)
  retryAfter                → the backoff contract (L318)
  quota.usage('month')      → the budget (L149, L334)
  the gateway before the model → the placement (L267)

  The bucket shapes, the quota bounds, the door guards (L318).

• The counter is the latency's cost (L318). The Redis read (L243) — the sub-millisecond (L318) at the door (L318).
• The bucket is the memory's cost (L243). The per-key counters (L243) — the keys' TTL (L244) bounds the memory (L243).
• The quota is the metering's cost (L332). The tokens (L332) per customer (L320) — the L332 metering (L332) feeds the quota (L318).
• The abuse is the bill's spike (L334). The junk calls (L317) — the limits (L318) and the quotas (L149) bound the cost (L334).

• The rate limiting = the door's control (L318): the limits, the algorithm, the quotas, the placement.
• The limits: the per-key (L319), the per-IP, the per-tenant (L320).
• The algorithm: the token bucket — the rate and the burst (L242).
• The quotas: the tokens per day, the cost per month (L149).
• The placement: at the gateway (L267) — before the model (L278).

RATE LIMITING & ABUSE PREVENTION = the control at the door

THE LIMITS (L318)
  the per-key (L319) — the customer's cap (L318)
  the per-IP (L318) · the per-tenant (L320)

THE ALGORITHM (L242)
  the token bucket (L242) — the rate and the burst (L242)
  the empty bucket (L318) → the 429 (L318)

THE QUOTAS (L149)
  the tokens per day (L332) · the cost per month (L334)
  the burning (L317) bounded (L318)

THE PLACEMENT (L318)
  at the gateway (L267) — before the model (L278)
  the Redis counters (L243) · the TTL (L244)

THE CONTRACT (L318)
  the 429 with the Retry-After (L318) — the backoff (L256)

INTERVIEW, 4 MOVES
  1 limits    "the per-key, the per-IP, the per-tenant (L318)"
  2 algorithm "the token bucket — the rate and the burst (L242)"
  3 quotas    "the tokens and the cost (L149)"
  4 placement "the gateway, before the model (L267)"

You now hold the control: the limits, the bucket, the quotas, and the placement — with the 429 at the door. The velvet rope is across — and the freeloaders are capped (L318).

Next: keys, scopes, and quotas for the model endpoint — Auth for AI APIs (L319).