Structured Outputs & JSON Schemas

Structured outputs are a provider feature that constrains a model's generation to conform to a JSON schema — so instead of hoping the model emits the shape you asked for, the provider guarantees the output is valid against your schema.

The one-sentence interview answer: "Structured outputs make the LLM's output machine-readable by construction: you pass a JSON schema, and the provider constrains generation so the returned content parses and validates against it. The model can still be semantically wrong — but it can no longer be malformed. That turns a text model into a typed function in your codebase."

Think of structured outputs as training wheels for the model's output — or better, a mould. The prompt says "please output JSON"; the schema is the mould the tokens are poured into. The model still chooses the content; the mould decides the shape.

   Prompt-as-JSON (hoping)              Structured output (guaranteed)
   ┌──────────────────────────┐        ┌──────────────────────────┐
   │ "Return JSON like:       │        │ response_format = {      │
   │  {name, price, qty}"     │        │   type: 'json_schema',   │
   │                          │        │   schema: {…}           │
   │ → model may return:      │        │ }                        │
   │  "Sure! Here is the      │        │                          │
   │   JSON: {name:…}"        │        │ → model MUST return      │
   │  or a prose paragraph,   │        │   valid JSON matching    │
   │  or {name:…} without     │        │   the schema — or the    │
   │  quotes, or truncated    │        │   call fails cleanly     │
   └──────────────────────────┘        └──────────────────────────┘
        your parser eats the mess             your parser never sees a mess

The key difference: prompting asks; structured output constrains. One is a hope you parse around; the other is a contract the provider enforces during generation.

   Your code
   ┌───────────────────────────────────────────────┐
   │ 1 · define the schema                         │
   │     { type: 'object',                         │
   │       properties: {                           │
   │         name:   { type: 'string' },           │
   │         amount: { type: 'number' },           │
   │         tags:   { type: 'array',              │
   │                   items: { type: 'string' } } │
   │       },                                      │
   │       required: ['name','amount'] }           │
   └──────────────────┬────────────────────────────┘
                      ▼
   ┌───────────────────────────────────────────────┐
   │ 2 · send with response_format = schema       │
   │     (the provider constrains the tokens)     │
   └──────────────────┬────────────────────────────┘
                      ▼
   ┌───────────────────────────────────────────────┐
   │ 3 · response: JSON that parses & validates   │
   │     { "name": "Widget", "amount": 4,         │
   │       "tags": ["tools"] }                    │
   └──────────────────┬────────────────────────────┘
                      ▼
   ┌───────────────────────────────────────────────┐
   │ 4 · YOU still validate (trust, verify):      │
   │     - schema-valid?  (yes, guaranteed)       │
   │     - semantically right? (NOT guaranteed)   │
   │     → the model can be valid-but-wrong       │
   └───────────────────────────────────────────────┘

The round trip has one asymmetry worth memorising: the provider guarantees shape, never truth. A schema-valid JSON can contain a hallucinated amount, a wrong name, or a fabricated tag. Structured output removes the parsing failure — it does not remove the content failure (that's L141 and L191's job).

The provider doesn't "validate after the fact" — it constrains during generation. Each token is sampled from a set that keeps the partial output on a path that can still complete to a schema-valid JSON:

• The model generates a grammar-constrained token stream: at every step, the possible next tokens are filtered to those that keep the output parseable against the schema.
• It cannot emit a stray ], a missing quote, or prose — those would make the output invalid, so they're never sampled.
• The result is deterministic in shape: valid JSON, matching your schema, every time (or a clean provider error, which you handle like any API error).

• Extraction pipelines. Invoice → {vendor, amount, dueDate}, email → {category, priority, actionItems}. Structured outputs make the downstream parser trivial and the pipeline typed.
• Classification at scale. Labels as a constrained enum (['refund','billing','other']) — the schema itself prevents the model from inventing a category (L142's contract, now enforced).
• Tool calling (L144). The tool-call arguments are structured outputs: {tool: 'get_stock', args: {ticker: 'AAPL'}}. The agent loop parses this shape; schema-conformance is what makes the loop reliable.
• Form-filling and data entry. Turn a document into a typed record — schema in, valid record out, with the model doing the semantic work and the schema doing the shape work.
• Multi-step agents. Each step's output feeds the next (L200); if every step is schema-valid, the loop's data flow is type-safe even though the model is stochastic.

The through-line: structured output is the seam that makes a stochastic text model composable into deterministic systems. It's not an output nicety — it's the interface design of AI products.

Say it in four moves:

1. The definition. "Structured outputs constrain the model's generation to match a JSON schema — the provider guarantees the output is valid against it, not just 'asked for'."
2. The mechanism. "The constraint is applied during generation: at each token, only continuations that can still complete to valid JSON are sampled. So malformed output is impossible, not improbable."
3. The honest limit. "The guarantee is syntactic. The model can still be semantically wrong — a schema-valid JSON with a hallucinated value. So I validate the shape (guaranteed) and eval the content (my job)."
4. The consequence. "That's what makes an LLM composable: a typed function in your codebase, with a defined input contract and a defined output contract — and a parser that never has to guess."

• Structured outputs are the difference between "an AI feature" and "a reliable AI feature". The parser is where text-model integration breaks in production; schema-constraining the model removes the entire failure class.
• Design the schema like an API contract. Field names, types, required vs optional, enum values — they're your interface. A good schema steers the model: enums prevent invented labels; required fields force completeness; descriptions in the schema guide values.
• Pair the schema with a validator on your side. Provider guarantees shape; your code validates the received payload (the SDK version, the model, the proxy can all differ). Validate at the boundary — same rule as any API integration.
• "Valid but wrong" is the remaining failure surface. Structured output eliminates parse errors, not hallucinations. The senior architecture is schema for shape + grounding for truth + evals for regression (L191, L343).

• Prompting "return JSON" and parsing with regex. That's hoping, not engineering — one model update or emoji away from a broken parser.
• Skipping client-side validation. The provider guarantees the response against the schema you sent — but your code should still validate what it receives (different model, SDK, proxy, version).
• Forgetting the schema steers semantics. enum: ['refund','billing'] isn't just validation — it tells the model the allowed answer set. Use the schema as a prompt, not just a check.
• Assuming valid = correct. A schema-valid {amount: 9999} can still be a hallucinated amount (L141). Structured output fixes the parser, not the truth.
• Not handling the provider error path. When the model can't produce valid output, the API errors — and unhandled, that's a 500 in your UI. Handle the structured-output failure like any dependency failure (L168).

• Use structured outputs for anything that feeds code. If you parse it, constrain it. Parser-driven features get schemas; prose features don't need them.
• Write schemas as contracts. Required fields, tight enums, descriptive field names — the schema is the interface and the instruction.
• Validate at the boundary. Parse + validate the response in one place; never trust content strings downstream.
• Keep the schema stable. The schema is a cache key and an eval contract; changing it is a breaking change (L341).
• Ground the values too. Schema for shape, retrieval/tools for truth (L191, L144) — the two guarantees compose.

Q: What are structured outputs, and why do they matter?

A: They constrain the model's generation to conform to a JSON schema — the provider guarantees valid output rather than 'asking' for it. They matter because they turn a stochastic text model into a typed function: the parser never guesses, and the failure class of malformed output is removed.

Q: How does the provider guarantee the JSON is valid?

A: The constraint is applied during generation. At every token step, the model can only sample continuations that keep the output on a path to schema-valid JSON. So it can't emit a stray brace or prose — invalid output is impossible, not improbable.

Q: Does structured output eliminate hallucinations?

A: No — and it's important to say so. It guarantees the shape, not the truth. The model can still return a schema-valid JSON with a hallucinated value. Structured output removes the parser failure; grounding (L191) and evals (L343) handle the content failure.

Q: When should you NOT use structured output?

A: When the output is prose for a human — chat, creative writing, open-ended answers. There's no parser to protect, and a schema would fight the naturalness. I use it wherever the output feeds code, and plain generation where it feeds eyes.

• How does a JSON schema steer the model's semantics, not just validate its shape?
• What's the relationship between structured outputs and tool calling (L144)?
• How would you handle the case where the model refuses to produce the schema-valid output?
• Why is client-side validation still needed if the provider guarantees the schema?
• How do you version a schema so old cached outputs stay compatible?

The senior read: the table is a decision rule. If a parser touches the output, use the schema — the "prompt says JSON" row is a demo, and the structured-output row is the product.

// The same extraction task, structured-output API on each provider.
// The schema is the contract — identical shape, provider-specific call.

const invoiceSchema = {
  type: 'object',
  properties: {
    vendor:   { type: 'string' },
    amount:   { type: 'number' },
    currency: { type: 'string', enum: ['USD', 'EUR', 'GBP'] },
    dueDate:  { type: 'string' },   // ISO date
    lineItems: {
      type: 'array',
      items: { type: 'string' },
    },
  },
  required: ['vendor', 'amount', 'currency', 'dueDate'],
  additionalProperties: false,
};

// OpenAI — response_format with json_schema
const res = await openai.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Extract from this invoice: …' }],
  response_format: {
    type: 'json_schema',
    json_schema: { name: 'invoice', schema: invoiceSchema, strict: true },
  },
});

// Anthropic — tool-use with an input schema (the tool is "return JSON")
//   tools: [{ name: 'extract_invoice', input_schema: invoiceSchema }]
//   → the model emits a tool_use block with the args

// Gemini — responseSchema on the generation config
//   generationConfig: { responseMimeType: 'application/json',
//                       responseSchema: invoiceSchema }

const parsed = JSON.parse(res.choices[0].message.content);
console.log(parsed.amount);   // a number, guaranteed by the schema

What the reader must SEE — one schema, three provider dialects:

  OpenAI    response_format.json_schema (strict)
  Anthropic tool_use with input_schema
  Gemini    generationConfig.responseSchema

  The contract is portable; the call is provider-specific (L152-154).

• Structured outputs can be slower than free-form for complex schemas. The constrained sampling is a constraint on the token choice; deeply nested or huge schemas cost more per token. Keep schemas flat and small when latency matters (L151).
• The schema is part of the prompt-cache story. A stable schema (and a stable system prompt, L142) keeps the request cache-friendly (L171).
• Malformed-output retries are the hidden cost of prompting-only JSON. Every "please fix the JSON" retry is extra tokens + latency; structured output removes the retry class entirely.
• Validation is cheap; do it always. A few microseconds of schema validation (L344-style, or plain zod) is nothing against a retry loop or a parser crash.

• Structured outputs = schema-constrained generation — valid JSON by construction, not by hope.
• The constraint is during generation (token filtering), not post-hoc validation.
• It guarantees shape, not truth — valid-but-wrong is still on you (L141, L191).
• Schema = contract + instruction: enums steer, required forces, descriptions guide.
• Validate at the boundary anyway — trust the shape, verify the payload.
• Feeds code → schema; feeds eyes → plain prose. The decision rule is a parser.

STRUCTURED OUTPUT = the mould the tokens are poured into

MECHANISM
  at each token, sample only continuations that can still
  complete to schema-valid JSON  →  malformed is impossible

THE CONTRACT
  type: object|array|string|number|boolean|null|enum
  required: […]          → forces completeness
  enum: […]              → steers the allowed values
  additionalProperties: false  → forbids extras
  descriptions           → guides what goes in

PROVIDER DIALECTS (same contract)
  OpenAI     response_format.json_schema (strict)
  Anthropic  tool_use input_schema
  Gemini     generationConfig.responseSchema

WHAT IT GUARANTEES / DOESN'T
  ✓ valid shape, every time (or clean provider error)
  ✗ truth — the values can still be hallucinated (L141)

THE DECISION RULE
  feeds code  → structured output
  feeds eyes  → plain prose

INTERVIEW, 4 MOVES
  1 definition "constrains generation to a schema"
  2 mechanism  "token filtering during generation"
  3 limit      "shape yes, truth no"
  4 practice   "schema for shape + grounding for truth + evals"

Structured outputs are the single most important reliability primitive in AI engineering: they are where a stochastic text generator becomes a typed function in your system — with a schema for input contract, a schema for output, and a parser that never guesses. Everything that follows — tool calling (L144), the Vercel AI SDK's parts (L160–161), agents (L200), extraction pipelines, evals (L343) — is built on this seam.

Keep the honest sentence close: the schema guarantees the shape; the architecture guarantees the truth. Next: the primitive that turns the model from a generator into an actor — function calling.