The Google Gemini API

The Google Gemini API is the multimodal-native provider surface: a generateContent endpoint where a request is a list of content parts (text, image, audio, video), with generationConfig for sampling (L139) and structured output via responseSchema (L143) — the same concepts as the baseline (L152), designed around multimodal input from day one.

The one-sentence interview answer: "Gemini's API is the same concepts as the baseline (L152), designed multimodal-first: a request is generateContent with content parts — text and images together (L146) — and the response is a list of candidate parts. Structured output is responseSchema (L143), tools are declared with a function-declaration schema (L144), and finishReason plays finish_reason's role. The dialect is built around parts instead of a single string."

Think of Gemini as the same engine, with "parts" instead of strings — a request is a list of content parts, which is what makes it multimodal-native.

   OpenAI baseline (L152)              Gemini dialect (this lesson)
   ┌─────────────────────────┐        ┌─────────────────────────┐
   │ messages: [             │        │ contents: [{            │
   │   {system}, {user}      │        │   parts: [              │
   │ ]                       │        │     {text: "…"},        │
   │ response_format: schema │        │     {inlineData: img}   │  ← parts, not strings
   │ tools: [ {function} ]   │        │   ]                     │
   │                         │        │ }]                      │
   │ response: content OR    │        │ generationConfig: {     │
   │   tool_calls            │        │   responseSchema }      │  ← structured (L143)
   └─────────────────────────┘        │ tools: [{functionDecl}] │
                                      └─────────────────────────┘
        same concepts ──── "parts" instead of strings ──── same concepts

The mental model for interviews: know the baseline (L152), and Gemini is "like that, except it's built around parts" — which is why multimodality (L146) is a first-class citizen instead of a bolt-on.

   your code
   ┌──────────────────────────────────────────────────────┐
   │ 1 · build:  contents: [{ parts: [                    │
   │     {text}, {inlineData: image}, …] }]               │
   │     + generationConfig: { temperature (L139),        │
   │         responseSchema (L143), maxOutputTokens }     │
   │     + tools: [{ functionDecl }]  (L144)              │
   └──────────────────┬───────────────────────────────────┘
                      ▼
   ┌──────────────────────────────────────────────────────┐
   │ 2 · POST /v1beta/models/{model}:generateContent      │
   └──────────────────┬───────────────────────────────────┘
                      ▼
   ┌──────────────────────────────────────────────────────┐
   │ 3 · response: candidates[0].content.parts            │
   │     [{text:"…"},                                     │
   │      {functionCall: {name, args}},                   │
   │      …]                                              │
   └──────────────────┬───────────────────────────────────┘
                      ▼
   ┌──────────────────────────────────────────────────────┐
   │ 4 · you:  render text · execute functionCall (L144)  │
   │     · append functionResponse part · loop            │
   │     · read finishReason + usageMetadata (L149)       │
   └──────────────────────────────────────────────────────┘

The lifecycle is L152's — build → post → branch on parts → loop on functionCall → read finishReason + usageMetadata — with parts carrying the multimodality (L146) that the other dialects bolt on.

• generateContent is the endpoint; a request is contents with parts. Each part is a typed unit — text, inlineData (image/audio/video bytes), fileData (referenced files). Multimodal input (L146) is structural, not a special mode: you just add an image part.
• generationConfig holds the knobs: temperature (L139), maxOutputTokens (L149), topP, and responseSchema — the structured-output mechanism (L143). It's the params object of the dialect.
• Structured output is responseSchema (with responseMimeType: 'application/json'): the provider constrains generation to your schema — L143's guarantee, in Gemini's dialect.
• The tool loop (L144) uses functionDecl declarations; the model responds with a functionCall part; you execute and return a functionResponse part (with name matching), then call again.
• finishReason plays finish_reason's role (L145): STOP (done), MAX_TOKENS (truncated), TOOL_CALLS (execute and loop), SAFETY (the safety filter stopped it — a Gemini-specific reason worth handling).
• usageMetadata is the token ledger (L149): promptTokenCount and candidatesTokenCount, logged per call (L332).
• The model family is multimodal-native (L146): the models are trained for text+image+audio+video together, and the API's parts structure reflects it.

• Multimodal-native products (L146). Vision, audio, video understanding where the media is the input — the parts structure is the natural fit.
• Google-ecosystem stacks. Vertex AI, Google Cloud (L261+), BigQuery — when the architecture is already on Google's cloud, Gemini is the integration path.
• Structured extraction (L143). responseSchema + JSON mime type is a clean structured-output dialect for extraction pipelines.
• The third provider in a multi-provider stack (L155, L156). Routing by task, cost, or eval — the abstraction makes Gemini one of three interchangeable dialects.
• Long-context media analysis. Video + long documents together — a multimodal-long-context combination the other two handle less natively.

The through-line: Gemini is the "multimodal-native dialect" — the same grammar as L152, built around parts, and the choice when the input is media or the stack is Google's.

Say it in four moves:

1. The frame. "Gemini's API is the same concepts as the baseline (L152), designed multimodal-first — a request is content parts, not strings."
2. The dialect. "generateContent with parts (text, image, audio); generationConfig for temperature (L139), maxOutputTokens (L149), and responseSchema for structured output (L143); tools via functionDecl (L144)."
3. The loop. "The model emits a functionCall part, I execute it, return a functionResponse part matching the name, and call again (L144)."
4. The choice. "Its differentiators are native multimodality (L146), long context, and Google-ecosystem integration — and L156 is where I'd actually choose between it and the other two."

• The parts structure is the multimodality (L146) made structural. Where OpenAI and Anthropic treat vision as a message-content type, Gemini is parts — the architecture of the API reflects the architecture of the model. Naming that is the senior read.
• finishReason: SAFETY is a first-class flow. The safety filter is a real branch in production (L317): handle it as a designed path, not a mystery — "the model refused" is a UX state, not a bug.
• Provider choice is a stack decision (L156). If the architecture is already on Google Cloud (L261+), Gemini's integration and pricing change the math. The abstraction (L155) keeps the option open either way.
• The abstraction normalises dialects, never prices (L150). Same concepts, different bills — Gemini's aggressive pricing (L150) is a reason to route there, measured, not assumed.
• Multimodal eval is a distinct axis (L343). When the input is media, the eval set is images/audio/video, not text — Gemini's native strength only matters if your eval confirms it.

• Building parts as one string. Text-only requests work, but the point of the dialect is parts — for media, add the inlineData/fileData part, not a base64 blob in the text.
• Forgetting responseMimeType: 'application/json' with responseSchema — the schema is inert without the JSON mime type (L143).
• Not handling finishReason: SAFETY. The safety filter stopping generation is a real production path (L317), not an error to ignore.
• Mismatching the functionResponse name. The tool loop (L144) breaks if the response part's name doesn't match the functionCall.
• Treating usageMetadata as optional. It's the token ledger (L149) — log it (L332), or the cost model (L150) is a guess.
• Assuming "Google = Gemini" without the abstraction. The dialect still lives behind the interface (L155); the stack choice and the model choice are separate decisions.

• Use parts for media — inlineData for bytes, fileData for references (L146).
• Set responseSchema + responseMimeType: 'application/json' together for structured output (L143).
• Complete the tool loop exactly: execute, return functionResponse with the matching name, call again (L144).
• Handle finishReason: SAFETY as a designed UX path (L317).
• Log usageMetadata per call (L332) — the cost ledger (L149).
• Keep it behind the abstraction (L155) — the dialect stays at the adapter, and L156 decides the routing.

Q: How does the Gemini API differ from OpenAI's (L152)?

A: Same concepts, a parts-based dialect. A request is contents with typed parts — text, image, audio — instead of strings; generationConfig holds temperature (L139) and responseSchema for structured output (L143); tools use functionDecl; finishReason plays finish_reason's role. It's multimodal-first by construction (L146).

Q: How does tool calling work in Gemini?

A: Declare tools as functionDecl. The model responds with a functionCall part; I execute it and return a functionResponse part with the matching name, then call again so the model continues with the result (L144). Same loop, parts-shaped.

Q: How do you get structured output from Gemini?

A: generationConfig.responseSchema plus responseMimeType: 'application/json' — the provider constrains generation to the schema (L143), the same guarantee as OpenAI's response_format, in Gemini's dialect.

Q: When would you choose Gemini?

A: When the input is genuinely multimodal (L146) — vision, audio, video — or when the stack is on Google Cloud (L261+), where integration and pricing change the math. For text-only features it's interchangeable behind the abstraction (L155), and L156's table is where the routing decision lives.

• What does the parts structure say about the model's architecture (L146)?
• How does finishReason: SAFETY change your production design (L317)?
• How does Gemini's pricing compare on the cost model (L150)?
• Where does Gemini fit in a multi-provider routing strategy (L155, L156)?
• How do you eval a multimodal model's real quality (L343)?

The senior read: the three columns map 1:1 at the concept level — which is precisely why the abstraction (L155) can present one interface, and why the comparison (L156) is a table of deltas, not a rewrite.

// The Gemini dialect: parts, generationConfig, the tool loop (L144).
const { GoogleGenerativeAI } = require('@google/generative-ai');
const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);

async function askWithImage(question, imageBase64) {
  const model = genAI.getGenerativeModel({
    model: 'gemini-1.5-flash',                     // tier: config, not code (L155)
    generationConfig: {
      temperature: 0.3,                            // sampling knob (L139)
      maxOutputTokens: 300,                        // the output budget (L149)
      responseMimeType: 'application/json',        // ← needed for the schema (L143)
      responseSchema: {                            // structured output (L143)
        type: 'OBJECT',
        properties: {
          answer: { type: 'STRING' },
          confidence: { type: 'NUMBER' },
        },
        required: ['answer'],
      },
    },
    tools: [{                                      // tool declarations (L144)
      functionDeclarations: [{
        name: 'get_stock',
        description: 'Current price of a US ticker.',
        parameters: {
          type: 'OBJECT',
          properties: { ticker: { type: 'STRING' } },
          required: ['ticker'],
        },
      }],
    }],
  });

  // Multimodal input as parts (L146) — text + image, structurally.
  const result = await model.generateContent([
    { text: question },
    { inlineData: { mimeType: 'image/png', data: imageBase64 } },
  ]);

  const parts = result.response.candidates[0].content.parts;
  const call = parts.find((p) => p.functionCall);
  if (call) {
    const { ticker } = call.functionCall.args;
    const price = await fetchPrice(ticker);        // ← YOUR execution
    // return a functionResponse part and loop (L144)
    return model.generateContent([
      { text: question },
      { inlineData: { mimeType: 'image/png', data: imageBase64 } },
      { functionCall: call.functionCall },
      { functionResponse: { name: 'get_stock', response: { price } } },
    ]).then((r2) => r2.response.text());
  }
  return parts.find((p) => p.text)?.text;
}

What the reader must SEE — the dialect, named:

  generateContent + parts      → multimodal is structural (L146)
  generationConfig             → temperature + maxOutputTokens (L139, L149)
  responseSchema + MIME json   → structured output (L143)
  functionDecl → functionCall  → execute → functionResponse → loop (L144)
  finishReason + usageMetadata → the "why" + the ledger (L145, L149)

• Multimodal input costs image tokens (L146, L150) — the parts structure doesn't make vision free; budget the media path (L149).
• responseSchema constrained generation can add latency on complex schemas (L143, L151) — keep schemas flat.
• Long context + media is the expensive combo (L138, L150) — video and huge documents together scale tokens fast; sample frames and retrieve tightly (L146, L189).
• Prompt caching (L171) applies to stable prefixes — a byte-stable system text part is the cache key, same as the other dialects.

• Gemini = the same concepts as L152, a parts-based, multimodal-native dialect.
• generateContent + parts (text, image, audio) — media is structural (L146).
• generationConfig: temperature (L139), maxOutputTokens (L149), responseSchema (L143).
• The tool loop: functionDecl → functionCall → execute → functionResponse → loop (L144).
• finishReason includes SAFETY — a designed path, not a bug (L317).
• Provider choice is a stack + task decision (L156), behind the abstraction (L155).

GEMINI API = the baseline (L152), parts-based and multimodal-native

DIALECT
  generateContent       the endpoint
  contents + parts      text | inlineData | fileData  (L146)
  generationConfig      temperature (L139) · maxOutputTokens (L149)
                        responseMimeType + responseSchema (L143)
  tools                 functionDeclarations (L144)

THE TOOL LOOP (L144)
  model → functionCall {name, args}
  you   → execute
  you   → functionResponse {name, response}
  call again → model continues

FINISH REASONS (L145)
  STOP · MAX_TOKENS · TOOL_CALLS · SAFETY (handle it, L317)

USAGE (L149, L332)
  usageMetadata → promptTokenCount + candidatesTokenCount → the ledger

DIFFERENTIATORS
  native multimodality (L146) · long context · Google-ecosystem
  aggressive pricing — measured, not assumed (L150)

RULES
  parts for media, always
  schema + JSON mime type together (L143)
  handle SAFETY as a UX path (L317)
  behind the abstraction (L155) — L156 decides routing

INTERVIEW, 4 MOVES
  1 frame    "same concepts, parts-based dialect"
  2 dialect  "contents + parts, generationConfig, responseSchema"
  3 loop     "functionDecl → execute → functionResponse → continue"
  4 choice   "multimodal + Google stack → L156 decides"

You now speak all three dialects of the frontier: the baseline (L152), Anthropic's (L153), and Gemini's (this lesson). Hold the concept map — messages/parts, tools, structured output, stop reasons, usage — and the three are variations on one grammar. That map is exactly what the next lesson formalises: the abstraction (L155) that lets your code speak one interface and route across all three — and the comparison (L156) that turns the map into a decision table.