Chunking Fundamentals

Chunking is the granularity decision of RAG — splitting documents into retrieval-sized pieces, balancing context (bigger chunks carry more) against precision (smaller chunks retrieve cleaner), with overlap to protect boundary answers — a size, an overlap, and a boundary strategy, chosen for the retrieval task and measured by retrieval quality (L195).

The one-sentence interview answer: "Chunking is the granularity trade. Too small — a chunk lacks the context to answer (L178). Too big — the answer is buried in noise, and the context blows the token budget (L149, L191). I choose size by the task: sentence-ish chunks for precise Q&A, section chunks for summaries (L179). I add overlap so answers split across boundaries still retrieve whole (L179). And I make the boundaries respect the document's structure — headings, paragraphs, code blocks (L177) — because structure-aware chunks retrieve better than fixed-size slices (L195)."

Think of chunking as cutting a loaf of bread for sandwiches — the slice size decides the sandwich. Too thin, and the filling (the answer) doesn't fit on one slice — it's split across two, and neither sandwich works alone. Too thick, and every sandwich is mostly bread (noise) — the filling is in there, but so is a lot you didn't want, and the box (the token budget, L149) only holds so many sandwiches. The baker (you) chooses the slice for the meal (the task): thin for a precise bite, thick for a hearty one.

   too small (thin slices)        too big (thick slices)
   ┌──────┐ ┌──────┐ ┌──────┐    ┌──────────────────────────┐
   │ chunk │ │ the  │ │ is   │    │ the whole section in one  │
   │ 1     │ │answer│ │ split│    │ chunk — the answer is     │
   │ (no   │ │is    │ │      │    │ in there, but buried in   │
   │  ctx) │ │split │ │      │    │ noise + budget blowout    │
   └──────┘ └──────┘ └──────┘    └──────────────────────────┘
     no context                    too much noise (L149, L191)

   just right — task-sized, boundary-aware (L179)
   ┌──────────────────────────┐
   │ the section, as one chunk │
   │ with its heading (L178)   │
   └──────────────────────────┘

The mental model is slice for the meal: the chunk is sized for the retrieval task, not for the file — and the boundaries follow the bread's own structure (headings, paragraphs).

   clean, structured text (L177)
        │
        ▼
   ┌──────────────────────────────────────────────────────────┐
   │ 1 · CHOOSE SIZE (L178)                                   │
   │     task: precise Q&A → smaller; summary → larger        │
   │     content: dense (contracts) → smaller; prose → larger │
   │     budget: chunk × top-k ≤ context budget (L149)        │
   └──────────────────┬───────────────────────────────────────┘
                      ▼
   ┌──────────────────────────────────────────────────────────┐
   │ 2 · CHOOSE BOUNDARIES (L179)                             │
   │     respect structure: headings, paragraphs, code blocks │
   │     fixed-size only as a fallback (L179)                 │
   └──────────────────┬───────────────────────────────────────┘
                      ▼
   ┌──────────────────────────────────────────────────────────┐
   │ 3 · ADD OVERLAP (L179)                                   │
   │     small overlap at boundaries so answers split across  │
   │     chunks still retrieve whole                          │
   └──────────────────┬───────────────────────────────────────┘
                      ▼
   ┌──────────────────────────────────────────────────────────┐
   │ 4 · MEASURE (L195)                                       │
   │     retrieval on a golden set → tune size / overlap      │
   └──────────────────────────────────────────────────────────┘
                      ▼
   chunks → embed (L181) → index (L180)

The flow is the decision: size for the task, boundaries from structure, overlap for safety, measured by retrieval. Chunking is a tunable knob with a measuring loop (L195), not a fixed default.

• The trade. A chunk is the unit of retrieval (L189) — it's what the model reads (L191), so it must contain enough context to answer and be specific enough to match the query. Too small: the answer lives outside the retrieved chunk (a miss). Too big: the chunk matches the query but the answer is buried in noise — and top-k × chunk-size explodes the token budget (L149).
• Size by task (L179). Precise Q&A wants sentence/paragraph chunks — tight, specific. Summaries and overviews want section chunks — the full context of a topic. There's no universal number; the size is a function of the question being asked (L179).
• Overlap (L179). A small overlap (e.g. 10–15%) at chunk boundaries keeps answers that straddle a split retrievable whole — the "split-answer" failure mode is the reason overlap exists.
• Boundaries (L179). Structure-aware boundaries — split at headings, paragraphs, lists, code blocks (L177) — produce chunks that are internally coherent, which retrieves better than fixed-size slices that cut mid-sentence. The document's own structure is the best chunking map.

• Legal contracts. Dense clauses → smaller chunks by clause (L179) → "what's the termination clause?" retrieves the exact clause, not the whole contract.
• Product manuals. Sections with tables → section chunks preserving tables (L177) → "torque spec" finds the table chunk.
• Support docs. FAQ-style → sentence/paragraph chunks → precise answers with citations (L192).
• Research papers. Section chunks by heading (abstract, methods, results) → "what method did they use?" retrieves the methods section.
• Code documentation. Code blocks as chunk boundaries (L177) → "how do I use this function?" retrieves the function's docs + example together.

The through-line: chunking is the retrieval task wearing a text-splitting hat — the size, boundary and overlap choices are quality decisions (L195), not defaults.

Say it in four moves:

1. The trade. "Chunking balances context against precision: too small loses context, too big buries the answer and blows the budget (L149)."
2. The size. "I size for the task — precise Q&A wants sentence/paragraph chunks, summaries want sections (L179). The budget comes first (L149): chunk × top-k must fit the context."
3. The boundaries. "I split on structure — headings, paragraphs, code blocks (L177) — with a small overlap so straddling answers stay whole (L179)."
4. The measurement. "Then I measure retrieval on a golden set (L195) and tune size and overlap — chunking is a knob with a gauge, not a default."

• Chunking is a retrieval-quality decision (L195). The senior answer treats chunk size as a measured hyperparameter — golden set, retrieval metrics, tune. The demo treats it as a constant.
• The budget is the constraint (L149, L191). Chunk size and top-k compose on one token budget — the senior design sets the budget first (L149) and derives the chunk size inside it, keeping the context cost (L150) bounded.
• Structure is the best chunking map (L177, L179). Headings, paragraphs and code blocks are the document's own outline — structure-aware chunking retrieves better than any fixed-size split (L179).
• Overlap is the boundary insurance (L179). The split-answer failure — the answer straddling two chunks — is prevented by a small overlap and caught by evals (L195). Both are the design.
• Chunking interacts with everything downstream (L189, L191). Chunk size sets the retrieval unit, the context budget, and the citation granularity (L192). A change to chunking re-tunes the whole pipeline — which is why it's measured (L195), not guessed.

• A fixed character count everywhere. No task, no structure (L179) — the universal default that fits no document well.
• Too small. Sentence fragments with no context (L178) — the retrieved chunk can't answer alone.
• Too big. Whole documents as "chunks" (L149) — the answer is buried, the budget explodes, top-k is meaningless.
• No overlap. Answers split across boundaries retrieve as halves (L179).
• Cutting through structure. Mid-table, mid-code-block, mid-heading splits (L177) — chunks that are internally incoherent.
• Never measuring. Chunking set once and never tuned (L195) — the knob with no gauge.

• Set the budget first (L149) — context window minus system/instructions, then chunk size and top-k fit inside it (L191).
• Choose size by task (L179) — precise Q&A: smaller; summary: larger; dense content: smaller.
• Split on structure (L177) — headings, paragraphs, lists, code blocks — with fixed-size only as fallback.
• Add a small overlap (L179) — 10–15% keeps boundary answers whole.
• Keep tables and code intact (L177) — a chunk is one coherent unit.
• Measure and tune (L195) — a golden set scores the chunking config; change it with data, not vibes.

Q: How do you chunk documents for RAG?

A: As a granularity trade (L178). Too small — no context to answer; too big — the answer is buried and the budget blows up (L149). I size for the task: precise Q&A wants sentence/paragraph chunks, summaries want sections (L179). I split on structure — headings, paragraphs, code (L177) — with a small overlap for boundary answers. Then I measure retrieval on a golden set (L195) and tune.

Q: What's the right chunk size?

A: There's no universal number — it's a function of the task and the budget (L149). The constraint is first: top-k × chunk size must fit the context window (L191). Inside that, precise Q&A wants smaller chunks (tight, specific), summaries want larger (full context). The golden set (L195) decides between candidates — measured, not guessed.

Q: Why overlap?

A: Because answers straddle boundaries (L179). Without overlap, a question whose answer is split across two chunks retrieves each half — and neither answers alone. A small overlap (10–15%) keeps the straddling answer intact in at least one chunk. It's cheap insurance against a specific failure mode.

Q: Why do boundaries matter more than size?

A: Because a chunk is a unit of meaning, and structure is the document's own meaning map (L177). A chunk that respects headings and paragraphs is internally coherent — it retrieves as a unit and answers alone. A fixed-size slice cuts mid-sentence, mid-table, mid-code — internally incoherent, retrieves poorly (L195). Boundaries are the quality lever; size is the budget lever.

• How do chunk size and top-k share the budget (L149)?
• When is semantic chunking (L179) the right strategy?
• How do you measure a chunking change (L195)?
• How does chunk size affect citation granularity (L192)?
• How does chunking interact with hybrid search (L187)?

The senior read: the middle column is the discipline — chunking is a measured trade, not a default.

// Structure-aware chunking: size for the task, boundaries from the doc (L178-179).
function chunkDocument(text, { targetTokens = 200, overlap = 0.15 } = {}) {
  const sections = splitByHeadings(text);           // structure first (L177)
  const chunks = [];

  for (const section of sections) {
    const blocks = splitByBlocks(section);           // paragraphs, lists, code (L177)
    let current = [];

    for (const block of blocks) {
      const next = [...current, block].join('\n\n');
      if (tokens(next) > targetTokens && current.length) {
        // Emit the chunk with overlap: keep the last block for the next one (L179).
        chunks.push(current.join('\n\n'));
        const overlapBlock = block.slice(0, Math.round(block.length * overlap));
        current = [overlapBlock, block];
      } else {
        current.push(block);
      }
    }
    if (current.length) chunks.push(current.join('\n\n'));
  }
  return chunks;
}

// The budget check (L149, L191): top-k × chunk tokens ≤ context budget.
const budget = 4000;                                  // context window (L138)
const topK = 5;
const maxChunkTokens = Math.floor(budget / topK);     // size derived FROM the budget

What the reader must SEE — the three decisions in code:

  splitByHeadings / splitByBlocks → boundaries from structure (L177)
  overlap slice                  → boundary answers stay whole (L179)
  budget / topK                  → size derived from the budget (L149, L191)

  The chunk is sized by the budget and shaped by the document.

• Chunk size sets the context cost (L149, L150). Doubling the chunk size doubles the per-question token spend at the same top-k — the budget arithmetic (L191) is the cost control.
• Overlap is a storage tax (L180). Overlap duplicates tokens in the index (L150) — keep it small; it's insurance, not content.
• Embedding cost scales with chunk count (L181). Smaller chunks = more chunks = more embedding calls (L150) — another reason size is a budget decision.
• Retrieval latency is roughly constant (L151). The vector search cost (L182) doesn't change much with chunk size — but the context generation cost (L145, L191) does. The bottleneck moves to the model.

• Chunking = the granularity trade: context vs precision (L178).
• Budget first (L149, L191) — top-k × chunk size ≤ context.
• Size for the task (L179): Q&A smaller, summary larger.
• Boundaries from structure (L177): headings, paragraphs, code.
• Overlap for straddling answers (L179) — small, as insurance.
• Measure and tune (L195) — chunking is a knob with a gauge.

CHUNKING = the granularity decision of RAG

THE TRADE (L178)
  too small → no context to answer (miss)
  too big   → buried in noise + budget blowout (L149)

THE THREE KNOBS
  size       for the task (L179): Q&A small, summary large
  boundaries from structure (L177): headings, paragraphs, code
  overlap    10-15% — straddling answers stay whole (L179)

THE BUDGET ARITHMETIC (L149, L191)
  context budget = top-k × chunk size
  set the budget first — size derives from it

THE MEASUREMENT (L195)
  golden set → retrieval precision/recall → tune size/overlap
  chunking is a knob with a gauge, not a default

INTERVIEW, 4 MOVES
  1 trade    "context vs precision"
  2 size     "for the task, inside the budget (L149)"
  3 boundaries "structure first (L177), overlap for safety (L179)"
  4 measure  "golden set tunes it (L195)"

You now hold the granularity decision: size for the task, boundaries from structure, overlap for safety, and a budget that sizes it all. Chunking is the cheapest quality lever in RAG — and the most-tuned, because it's the one that decides what retrieval can find (L195).

Next: the strategies — chunking strategies (L179), from fixed-size to semantic and document-aware splits.