The Transformer & Attention Mechanism

The transformer is the neural-network architecture that powers modern LLMs, and attention is its core operation: for every token, it computes a weighted combination of all the other tokens, where the weights — learned during training — decide how much each other token should influence the prediction.

The one-sentence interview answer: "The transformer is the architecture; attention is the operation. At each position, the model computes a query and compares it against the keys of every earlier position; the match scores become weights that mix the values. So every token looks at every earlier token — which is why it can use the whole context — and the whole thing runs in parallel, which is why it scaled."

Think of attention as a relevance-weighted lookup, performed for every token, over the whole context.

Your phone's autocomplete looks at the last few words. A transformer looks at everything so far — and, at each step, decides what matters. In "The capital of France is", the token "is" should attend strongly to "France" (which country?) and to "capital" (what kind of fact?) — and only weakly to "The". Those weights are not hand-written; they are learned from data during pretraining.

Three characters do the work, and their names are the whole architecture:

Attention = the query of every token against the keys of every other token → scores → weighted sum of the values.

  "The capital of France is ___"
    │        │       │       │
    ▼        ▼       ▼       ▼
  q1       q2      q3      q4        query for each token
   │        │       │       │
   └────────┴───┬───┴───────┘
                ▼
        scores = query × keys          ← "how relevant is
   The     0.02                           each earlier token
   capital 0.35                           to 'is'?"
   of      0.08
   France  0.55
                │
                ▼
        weights = softmax(scores)       ← normalised to sum to 1
   The     0.02
   capital 0.35
   of      0.08
   France  0.55
                │
                ▼
        output = Σ weights × values     ← the "context-aware"
                                           representation of 'is'
                │
                ▼
   …fed onward to predict the next token → "Paris"

The key insight is in the middle row: the weights are the model's answer to "what matters here?" — and they are computed from learned parameters, so the model learned which things to attend to.

A transformer is built from stacked blocks. Each block does the same two things, and the depth of the stack (dozens of blocks) is part of what "large" means:

   ┌────────────────────────────────────────────┐
   │  TRANSFORMER BLOCK (×N, stacked)           │
   │                                            │
   │  input tokens                              │
   │      │                                     │
   │      ▼                                     │
   │  ┌────────────────────────────┐            │
   │  │ MULTI-HEAD ATTENTION       │            │
   │  │  several attention passes  │            │
   │  │  in parallel, each looking │            │
   │  │  for a different kind of   │            │
   │  │  relation ("head")         │            │
   │  └────────────────────────────┘            │
   │      │  + residual (add the input back)    │
   │      ▼                                     │
   │  ┌────────────────────────────┐            │
   │  │ FEED-FORWARD NETWORK       │            │
   │  │  a per-token MLP:          │            │
   │  │  transform, don't mix      │            │
   │  └────────────────────────────┘            │
   │      │  + residual                          │
   │      ▼                                     │
   │  output tokens (same shape as input)       │
   └────────────────────────────────────────────┘

Two divisions of labour worth naming:

• Attention mixes information across tokens. It's the only place in the block where tokens talk to each other. Without it, every token would be processed in isolation.
• The feed-forward network processes each token independently. It's where the "thinking" about the token's content happens, per position.

Together with residual connections (the input is added back after each sub-layer, which is what makes it possible to stack hundreds of layers), this block — repeated — is the whole model.

• The "why" behind context windows. Because attention gives every token a path to every earlier token, a transformer can use a long context (L138) — and because that path costs O(n²), the window is a budget you manage, not a feature you ignore.
• The "why" behind reasoning improvements. When the model "thinks" (chain-of-thought, extended thinking), it is generating intermediate tokens — and each one becomes an attended key for the tokens after it. Attention is why writing out the reasoning helps.
• The "why" behind retrieval. Putting documents into the context (L191) works because attention lets the model find the relevant sentence anywhere in that context. RAG is, in a sense, "buying the model better keys to attend to".
• The "why" behind architecture trade-offs. Sparse attention and sliding-window variants exist precisely to cut the O(n²) cost. When a provider advertises "128K context", the engineering question is always what attention made that affordable.

Say it in four moves:

1. The operation. "Attention is a relevance-weighted lookup: each token forms a query, compares it with the keys of the other tokens, and mixes their values by the match scores."
2. The names. "Query, key, value — that's the whole API of attention. The weights are learned during training."
3. The breakthrough. "Before transformers, recurrence processed sequences one step at a time — no parallelism, and distant tokens had to travel through many steps to influence each other. Attention is direct: any token can look at any earlier token in one operation, and the whole sequence is processed in parallel."
4. The cost. "The catch is that every pair of tokens is compared, so attention is O(n²) in sequence length. That's the engineering reality behind context windows and the reason we budget context."

• Attention is a learned relevance, not a fixed rule. The model didn't know that "France" matters to "capital" — it learned it from data. That's why a bigger model on more data attends better, and why attention-based models surprised everyone.
• The residual connection is load-bearing. "Attention is all you need" is only true with the residual paths; they're what let the stack be deep. Mentioning them signals you've read past the headline.
• Multi-head means multi-question. Several attention passes in parallel, each looking for a different kind of relation — syntax in one, co-reference in another, position in another. "Multi-head attention" is the interview phrase for "several relevance questions at once".
• Position is added, not inherent. A transformer has no intrinsic notion of order; it gets position from positional encodings injected into the embeddings. That's why the "order" of your prompt matters — it's a learned positional pattern, and it's why reordering a prompt changes the answer.

• Saying "attention looks up facts". It computes relevance and mixes representations; it doesn't retrieve a stored answer. (Same trap as L135, one level deeper.)
• Confusing the transformer with the model. The transformer is the architecture; the model is the trained weights. "GPT-4 is a transformer" is fair; "the transformer is GPT-4" blurs it.
• Forgetting that attention runs inside the model, not over your documents. RAG puts text into the context so attention can use it; attention itself has no access to anything outside the context.
• Stating the cost wrong. It's quadratic in sequence length (n² token pairs), not in model size — which is why the context window, not the parameter count, is the number that bites you at inference.

• Use the QKV analogy to check yourself. If you can't say what the query, key, and value each do, you don't have the mechanism yet.
• Blame the O(n²) — correctly. When a prompt is too long, you're paying attention cost on every pair of tokens; that's the correct sentence for "why is my long prompt slow" (L138, L151).
• Ground explanations in "learned relevance". Any time you're tempted to say "the model knows", replace it with "the model learned to attend" — it keeps the mechanism honest.
• Know what you don't need to know. You don't need to implement backprop or derive softmax to be a solutions architect. Know the shape of the operation, not the calculus.

Q: What is attention, in a transformer?

A: It's a relevance-weighted operation over the sequence: every token forms a query, compares it against the keys of every other token, and produces a weighted sum of their values. The weights are learned, so the model learns what to attend to.

Q: Why did attention beat the previous approaches (RNNs, LSTMs)?

A: Two reasons. Parallelism — attention processes the whole sequence in one shot, while recurrent networks had to go step by step. And a direct path — any token can influence any earlier token in one operation, while in a recurrent network the influence had to travel through every intermediate step, which is how long-range information got lost.

Q: What are the query, key, and value?

A: Query is what a token is looking for; key is what each other token offers; value is what each token contributes. Attention scores each query against all keys and mixes the values by those scores. It's a learned, differentiable lookup.

Q: What does O(n²) mean for LLMs in practice?

A: Doubling the context quadruples the attention work, so context length has a real compute and latency price. That's why providers engineer sparse or sliding-window attention for long contexts, and why we budget context rather than using it all.

• What is a "head" in multi-head attention?
• Why are residual connections important in a deep transformer?
• How does the model know the position of a token, if attention is order-agnostic?
• Where does the feed-forward network fit, and why is it separate from attention?
• If attention gives every token a path to every earlier token, why do long contexts still lose information (L138)?

The senior read: attention won on directness and parallelism; its tax is quadratic cost — and every context-engineering trick in this module exists to pay that tax wisely.

This is a sketch — the real thing is matrix multiplications on GPU kernels. The sketch is worth writing because it makes "query, key, value, weights, mix" concrete:

// One attention head, in ~10 lines — the concept, not the implementation.
function attention(Q, K, V) {
  // Q, K, V are arrays of vectors, one per token (simplified scalars here)
  const scores = Q.map((q) => K.map((k) => q * k));     // query × keys
  const weights = scores.map((row) => softmax(row));      // normalise → sum to 1
  return weights.map((row, i) =>
    row.reduce((sum, w, j) => sum + w * V[j], 0)          // weighted mix of values
  );
}

function softmax(xs) {
  const exps = xs.map((x) => Math.exp(x));
  const total = exps.reduce((a, b) => a + b, 0);
  return exps.map((e) => e / total);
}

// "The capital of France is" → attention over the earlier tokens
const q = [0.1, 0.2, 0.3, 0.9];      // what "is" is looking for
const k = [0.9, 0.4, 0.2, 0.1];      // what each token offers
const v = [1.0, 2.0, 3.0, 4.0];      // what each token contributes
console.log(attention(q, k, v));
// → weights favour the 4th token ("France"), so the output is close to its value

What the reader must SEE — the three steps are literally in the code:

  scores   = query × every key        → "how relevant is each earlier token?"
  weights  = softmax(scores)          → "relevance, normalised to sum to 1"
  output   = Σ weights × values       → "the context-aware mix"

• The quadratic wall is real. At 8K tokens that's ~64M pairwise scores; at 128K it's ~16 billion. Long-context models use sparse/local attention, chunked processing, or both — the O(n²) is the reason your 100K-token prompt is slow (L138, L151).
• Attention is memory-hungry. The scores matrix must be held (or recomputed) during the forward pass; memory grows with n² too. That's why context length and batch size fight each other on the GPU.
• KV cache. Inference caches the keys and values of already-generated tokens so each new token only attends to new queries — that's why long generations are cheaper per token after the first, and why the KV cache is a deployment cost number (L151, L305).
• Parallelism is the scale win. Training and batched inference parallelise across tokens; that's what made "scaling laws" (L140) possible.

• The transformer is the architecture; attention is the operation — "Attention Is All You Need" (2017).
• Attention = query × keys → softmax weights → weighted sum of values. Relevance-weighted lookup, learned from data.
• QKV roles: query = what I'm looking for, key = what you offer, value = what you contribute.
• Multi-head = several parallel attention questions per layer.
• Block = attention (mixes across tokens) → feed-forward (transforms per token) + residuals — stacked dozens deep.
• Position is injected, not inherent — order is meaningful.
• Cost = O(n²) in sequence length — the reason context is a budget, not a feature.

TRANSFORMER = stacked blocks:
  attention (across tokens) → feed-forward (per token) + residual

ATTENTION, 3 STEPS
  1 scores   query × keys          "how relevant is each token?"
  2 weights  softmax(scores)       "relevance, summing to 1"
  3 output   Σ weights × values    "the context-aware mix"

QKV
  Q = what I'm looking for
  K = what you offer
  V = what you contribute

WHY IT WON
  direct path between distant tokens   (no vanishing gradient)
  full parallelism                     (no sequential steps)
  learned relevance                    (weights from data)

THE TAX
  O(n²) in sequence length  →  context windows are a budget
  KV cache makes generation cheaper per token after the first

INTERVIEW, 4 MOVES
  1 operation "relevance-weighted lookup"
  2 names     "query, key, value"
  3 win       "parallel + direct path"
  4 tax       "O(n²) → context budgeting"

Lesson 135 gave you the sentence — the model predicts the next token. This lesson gave you the machine that makes it true — attention, the learned relevance-weighted lookup. Those two ideas, held together, unlock every remaining lesson in this module: context windows are the O(n²) made visible (L138), hallucinations are the learned-relevance gone wrong (L141), prompt engineering is teaching attention what to attend to (L142), and embeddings (L147) are attention's neighbours in the same family of "learned representations".

When you can say "the model predicts the next token, and attention is how it decides what matters for that prediction" — you've stopped describing AI and started explaining it. That's the step this module is for. Next: tokens, the units the whole thing runs on.