---
name: mechanism-derivation
description: Derive a formalism, architecture, learning rule, or algorithm so that each piece is forced rather than asserted — and stress-test it afterwards. Use when designing or explaining a model architecture, deriving a loss or update rule, reading a paper whose equations feel arbitrary, choosing an objective, deciding what to ablate, or figuring out what a system has actually learned.
---

# Mechanism Derivation

For getting a *formalism* out: an architecture, an objective, an update rule, an algorithm. Every equation should arrive as the only thing that could have answered a question you already asked.

Pairs with the `derive-dont-declare` skill, which covers the framing moves — kill the naive solution with a number, dissolve the ill-posed question, import a distant isomorph, shrink to the smallest instance that hurts. Assume the problem has already been cornered that way. These are the moves that need an equation or a trainable system in front of you.

## Deriving the math

**1. Read the equation for its free variables.**
Ask of any expression: *what is allowed to change here?* The Hopfield energy depends on both the unit states **and** the weights — so ask which one is being adjusted. There are two answers, and that fork **is** the learning/inference distinction. Adjusting weights sculpts the landscape (learning); adjusting states descends it (inference).

The decomposition of the whole system was sitting in the argument list of one function. Do this before writing any code: enumerate what varies, on what timescale, and you have your components.

**2. Cancel what you cannot compute, before approximating it.**
Two candidate states differ only in unit *i*. Take their ratio, and the entire rest-of-network energy term cancels — locality, the whole point of the model, falls out for free and exactly.

When a global or intractable quantity blocks you (a partition function, a normalizer, an expectation over everything), search for a **ratio, difference, or symmetry that removes it** before reaching for a sampling scheme or a bound. Exact cancellation beats good approximation, and it is astonishingly often available.

**3. Derive the functional form; don't assert it.**
Where does the exponential in the Boltzmann distribution come from? Energy levels as staircase steps; each hop up has probability *p*; independent hops multiply → p, p², pⁿ with n = ΔE/ε → an exponential in ΔE. The sigmoid then falls out of applying that to a two-state choice; it isn't chosen for being smooth and differentiable.

Any time you write softmax, sigmoid, exponential, or a squared term, you should be able to say what constraint forced it. If the honest answer is "convention," say so — that's a real finding about where the design is unexamined.

**4. Name the leftover the derivation didn't give you.**
The staircase argument yields only *relative* probabilities. Getting absolute ones requires summing over all states — and that leftover has a name, the partition function, and it becomes the central computational difficulty of the whole model family.

Never paper over the gap between what you derived and what you need. **The gap is usually the next named concept**, and it is usually where the field's hard problems live.

**5. Build the global objective from a local feeling.**
Each connection is "happy" or "unhappy" with its two units' states → happiness is `w_ij · x_i · x_j` → sum over connections → energy is the negative of that total. A global objective assembled from a per-part quantity you can reason about individually is far easier to get right, debug, and modify than the same objective handed down whole.

**6. Solve the single-item case exactly, then superpose.**
Storing *one* pattern makes the optimal weights obvious by inspection — set each weight to the product of its two states, and every connection is satisfied. That's the Hebbian rule, derived in one line on a case simple enough that it can't hide. Storing many is then just summing those weight matrices, and the reason it *works* (wells dug independently, far apart in state space, stay separate wells) is also the reason it *fails* at capacity.

The exact solution to N=1 usually reveals both the rule and its breaking point at once.

**7. Turn the capability into a concrete objective, and let structure emerge.**
"Generalize across environments" is not trainable. "Predict the next observation given the sequence of observations and actions" is. Then make the argument explicitly: because the observations come from a structured world rather than at random, the *optimal solution to the prediction problem* is to internally construct that structure. Nothing tells the model to build a map; the map is what minimizing the loss requires.

Whenever you want a property, ask what task makes that property the cheapest way to succeed — rather than adding a term to the loss that names it.

**8. Copy the decomposition of something that already solves it.**
Take the factorization from the working system, not just inspiration. The hippocampal *what* / *where* split becomes two modules: a position module that receives only actions and path-integrates, and a memory module that binds position to observation and supports retrieval from either half. The biology contributed the **interface between parts**, which is the hard part of architecture design.

**9. Frame a new model as N modifications to an understood one.**
"Two changes turn a Hopfield network into a Boltzmann machine: stochastic updates and hidden units." Name the modifications up front, then take them one at a time and follow each consequence — stochasticity forces a new objective, which forces a new learning rule, whose two terms then have a plain-language reading (raise data, lower dreams). This is worth more than a self-contained exposition, and it makes the ablation list write itself.

**10. Add a restriction and ask what it buys.**
Forbid visible-visible and hidden-hidden connections and you've obviously lost expressiveness — but gained the ability to update an entire layer in parallel, which is what made the model practical. Every constraint is a trade; if you can't state what a restriction buys computationally, it's probably the wrong restriction.

## Interrogating what you built

**11. Trace one concrete input end-to-end through the trained system.**
Not the training dynamics — one forward pass, narrated. John →(sister)→ Mary →(daughter)→ Kate →(uncle)→ ?, following exactly what each module holds at each step, ending with the retrieval that produces the answer. This catches interface errors that no equation review will, and it is the fastest way to find out whether you actually understand your own design.

**12. Choose the scaling axis on which the mechanism must show up.**
A lookup table needs every **edge** visited; a model that has inferred the structure needs only every **node**. Plotting accuracy against fraction-of-nodes-visited versus fraction-of-edges-visited *is* the evidence that structure was learned — aggregate accuracy alone would show almost nothing.

Pick the axis where your claimed mechanism predicts a different curve than the dumb baseline. On any other axis, a favorable comparison proves nothing while looking rigorous.

**13. Probe internal units, not just the loss.**
If you claim a system learned structure, show the structure inside it. Individual units in a position module trained only on next-step prediction develop hexagonal periodic firing patterns and directional stripes — grid cells and band cells, never hard-coded. Units in the memory module become place-like and remap across environments.

Aggregate metrics can't distinguish "learned the mechanism" from "learned a shortcut that scores the same." Internal structure can.

**14. Do the capacity and interference analysis.**
State the limit, the failure mode, and the mechanism of failure: capacity ≈ 0.14 × units; past it, retrieval returns blended states that were never stored; and correlated patterns interfere well *before* the nominal ceiling. Notice which of those three is the practically important one — here it's the third, because real data is always correlated. A model whose failure mode you can't describe isn't characterized.

**15. Change the input statistics and see what survives.**
Train on uniform random walks and you get grid cells. Train on trajectories that mimic a real animal — hugging walls, approaching objects — and boundary cells and object-vector cells appear that simply weren't there before. The representations are a function of the data distribution as much as the architecture. Perturbing the distribution is a sharper probe than perturbing the architecture, and much cheaper.

**16. Extract a falsifiable prediction about the real system.**
The strongest validation available. Because place-like units are conjunctions gated by grid-like units, remapping across environments should *not* be random — the overlap of place fields between two environments should correlate with the overlap of grid fields. That held in the model, and then held in real recordings.

If your model implies nothing checkable that you didn't already know, you may have a re-description rather than a mechanism.

**17. Keep the concept and the paper's notation separate.**
Conceptually there's one low-rank map from embedding space back to embedding space; in practice the papers factor it, staple all the up-projections into a single output matrix, and reserve the name "value matrix" for the down-projection alone. Both framings are correct and they will not line up when you read code against the paper.

State the clean object, state what the literature calls it, and say which one you are reasoning with right now. Most persistent confusion about an architecture is a notation collision, not a conceptual gap.

## Order of operations

1. Corner the problem (`derive-dont-declare`: naive solution killed by a number, ill-posed framing dissolved, distant isomorph imported, spec written).
2. Solve the smallest exact case (6) and read the free variables (1) — you now have your components.
3. Derive forms rather than asserting them (3), cancelling what you can (2), naming leftovers (4), building globals from locals (5).
4. Turn the capability into an objective (7); borrow a decomposition (8); frame as deltas (9); justify restrictions (10).
5. Trace one input end-to-end (11).
6. Test on the revealing axis (12), probe internals (13), find the capacity limit (14), perturb the data distribution (15).
7. Extract a prediction (16) and reconcile your notation with the field's (17).

## Anti-patterns

| Don't | Do |
|---|---|
| Assert softmax / sigmoid / an exponential. | Say what constraint forces that form. |
| Approximate an intractable term immediately. | First look for a ratio or symmetry that cancels it exactly. |
| Add a loss term named after the property you want. | Find the task where that property is the cheapest way to win. |
| Present a new model standalone. | Present it as N named modifications to an understood one. |
| Report aggregate accuracy. | Report the axis where the mechanism must diverge from the baseline. |
| "It learned a good representation." | Probe units and show what they became. |
| Stop at "it works." | Give capacity, failure mode, and what actually interferes. |
| Silently adopt the paper's variable names. | Say the clean object *and* the convention, and which you're using. |

## Provenance

Distilled from Artem Kirsanov's *Hopfield Networks* (`1WPJdAW-sFo`), *Boltzmann Machines* (`_bqa_I5hNAo`), and *Tolman-Eichenbaum Machine* (`cufOEzoVMVA`), and 3Blue1Brown's *Visualizing Attention* (`eMlx5fFNoYc`). All examples above are drawn from them.