Auditing your implementation against the textbook, one definition at a time

Here is a category of bug that testing does not find.

You implement twenty variants of something — twenty option structures, twenty estimators, twenty transforms — from a table in a README. Each one runs. Each produces output with the right shape and a plausible magnitude. Downstream code consumes them uniformly, so nothing crashes. Your tests check that a structure with four legs has four legs, that prices are positive, that P&L arithmetic is consistent.

And one of them is not the thing it is named after.

No test catches this, because the tests were written from the same misunderstanding as the code. The output is not malformed; it is well-formed and mislabelled. Everything downstream treats it as what its name says, and the error surfaces only as a scoring system that keeps preferring a structure whose risk profile is not what the scorer thinks it is.

The format that works

The only reliable method I have found is unglamorous. For each item, produce four things, side by side, in a document:

1. What we claim it is — the one-line description from the README or the docs.
2. What the canonical definition says — the construction as a standard reference gives it. Name the reference.
3. What the code actually builds — read the construction function and write down, in plain terms, the object it emits. Cite the file and the lines.
4. A verdict — correct, correct with minor concerns, partially correct, or incorrect. Plus, where it is not correct, the specific change, written for a programmer who is not a domain expert.

That last qualification carries more weight than it looks. "Fix the strike selection to be delta-based" is an instruction only a specialist can execute. "Select the call whose delta is nearest to +0.16, then clamp the chosen strike to within six strike-steps of at-the-money" is executable by anyone, and the act of writing it that precisely is itself a check — if you cannot write the instruction that way, you have not finished understanding the problem.

The claim I will defend: writing down the definition and the implementation as two separate statements, and then comparing them deliberately, is a categorically different activity from reading the code and asking whether it looks right. Reading code invites you to reconstruct the intent from the implementation, which means you evaluate the code against itself. Writing the definition first, from an external source, gives you something the code can disagree with. It is the same reason a pre-registered analysis plan is worth more than the same analysis chosen after seeing the data.

What it surfaces

Three kinds of finding, in rough order of how often they show up.

Silent parameter degeneracy. A construction that is supposed to pick strikes by some criterion picks them by a fallback when the criterion cannot be met, and the fallback produces something structurally different. Nothing errors. The object is well-formed and is now a different structure with a different risk profile than the one being scored.

Gates that do not match the risk. Every structure has conditions it should refuse — a short-premium position wants a floor on time to expiry, a defined-risk one might not. When these gates are set by copy-paste from a similar-looking neighbour, you get a structure whose admission rules belong to a different risk profile. This is invisible in output because the gate only fires sometimes, and when it fails to fire you simply get a trade you should not have had.

Scorers rewarding the wrong quantity. A scoring function that adds points for a quantity that the structure is actually hurt by. These are the hardest to catch by inspection, because the scorer's code is short, readable, and wrong only if you know the structure's Greek profile — which is exactly the knowledge the audit is forcing you to write down.

The audit also finds the opposite, which is worth saying: most of the items usually turn out to be correct, and confirming that is not wasted work. It converts "we think these are right" into "these were checked against a named reference on a stated date," which is a different asset — one that survives the author leaving and can be re-run against a changed codebase.

Why it has to be written down

The tempting shortcut is to do this in your head while reading the code. It does not work, for a reason that is not about diligence.

When you read an implementation, you build a model of what it is trying to do from the implementation. That model is contaminated by the code, and it will accommodate the code's mistakes, because a mistake presents as an intention you had not considered. Writing the canonical definition first — ideally from a source you did not write — is what makes the comparison possible at all.

The document also does work that the check itself does not. It tells the next person which of twenty items were verified, against what, and when. Without it, the natural half-life of "we audited that" is about one quarter, after which it reverts to folklore.

And there is a version-control benefit that surprised me: a dated audit document diffs. Re-run the audit six months later, and the changed verdicts are the review. That is a much cheaper artefact to maintain than a test suite that would have to encode all the same domain knowledge in assertions.

The limits

An audit against textbook definitions verifies that you built the thing you named. It does not verify that the thing is useful, that the scoring logic ranks well, or that any of it makes money. It is a correctness check against a definition, and definitions can be right while everything built on them is unprofitable.

It also inherits the reference's authority. Where standard sources genuinely disagree — and for some constructions they do, particularly on strike selection conventions — the audit can only record the disagreement and state which convention was adopted. That is still progress: an explicit choice beats an accidental one. But it is not the same as being right, and a document that hides a contested convention behind a confident "CORRECT" is worse than no document.

The general form: any system with many similar-looking components accumulates ones that are not what they claim to be, and the failure is silent because the type system is satisfied. Names are not checked by the compiler. Periodically, somebody has to sit down with the definitions and confirm that the twenty things named after twenty concepts are, in fact, those concepts.