Live Mutants: A Driver, an Observer, a Correlation Id and a Checker
Change one call site while the program runs, and know whether the tests noticed, and why
Table of Contents
Where the work happens
Getting Things Done sorts tasks by context: what you can do at a phone, at a computer, on an errand. Agent work sorts the same way, and the context decides what the work can produce.
| context | good for | produces |
|---|---|---|
| phone, voice | deciding, naming, asking the next question | chat; a decision to act on later |
| web chat | drafting, comparing, explaining | chat, documents, PDFs, a page to share |
| Claude Code in a repository | building and checking | files, runnable code, tests, commits, a spec scaffold |
| a running process (REPL, debugger, browser) | observing and changing the system while it runs | evidence: logs, traces, screenshots, a verdict |
The last row is the subject of this note. The first three make the spec; the last one checks whether the spec and the code agree while something real is happening. A session moves down the table and back up: a question asked by voice becomes a spec in a repository, which becomes a run against a live process, whose evidence goes back up as a decision.
When to stop and reflect
For specs and builds, the work is easy to start and hard to stop well. Four moments came up in this round, each a point to pause before building more:
- Reseal the spec when a run teaches you a rule. The contract below gained rule C1 from a bug in the test harness, not in the code, and rule C5 from a mutant that did more than the fault it stood for. Writing them down is what lets three other languages get them right first time.
- Make a clean baseline, or a factory, before measuring anything. A namespace with no planted defects means any failure under a mutant is the mutant's doing.
- Prepare for change at named versions. A plugin's
versionis the only thing that moves an existing install; a mutant's apply time is the only thing that separates a kill from a false alarm. Both are named points in time that later evidence is measured against. - Reflect when the instruments disagree. When the probe said "never changed anything" and a property killed the mutant anyway, the right move was to distrust the probe.
The four parts
A live mutant is a function redefined in a running process. To learn anything from one, four parts have to be named:
| part | job |
|---|---|
| driver | puts the change in, at a chosen moment, and records when |
| observer | at the redefined call site, runs the original and the mutant on the same arguments; counts calls, and calls where they differed; returns the mutant's value |
| correlation id | a key carried by the driver's record, every observer entry and every verdict, so they can be joined afterwards |
| checker | decides pass or fail, and never reads the observer |
Joined, they give each mutant one of four outcomes, the cells of the RIPR model (reach, infection, propagation, revealability): never reached; reached but never changed an output; changed outputs but no check noticed; killed. A property-based tester on its own reports only the last.
This is also what context surfaces lacked: a surface says what can be read from a running system, and nothing about what moves it. Every new surface added to an agent's context needs the same four named.
Rung 2: a defect-free baseline under test.check
The baseline is about a hundred lines of plain Clojure with the classic property shapes:
run-length encoding (round trip, maximal runs), merge sort (against sort, idempotent,
stable), binary search (against a linear scan), a sorted-vector set and a two-list queue (each
against a model, with its invariant). All eight properties hold.
The observer is a probe that replaces one var:
(defn probe [original mutant log]
(fn [& args]
(if *shadow*
(apply original args) ; the original's own recursive calls skip the probe
(let [expected (binding [*shadow* true] (apply original args))
actual (apply mutant args)]
(swap! log #(-> % (update :calls inc)
(update :infected + (if (same? expected actual) 0 1))))
actual))))
Each mutant is swapped in with with-redefs-fn (what alter-var-root does from a REPL), and
every property runs at a fixed seed. Six mutants, seeds 42, 1 and 7, the same result at each:
| mutant | change | calls | infected | outcome | killed by |
|---|---|---|---|---|---|
| merge-ties-right | <= to < on ties |
36,154 | 4 | killed | merge-sort-stable |
| merge-sort-pairs-unsorted | base case < 2 to < 3 |
533 | 108 | killed | matches sort, idempotent, stable |
| bsearch-skips-last | upper bound starts one short | 113 | 13 | killed | bsearch agrees with scan, set model |
| rle-runs-of-two | runs stop growing at 2 | 266 | 62 | killed | runs are maximal (not the round trip) |
| insert-le | neg? to < 0= in the split |
3,005 | 0 | reached, never infected | none: equivalent here |
| queue-pop-no-rebalance | pop forgets to rebalance | 22 | 6 | killed | queue matches vector model |
Counts are from seed 42. Two rows say more than "killed". rle-runs-of-two still decodes
correctly, so the round trip, the property everyone writes first, cannot see it; only the
invariant on the encoding can. And insert-le was reached three thousand times without once
changing an output: no property should kill it, and a mutation score that counts it as a
survivor is wrong about the tests.
The probe was wrong first
On the first run, merge-ties-right came back "reached, never infected" while
merge-sort-stable killed it. Both cannot be true. The probe compared outputs with
Clojure's =, which ignores metadata, and the stability property tagged each element's
original position in metadata. The checker could see a difference the observer could not.
The fix is one function; the rule is the finding:
The observer's equality must be no coarser than the checker's.
The same day, a game harness writing into a live umoria process through lldb found the
same weakness on its side: its save round trip compared what the game displays, which is
coarser than the item record, so a corrupted field that never displays would pass.
Rung 4: the same probe under a running explorer
The page is a small shop whose catalogue changes underneath it: a data driver reprices pets, sells them out and puts them on backorder on a schedule, and the page has to keep up within 2,000 ms (Bombadil for SPAs covers the explorer). Bombadil explores it for 75 s with 18 temporal laws. Thirty seconds in, one store function in the running server is redefined through a loopback-only route, and the moment it went in is recorded. A reset route undoes it, on every exit path.
| mutant | first changed output | first violation | detection | killed by |
|---|---|---|---|---|
| none | none | |||
| silent-reprice | +348 ms | +368 ms | 20 ms | charged as quoted; server invariants |
| stale-version | +1,469 ms | +1,662 ms | 193 ms | version faithful; server invariants |
| backorder-as-201 | +10,280 ms | +10,502 ms | 222 ms | server invariants |
No violation came before a mutant went in, and the clean run had none. The killers match those
of the same faults set by a switch at start. The new column is the first one: because the
probe records when the mutated call first made a difference, time-to-kill splits into reach
(waiting for the program to exercise the change) and detection (the laws noticing). Detection
was 20 to 222 ms. backorder-as-201 took ten seconds because an order has to land on a
backordered pet before the mutant does anything, the same ten seconds it took as a switch.
silent-reprice, frame by frame. Before the mutant, the catalogue is at v49:
368 ms after it, the first violation: the cart still quotes Waffles at $29.99 while the catalogue says $30.99, totals $290.85, and checkout is in flight. A correct store answers 409, stale price:
The next state: "Order #33 placed: $294.85". Four dollars more than the quote, four Waffles at a dollar each, reported to the shopper as success:
A mutant that did too much
The first silent-reprice returned "no objection" for a stale price. That also skipped the
stock check that came after it, so the unit test showed it killing the oversell invariants as
well. A live mutant changes one decision and lets the rest of the function run; this one now
re-checks the line at the current price.
Code, not state
A live change goes in at one of two places. Redefining code keeps every state the run reaches
one the program built itself; only a transition is wrong, so a counterexample is a finding.
Writing state (memory through a debugger, a database row, the DOM) can start the run from a
state the program could never reach, and a counterexample from there is a bug in the harness.
The umoria harness's crashes were mostly of that kind: a player outside his own screen panel,
an over-full pack, an uninitialised item. Its fix is the one the shop's data driver already
follows: change the world only through the program's own transitions.
The same rung in other stacks
The contract is language-neutral; what changes is how a call site is redefined while the process runs.
| stack | PBT runner | live redefinition | status |
|---|---|---|---|
| Clojure | test.check | with-redefs / alter-var-root over nREPL |
rungs 2 and 4 run |
| TypeScript | fast-check 4.10 | a property swapped on the module object, restored in finally |
rung 2 run: same outcomes and killers at all three seeds |
| Python | Hypothesis 6.170 | unittest.mock.patch on module attributes, restored in finally |
rung 2 run: same outcomes and killers at all three seeds |
| Ruby | prop_check 1.0.2 | define_singleton_method, removed in ensure so lookup falls back to the original |
rung 2 run: same outcomes and killers at all three seeds |
The TypeScript port made a calls count about ten times lower (fast-check's default arrays stop
at ten elements; test.check's grow) and the same outcome in every cell. The Ruby port's shrinker
barely shortens arrays, so its witnesses are longer (eleven zeros for rle-runs-of-two, not
three fours), and the verdicts are the same. Counts are a property
of the generator; the cells and the killers are a property of the tests.
The contract
- C1. The observer's equality is no coarser than the checker's.
- C2. Redefine code; don't write state, or check the program's invariants on any state you write.
- C3. Restore on every exit path, and have the reset path undo every redefinition.
- C4. One mutant at a time.
- C5. A mutant changes one decision and lets the rest run.
- C6. The checker never reads the observer.
Next
- Apply times of 10 s and 50 s, twice each, before the detection lags are more than one sample.
- The browser-side version: a page function patched over the DevTools protocol while Bombadil drives the same Chrome.
- Rung 4 in the other stacks, behind Express, Django and Rails versions of the shop.