Live Mutants: A Driver, an Observer, a Correlation Id and a Checker
Change one call site while the program runs, and know whether the tests noticed, and why

Table of Contents

Where the work happens

Getting Things Done sorts tasks by context: what you can do at a phone, at a computer, on an errand. Agent work sorts the same way, and the context decides what the work can produce.

context good for produces
phone, voice deciding, naming, asking the next question chat; a decision to act on later
web chat drafting, comparing, explaining chat, documents, PDFs, a page to share
Claude Code in a repository building and checking files, runnable code, tests, commits, a spec scaffold
a running process (REPL, debugger, browser) observing and changing the system while it runs evidence: logs, traces, screenshots, a verdict

The last row is the subject of this note. The first three make the spec; the last one checks whether the spec and the code agree while something real is happening. A session moves down the table and back up: a question asked by voice becomes a spec in a repository, which becomes a run against a live process, whose evidence goes back up as a decision.

When to stop and reflect

For specs and builds, the work is easy to start and hard to stop well. Four moments came up in this round, each a point to pause before building more:

  • Reseal the spec when a run teaches you a rule. The contract below gained rule C1 from a bug in the test harness, not in the code, and rule C5 from a mutant that did more than the fault it stood for. Writing them down is what lets three other languages get them right first time.
  • Make a clean baseline, or a factory, before measuring anything. A namespace with no planted defects means any failure under a mutant is the mutant's doing.
  • Prepare for change at named versions. A plugin's version is the only thing that moves an existing install; a mutant's apply time is the only thing that separates a kill from a false alarm. Both are named points in time that later evidence is measured against.
  • Reflect when the instruments disagree. When the probe said "never changed anything" and a property killed the mutant anyway, the right move was to distrust the probe.

The four parts

A live mutant is a function redefined in a running process. To learn anything from one, four parts have to be named:

part job
driver puts the change in, at a chosen moment, and records when
observer at the redefined call site, runs the original and the mutant on the same arguments; counts calls, and calls where they differed; returns the mutant's value
correlation id a key carried by the driver's record, every observer entry and every verdict, so they can be joined afterwards
checker decides pass or fail, and never reads the observer

Joined, they give each mutant one of four outcomes, the cells of the RIPR model (reach, infection, propagation, revealability): never reached; reached but never changed an output; changed outputs but no check noticed; killed. A property-based tester on its own reports only the last.

This is also what context surfaces lacked: a surface says what can be read from a running system, and nothing about what moves it. Every new surface added to an agent's context needs the same four named.

Rung 2: a defect-free baseline under test.check

The baseline is about a hundred lines of plain Clojure with the classic property shapes: run-length encoding (round trip, maximal runs), merge sort (against sort, idempotent, stable), binary search (against a linear scan), a sorted-vector set and a two-list queue (each against a model, with its invariant). All eight properties hold.

The observer is a probe that replaces one var:

(defn probe [original mutant log]
  (fn [& args]
    (if *shadow*
      (apply original args)                 ; the original's own recursive calls skip the probe
      (let [expected (binding [*shadow* true] (apply original args))
            actual   (apply mutant args)]
        (swap! log #(-> % (update :calls inc)
                          (update :infected + (if (same? expected actual) 0 1))))
        actual))))

Each mutant is swapped in with with-redefs-fn (what alter-var-root does from a REPL), and every property runs at a fixed seed. Six mutants, seeds 42, 1 and 7, the same result at each:

mutant change calls infected outcome killed by
merge-ties-right <= to < on ties 36,154 4 killed merge-sort-stable
merge-sort-pairs-unsorted base case < 2 to < 3 533 108 killed matches sort, idempotent, stable
bsearch-skips-last upper bound starts one short 113 13 killed bsearch agrees with scan, set model
rle-runs-of-two runs stop growing at 2 266 62 killed runs are maximal (not the round trip)
insert-le neg? to < 0= in the split 3,005 0 reached, never infected none: equivalent here
queue-pop-no-rebalance pop forgets to rebalance 22 6 killed queue matches vector model

Counts are from seed 42. Two rows say more than "killed". rle-runs-of-two still decodes correctly, so the round trip, the property everyone writes first, cannot see it; only the invariant on the encoding can. And insert-le was reached three thousand times without once changing an output: no property should kill it, and a mutation score that counts it as a survivor is wrong about the tests.

The probe was wrong first

On the first run, merge-ties-right came back "reached, never infected" while merge-sort-stable killed it. Both cannot be true. The probe compared outputs with Clojure's =, which ignores metadata, and the stability property tagged each element's original position in metadata. The checker could see a difference the observer could not.

The fix is one function; the rule is the finding:

The observer's equality must be no coarser than the checker's.

The same day, a game harness writing into a live umoria process through lldb found the same weakness on its side: its save round trip compared what the game displays, which is coarser than the item record, so a corrupted field that never displays would pass.

Rung 4: the same probe under a running explorer

The page is a small shop whose catalogue changes underneath it: a data driver reprices pets, sells them out and puts them on backorder on a schedule, and the page has to keep up within 2,000 ms (Bombadil for SPAs covers the explorer). Bombadil explores it for 75 s with 18 temporal laws. Thirty seconds in, one store function in the running server is redefined through a loopback-only route, and the moment it went in is recorded. A reset route undoes it, on every exit path.

mutant first changed output first violation detection killed by
none   none    
silent-reprice +348 ms +368 ms 20 ms charged as quoted; server invariants
stale-version +1,469 ms +1,662 ms 193 ms version faithful; server invariants
backorder-as-201 +10,280 ms +10,502 ms 222 ms server invariants

No violation came before a mutant went in, and the clean run had none. The killers match those of the same faults set by a switch at start. The new column is the first one: because the probe records when the mutated call first made a difference, time-to-kill splits into reach (waiting for the program to exercise the change) and detection (the laws noticing). Detection was 20 to 222 ms. backorder-as-201 took ten seconds because an order has to land on a backordered pet before the mutant does anything, the same ten seconds it took as a switch.

silent-reprice, frame by frame. Before the mutant, the catalogue is at v49:

The shop before the mutant: six pets, a cart of Pickles, Gravy and Waffles at 29.99, catalogue v49

368 ms after it, the first violation: the cart still quotes Waffles at $29.99 while the catalogue says $30.99, totals $290.85, and checkout is in flight. A correct store answers 409, stale price:

The first violating state: cart total 290.85 with Waffles quoted at 29.99, catalogue shows 30.99, checkout in flight

The next state: "Order #33 placed: $294.85". Four dollars more than the quote, four Waffles at a dollar each, reported to the shopper as success:

The next state: Order 33 placed for 294.85, four dollars more than the 290.85 quoted

A mutant that did too much

The first silent-reprice returned "no objection" for a stale price. That also skipped the stock check that came after it, so the unit test showed it killing the oversell invariants as well. A live mutant changes one decision and lets the rest of the function run; this one now re-checks the line at the current price.

Code, not state

A live change goes in at one of two places. Redefining code keeps every state the run reaches one the program built itself; only a transition is wrong, so a counterexample is a finding. Writing state (memory through a debugger, a database row, the DOM) can start the run from a state the program could never reach, and a counterexample from there is a bug in the harness. The umoria harness's crashes were mostly of that kind: a player outside his own screen panel, an over-full pack, an uninitialised item. Its fix is the one the shop's data driver already follows: change the world only through the program's own transitions.

The same rung in other stacks

The contract is language-neutral; what changes is how a call site is redefined while the process runs.

stack PBT runner live redefinition status
Clojure test.check with-redefs / alter-var-root over nREPL rungs 2 and 4 run
TypeScript fast-check 4.10 a property swapped on the module object, restored in finally rung 2 run: same outcomes and killers at all three seeds
Python Hypothesis 6.170 unittest.mock.patch on module attributes, restored in finally rung 2 run: same outcomes and killers at all three seeds
Ruby prop_check 1.0.2 define_singleton_method, removed in ensure so lookup falls back to the original rung 2 run: same outcomes and killers at all three seeds

The TypeScript port made a calls count about ten times lower (fast-check's default arrays stop at ten elements; test.check's grow) and the same outcome in every cell. The Ruby port's shrinker barely shortens arrays, so its witnesses are longer (eleven zeros for rle-runs-of-two, not three fours), and the verdicts are the same. Counts are a property of the generator; the cells and the killers are a property of the tests.

The contract

  • C1. The observer's equality is no coarser than the checker's.
  • C2. Redefine code; don't write state, or check the program's invariants on any state you write.
  • C3. Restore on every exit path, and have the reset path undo every redefinition.
  • C4. One mutant at a time.
  • C5. A mutant changes one decision and lets the rest run.
  • C6. The checker never reads the observer.

Next

  • Apply times of 10 s and 50 s, twice each, before the detection lags are more than one sample.
  • The browser-side version: a page function patched over the DevTools protocol while Bombadil drives the same Chrome.
  • Rung 4 in the other stacks, behind Express, Django and Rails versions of the shop.