The oracle that read the raster
Property-based testing through a terminal, and the blind spot in every grid read
Table of Contents
Summary
A property test fuzzed a REPL through a terminal and passed. It had silently skipped the hard cases: any output line longer than the terminal is wide wraps across rows, and an oracle that reads the screen row by row cannot see a wrapped record. The same blindness was then found in the production code that reads the same output. Neither bug was in the program under test; both were in the readers.
Two rules carry over to any UI-level property test:
- Assert on the model, not the raster. A view that reflows your data makes a blind oracle pass on exactly the inputs you meant to stress.
- Make the oracle report its coverage count. A green run that validated nothing looks identical to one that validated everything.
The setup
The program under test is a small Forth word, .state, that prints the interpreter's
stacks as one line of JSON for a browser view. It promises to be stack-neutral and to emit
decimal whatever BASE the session has set. The emitter has its own writeup; here it is
only the thing being read. A real frame from the run:
STATE:{"seq":114,"data":[66,1,2,3,350,0,0,0,0,1,2,3],"fpd":0,"rpd":136}
The engine is gforth 0.7.3 in a container capped at 512 MiB, two CPUs and 128 processes,
so a begin again or an allot bomb stays inside it. The property tester is
Bombadil's experimental terminal driver,
running on the host and driving the container over a pseudo-terminal.
The default input generator types random Unicode. That run reached 3,747 states, all of
them bounced off gforth's "Undefined word" handler, and never once spelled .state, so it
tested the REPL's error recovery and not the emitter. The working generator uses a Forth
alphabet: stack operators, hex, decimal and 2 base !, colon redefinitions,
error-provoking words, float-stack noise and real .state calls, weighted so about a third
of lines sample the emitter on whatever stack and base the other lines have built. The
oracle: every STATE:{...} on screen must parse as JSON of shape
{seq:int, data:int[], fpd:int, rpd:int}.
The emitter held: 15,782 frames validated, none malformed, stacks up to 126 cells deep.
A green run that skipped the hard cases
The first oracle passed too, and it was wrong.
A terminal grid is a raster. The obvious oracle joins the rows with newlines and matches
one frame per line, STATE:(\{[^\n]*?\}). That pattern cannot cross a newline. At 120
columns, a stack of more than about 25 cells makes a frame longer than a row; the terminal
wraps it, the join puts a newline inside the data array, and neither half matches. The
frame is skipped without a failure.
The frames that wrap are the deep-stack frames, which are the cases the experiment existed to stress. Making the oracle wrap-aware brought 2,944 previously skipped frames under test, up to the 126-cell stack, still with none malformed. The fix has three parts:
- match across rows but stop at the next record boundary,
\{[^{}]*\}, so a half-drawn frame is skipped rather than failed; - strip the wrap newlines before parsing;
- count what was validated, wrapped frames separately.
Widening the terminal is the wrong fix. The stack is unbounded, so any fixed width wraps eventually; a wider viewport moves the blind spot without removing it.
When the UI under test reflows what you care about, the property must read the UI's model, not its rendering.
A row-indexed grid read inherits wrapping as if it were content. An HTML-render read
inherits layout and truncation; a screenshot read inherits anti-aliasing. Where a
model-level signal exists, such as the program's own stdout, a structured event, or a DOM
node's textContent, assert on that. Where only the rendered view exists, invert the
rendering before judging it.
The same bug in production
The wrap-blindness was not confined to the test. Re-running the same adversarial input set
through the live runner, a shell script that drives a real gforth pane in tmux, found the
identical bug in shipping code. The runner read frames with grep over
tmux capture-pane -p, which returns the pane as displayed rows. A stack deep enough to
wrap the pane (checked with a 70-cell frame) was missed, and the runner silently returned
the previous, stale frame.
The fix is the runner's version of the oracle's: capture-pane -J, which joins wrapped
lines before the match 1. The emitter had printed the deep frame correctly the
whole time. Two independent readers, the test oracle and the production runner, were blind
to it in the same way, and the test found it first only because it pushed deeper stacks
than any person had typed.
The same pass found an unrelated runner bug. tmux send-keys -l "$line" without --
parsed any line starting with a dash, such as -9, as a tmux option and aborted. The fix
is the usual one for shelling out: put -- before the argument.
The vacuous pass
The random-Unicode run and the first wrap-blind run share a failure that the word "pass" hides. A property run can pass because the generator never produced what the oracle checks for. The Unicode run validated zero frames and exited green. "No counterexample found" and "the property holds" print the same line and are different facts.
The defence is a count. The oracle reports how many records it validated: not pass or fail, but 15,782 frames, of which 2,944 wrapped. A property checked zero times is not a gate, and only the count distinguishes the two.
Reproducing
The experiment is experiments/017-pbt-forth in the sicp-minus-sicp repository (commit
9615751), with the runner fixes in ec18fb7. gmake pbt rebuilds the container and runs
the property. Results are from 2026-11-15, gforth 0.7.3, Podman 5.4.2.
Footnotes:
tmux(1), capture-pane: "-J preserves trailing spaces and joins any wrapped
lines". https://man.openbsd.org/tmux.1