Bombadil for SPAs: Property-Based Testing Best Practices
Table of Contents
1. Overview
Bombadil (Antithesis, successor to
Quickstrom) tests a web UI by exploring it like a fuzzer and checking LTL
properties – always, eventually, now ... implies ... within – against
state extracted from the live DOM. This note tracks what actually works when
the target is a single-page app, using wal.sh's own pocket-es search
surface as the running example. It is a living document; entries are added as
the eldest-spec suite evolves.
The recurring theme: a property-based suite is only as good as its extractors. Most "violations" you hit early are refutations of the spec, not the site. 8.1 tests that theme against LLM-inferred properties compiled to Bombadil.
2. Installation and environment
- The npm package bundles the Rust binary for three platforms.
@antithesishq/bombadilships prebuilt binaries forlinux-x64,linux-arm64anddarwin-arm64underbinaries/, dispatched by abin/bombadil.jsshim –npm installalone gives you a workingbombadilon those targets. Outside them (FreeBSD, Windows,darwin-x64) there is no binary in the package; install separately (release assetbombadil-<arch>-<os>,cargo, or Nix) and keep it onPATH. - Bombadil manages its own Chromium.
bombadil testresolveschromiumonPATH. If you only have Google Chrome, point achromiumshim at it. On a CI runner, installchromium-browserexplicitly. - Drive an existing browser with
test-external. When the managed-Chromium path is broken or you want to watch the run, launch Chrome with--remote-debugging-port=9992and usebombadil test-external --remote-debugger http://localhost:9992 --create-target.
3. SPA-specific behavior
- The URL stays constant; exploration happens in-page. For a client-side
surface like pocket-es, the whole run can sit on
/search/– Bombadil types into inputs and clicks results without a navigation. Do not assume the trace will show many distinct URLs; assert on DOM state, not on route changes. - Use the origin as a boundary. The first argument to
testis both the start URL and the boundary – Bombadil will not navigate off it. Point it at the exact surface you mean to exercise. - Prefer quiescence over fixed waits. Bombadil 0.5.0 replaced fixed timeouts
with quiescence timers (#176), which settles SPA re-renders far better. Bound
the run with
--time-limitrather than a step count. - Watch for navigation stalls. A single slow or never-settling route can trip
navigation timed out ... during Loading. Keep the boundary tight and the time limit modest while iterating.
4. Extractors must match the real DOM
Verify every selector against the live DOM in a real browser before trusting a refutation. Three false positives from the wal.sh suite, all spec bugs:
- Over-broad selectors capture the wrong element.
ul:first-of-type > li > amatched the page table-of-contents, not<nav>, so the nav check was polluted with article titles. Scope tonav a. - Encode the site's actual structure, not an assumed one. The conjecture assumed a "Home" nav item; wal.sh has Research/Events/Current/Search and a wordmark that is the home link. A property asserting something the site never had fails forever and teaches you nothing.
- Logos are not always images.
a:has(img[alt*"wal.sh"])= matched nothing – the home link is an<a class"wordmark">= text logo. The extractor silently returnednull.
5. Headless vs. real-browser divergence
- Cross-origin resources flake under headless. An external badge
(
static.fsf.org/...) reportednaturalWidth ==0= headless but loaded fine (182×45) in a real browser. A "no broken images" property fired on a network artifact, not a defect. Restrict such checks to same-origin resources, or wait for load before asserting. - Always confirm a refutation in a headed browser before filing it as a site bug. The cheapest debugging step is opening the witness URL yourself.
6. Guarding extractors against their own crashes
An extractor that throws aborts the entire run, costing you every other property. Defend the DOM calls:
querySelector(href)wherehref =="#"= throws (SyntaxError: '#' is not a valid selector). Exclude bare fragments (a[href^"#"]:not([href="#"])=) and wrap the lookup intry/catch.- Treat any malformed-input path as "no result," never as an exception.
7. Determinism and reproduction
- Bombadil 0.5.0 has no
--seedflag. Determinism comes from--reproduce <TRACE_FILE>, which replays a recorded trace exactly (#177). Do not design a seed-keyed reproduction scheme around a flag that does not exist. - Key artifacts to the build. Write traces and screenshots under
runs/<build-sha>/so a refutation is tied to the commit that produced it. The traceruns/<sha>/trace.jsonlplus--reproduceis the witness.
8. Checklist
- Binary on
PATH(bombadil --version); Chromium resolvable. tsc --noEmitclean (addDOM.Iterabletolib;skipLibCheckfor upstream type bugs).- Every selector verified against the live DOM in a real browser.
- Same-origin guards on resource-loading properties.
- Extractors wrapped against
querySelector/ parse exceptions. - Run bounded by
--time-limit, output underruns/<sha>/. - Each refutation reproduced with
--reproducebefore it is believed.
8.1. Inferred properties: SPINACH
Everything above assumes a person writes the properties. SPINACH, presented at SpecOps 2026 by Savitha Ravi and Michael Coblenz (Ravi and Coblenz 2026), asks whether an LLM can write them instead, and targets Bombadil as its backend. That makes it a direct test of this note's recurring theme: a suite is only as good as its extractors, and most early violations refute the spec.
8.1.1. What SPINACH does
SPINACH is a VSCode extension with a three-phase pipeline, each phase gated by developer review.
- Concept model. An LLM agent crawls the running application and decomposes
it into concepts in Daniel Jackson's sense (Jackson 2021):
units such as
User,Articleorfollow, each with a purpose, an operational principle, state fields, actions and invariants, plus synchronizations that tie concepts together. The agent may infer actions it never reached: atrashconcept implies a "remove from trash" action whether or not exploration saw the button. - Natural-language properties. A second agent derives properties per concept and synchronization. It is deliberately not told it is writing property-based tests; the authors report that framing biased output toward "very mathematical, but inconsequential" properties.
- LTL and Bombadil. A final phase formalizes each approved sentence into LTL where it can, then emits a Bombadil test.
The paper's worked example: "a user must be logged in to follow another"
becomes \(G(\mathit{following} \rightarrow \mathit{authenticated})\). The
generated Bombadil test is an always over two extractors: when the viewer is
unauthenticated and on a profile page, the follow button's label must not
start with "Unfollow".
8.1.2. What the evaluation found
Two open-source applications, both previously used to evaluate GUI-testing tools.
| RealWorld (Angular Medium clone) | 4ga Boards (Kanban) | |
|---|---|---|
| concepts | 14 | 23 |
| dependencies between concepts | 9 | not reported |
| synchronizations | 2 | not reported |
| natural-language specifications | 94 | 161 |
| share on authentication/authorization | over half | 40% |
Very few specifications were fully formalized in LTL. Most of the remainder were entity-relationship constraints ("every card belongs to exactly one board") that need first-order quantification, which propositional LTL lacks. Accuracy was assessed on a sample of five specifications, judged by the first author and then discussed with the second; the paper reports no score.
The generated tests compiled. They found no bugs in either application. The authors attribute this to Bombadil's execution model rather than to the properties: actions are chosen by weighted random selection, so states behind a login, a navigation and a form fill are rarely reached, and an LLM-generated selector that misses the real DOM never fires at all.
Everything else is planned: mutation testing for RQ1 against two baselines (crawl then PBT directly; crawl then EARS requirements then PBT, the Kiro route), running tests across real version histories as a less synthetic fault source, an ablation that strips concepts down to purpose and operational principle, and a user study for RQ2 on legibility and maintenance. Proposed fixes: replace random action selection with constrained trace generators that synthesize directed interaction sequences (Zhou et al. 2026), and reconsider LTL itself, citing evidence that non-specialists misread it (Greenman et al. 2023).
8.1.3. Against Bombadil: written versus inferred
Bombadil, and Quickstrom before it (O’Connor and Wickström 2022), gives the author LTL over browser state and assumes the author knows which properties matter. SPINACH keeps that checker and replaces the author. The interesting result is what the inferred properties look like once checked.
- They are state invariants. The worked example is a
single \(G\) over a propositional implication. Nothing in the paper exercises
eventuallyorwithin, the operators that make Bombadil more than an assertion library. Conjecture: concept invariants, by construction, compile to safety properties of one state, so the temporal half of the language goes unused. - The extractor problem moves, it does not go away. The worked test returns
truewhen the follow-button label isnull. If the selector is wrong, the property passes on every state. This is the null-returning logo extractor from 4, generated rather than written, and it fails in the quiet direction. The paper names "imperfect selectors" as one cause of zero findings; this note's checklist item 3 (verify each selector against the live DOM) applies unchanged to generated tests. - Exploration is the bottleneck in both. The navigation stalls and quiescence notes in 3 are the hand-written version of the reachability limit the paper hits. Neither gets better by improving the properties.
8.1.4. Against the surfaces framing
Context Surfaces separates context a system emits from context a worker elicits, and files property-based driving under the second. SPINACH's crawl reads the DOM surface, but its properties come from two places, and only one of them is the surface.
The concept model's own integrity notes say a design choice "is the standard design for this class of app." That is the model's prior, not anything the page showed. The trash example is the same move: a state inferred because apps of this kind usually have it. Inferring past the surface is how SPINACH finds properties nobody exercised; it is also how a property comes to assert a "Home" link the site never had, the second false positive recorded above. The paper's developer-review gates are where that gets caught, so the review step is the mechanism that keeps the inference honest, not an add-on.
Some of the properties are not on the surface at all. "Email must be unique across all users" cannot be observed from one browser session; one user's DOM never shows every other user's email. Conjecture: part of the paper's formalization gap is a surface gap. These are database-surface properties reached through a DOM-surface tool, and no choice of logic fixes that.
8.1.5. Against the PBT discipline notes
- Calibration. Zero bugs found on two apps is an uncalibrated result. Until the suite has run against a known-bad build and gone red, a green run says nothing about the apps; the paper's planned mutation and version-history runs are exactly the calibration missing so far. The site's rule (a gate never run against a known-bad input is uncalibrated) applies directly, and the authors make the same point in their own terms.
- Oracles. The crowsnest rig checks the receiver against an independent model that re-derives expected state from the spec after every step. A Jackson concept already has state, actions and invariants, which is most of such a model. Conjecture: an executable concept, stepped alongside the browser, would be a stronger oracle than the per-state LTL sentences, and it would cover the entity-relationship constraints that do not fit propositional LTL.
- Shrinking and pin-on-find. With no violation, there was nothing to shrink
or pin. When one fires, the shrink discipline
carries over with Bombadil's substitutes: the recorded trace and
--reproducereplace the seed, and the minimal trace becomes the pinned regression. Nothing in SPINACH changes that step. - Where it fits. As a generator of candidate properties for a human to review, SPINACH matches what these notes argue: the 94 and 161 sentences are a reading list for whoever writes the suite, and the share on authentication suggests where the model's attention goes. As an oracle that runs unattended, it does not yet fit: the properties are uncalibrated, the extractors are unverified, and the exploration that would exercise them does not reach the states they constrain.
Daikon infers likely invariants from observed traces (Ernst et al. 2001); SPINACH infers them from a model of what the application should be. Conjecture: the failure modes mirror each other. Trace mining fits too closely to what happened; prior-driven inference fits too closely to what usually happens in apps of this class. The first misses properties the traces never violated, the second asserts properties the site never had. The review gate exists for the second.
8.1.6. References
Seen at
- BugBash 2026 — Oskar Wickström's talk, Old Tom Bombadil is a merry fuzzer!.