Bombadil for SPAs: Property-Based Testing Best Practices

Table of Contents

1. Overview

Bombadil (Antithesis, successor to Quickstrom) tests a web UI by exploring it like a fuzzer and checking LTL properties – always, eventually, now ... implies ... within – against state extracted from the live DOM. This note tracks what actually works when the target is a single-page app, using wal.sh's own pocket-es search surface as the running example. It is a living document; entries are added as the eldest-spec suite evolves.

The recurring theme: a property-based suite is only as good as its extractors. Most "violations" you hit early are refutations of the spec, not the site. 8.1 tests that theme against LLM-inferred properties compiled to Bombadil.

2. Installation and environment

  • The npm package bundles the Rust binary for three platforms. @antithesishq/bombadil ships prebuilt binaries for linux-x64, linux-arm64 and darwin-arm64 under binaries/, dispatched by a bin/bombadil.js shim – npm install alone gives you a working bombadil on those targets. Outside them (FreeBSD, Windows, darwin-x64) there is no binary in the package; install separately (release asset bombadil-<arch>-<os>, cargo, or Nix) and keep it on PATH.
  • Bombadil manages its own Chromium. bombadil test resolves chromium on PATH. If you only have Google Chrome, point a chromium shim at it. On a CI runner, install chromium-browser explicitly.
  • Drive an existing browser with test-external. When the managed-Chromium path is broken or you want to watch the run, launch Chrome with --remote-debugging-port=9992 and use bombadil test-external --remote-debugger http://localhost:9992 --create-target.

3. SPA-specific behavior

  • The URL stays constant; exploration happens in-page. For a client-side surface like pocket-es, the whole run can sit on /search/ – Bombadil types into inputs and clicks results without a navigation. Do not assume the trace will show many distinct URLs; assert on DOM state, not on route changes.
  • Use the origin as a boundary. The first argument to test is both the start URL and the boundary – Bombadil will not navigate off it. Point it at the exact surface you mean to exercise.
  • Prefer quiescence over fixed waits. Bombadil 0.5.0 replaced fixed timeouts with quiescence timers (#176), which settles SPA re-renders far better. Bound the run with --time-limit rather than a step count.
  • Watch for navigation stalls. A single slow or never-settling route can trip navigation timed out ... during Loading. Keep the boundary tight and the time limit modest while iterating.

4. Extractors must match the real DOM

Verify every selector against the live DOM in a real browser before trusting a refutation. Three false positives from the wal.sh suite, all spec bugs:

  • Over-broad selectors capture the wrong element. ul:first-of-type > li > a matched the page table-of-contents, not <nav>, so the nav check was polluted with article titles. Scope to nav a.
  • Encode the site's actual structure, not an assumed one. The conjecture assumed a "Home" nav item; wal.sh has Research/Events/Current/Search and a wordmark that is the home link. A property asserting something the site never had fails forever and teaches you nothing.
  • Logos are not always images. a:has(img[alt*"wal.sh"])= matched nothing – the home link is an <a class"wordmark">= text logo. The extractor silently returned null.

5. Headless vs. real-browser divergence

  • Cross-origin resources flake under headless. An external badge (static.fsf.org/...) reported naturalWidth == 0= headless but loaded fine (182×45) in a real browser. A "no broken images" property fired on a network artifact, not a defect. Restrict such checks to same-origin resources, or wait for load before asserting.
  • Always confirm a refutation in a headed browser before filing it as a site bug. The cheapest debugging step is opening the witness URL yourself.

6. Guarding extractors against their own crashes

An extractor that throws aborts the entire run, costing you every other property. Defend the DOM calls:

  • querySelector(href) where href == "#"= throws (SyntaxError: '#' is not a valid selector). Exclude bare fragments (a[href^"#"]:not([href="#"])=) and wrap the lookup in try/catch.
  • Treat any malformed-input path as "no result," never as an exception.

7. Determinism and reproduction

  • Bombadil 0.5.0 has no --seed flag. Determinism comes from --reproduce <TRACE_FILE>, which replays a recorded trace exactly (#177). Do not design a seed-keyed reproduction scheme around a flag that does not exist.
  • Key artifacts to the build. Write traces and screenshots under runs/<build-sha>/ so a refutation is tied to the commit that produced it. The trace runs/<sha>/trace.jsonl plus --reproduce is the witness.

8. Checklist

  1. Binary on PATH (bombadil --version); Chromium resolvable.
  2. tsc --noEmit clean (add DOM.Iterable to lib; skipLibCheck for upstream type bugs).
  3. Every selector verified against the live DOM in a real browser.
  4. Same-origin guards on resource-loading properties.
  5. Extractors wrapped against querySelector / parse exceptions.
  6. Run bounded by --time-limit, output under runs/<sha>/.
  7. Each refutation reproduced with --reproduce before it is believed.

8.1. Inferred properties: SPINACH

See also: 4 | 7 | 8

Everything above assumes a person writes the properties. SPINACH, presented at SpecOps 2026 by Savitha Ravi and Michael Coblenz (Ravi and Coblenz 2026), asks whether an LLM can write them instead, and targets Bombadil as its backend. That makes it a direct test of this note's recurring theme: a suite is only as good as its extractors, and most early violations refute the spec.

8.1.1. What SPINACH does

SPINACH is a VSCode extension with a three-phase pipeline, each phase gated by developer review.

  1. Concept model. An LLM agent crawls the running application and decomposes it into concepts in Daniel Jackson's sense (Jackson 2021): units such as User, Article or follow, each with a purpose, an operational principle, state fields, actions and invariants, plus synchronizations that tie concepts together. The agent may infer actions it never reached: a trash concept implies a "remove from trash" action whether or not exploration saw the button.
  2. Natural-language properties. A second agent derives properties per concept and synchronization. It is deliberately not told it is writing property-based tests; the authors report that framing biased output toward "very mathematical, but inconsequential" properties.
  3. LTL and Bombadil. A final phase formalizes each approved sentence into LTL where it can, then emits a Bombadil test.

The paper's worked example: "a user must be logged in to follow another" becomes \(G(\mathit{following} \rightarrow \mathit{authenticated})\). The generated Bombadil test is an always over two extractors: when the viewer is unauthenticated and on a profile page, the follow button's label must not start with "Unfollow".

8.1.2. What the evaluation found

Two open-source applications, both previously used to evaluate GUI-testing tools.

  RealWorld (Angular Medium clone) 4ga Boards (Kanban)
concepts 14 23
dependencies between concepts 9 not reported
synchronizations 2 not reported
natural-language specifications 94 161
share on authentication/authorization over half 40%

Very few specifications were fully formalized in LTL. Most of the remainder were entity-relationship constraints ("every card belongs to exactly one board") that need first-order quantification, which propositional LTL lacks. Accuracy was assessed on a sample of five specifications, judged by the first author and then discussed with the second; the paper reports no score.

The generated tests compiled. They found no bugs in either application. The authors attribute this to Bombadil's execution model rather than to the properties: actions are chosen by weighted random selection, so states behind a login, a navigation and a form fill are rarely reached, and an LLM-generated selector that misses the real DOM never fires at all.

Everything else is planned: mutation testing for RQ1 against two baselines (crawl then PBT directly; crawl then EARS requirements then PBT, the Kiro route), running tests across real version histories as a less synthetic fault source, an ablation that strips concepts down to purpose and operational principle, and a user study for RQ2 on legibility and maintenance. Proposed fixes: replace random action selection with constrained trace generators that synthesize directed interaction sequences (Zhou et al. 2026), and reconsider LTL itself, citing evidence that non-specialists misread it (Greenman et al. 2023).

8.1.3. Against Bombadil: written versus inferred

Bombadil, and Quickstrom before it (O’Connor and Wickström 2022), gives the author LTL over browser state and assumes the author knows which properties matter. SPINACH keeps that checker and replaces the author. The interesting result is what the inferred properties look like once checked.

  • They are state invariants. The worked example is a single \(G\) over a propositional implication. Nothing in the paper exercises eventually or within, the operators that make Bombadil more than an assertion library. Conjecture: concept invariants, by construction, compile to safety properties of one state, so the temporal half of the language goes unused.
  • The extractor problem moves, it does not go away. The worked test returns true when the follow-button label is null. If the selector is wrong, the property passes on every state. This is the null-returning logo extractor from 4, generated rather than written, and it fails in the quiet direction. The paper names "imperfect selectors" as one cause of zero findings; this note's checklist item 3 (verify each selector against the live DOM) applies unchanged to generated tests.
  • Exploration is the bottleneck in both. The navigation stalls and quiescence notes in 3 are the hand-written version of the reachability limit the paper hits. Neither gets better by improving the properties.

8.1.4. Against the surfaces framing

Context Surfaces separates context a system emits from context a worker elicits, and files property-based driving under the second. SPINACH's crawl reads the DOM surface, but its properties come from two places, and only one of them is the surface.

The concept model's own integrity notes say a design choice "is the standard design for this class of app." That is the model's prior, not anything the page showed. The trash example is the same move: a state inferred because apps of this kind usually have it. Inferring past the surface is how SPINACH finds properties nobody exercised; it is also how a property comes to assert a "Home" link the site never had, the second false positive recorded above. The paper's developer-review gates are where that gets caught, so the review step is the mechanism that keeps the inference honest, not an add-on.

Some of the properties are not on the surface at all. "Email must be unique across all users" cannot be observed from one browser session; one user's DOM never shows every other user's email. Conjecture: part of the paper's formalization gap is a surface gap. These are database-surface properties reached through a DOM-surface tool, and no choice of logic fixes that.

8.1.5. Against the PBT discipline notes

  • Calibration. Zero bugs found on two apps is an uncalibrated result. Until the suite has run against a known-bad build and gone red, a green run says nothing about the apps; the paper's planned mutation and version-history runs are exactly the calibration missing so far. The site's rule (a gate never run against a known-bad input is uncalibrated) applies directly, and the authors make the same point in their own terms.
  • Oracles. The crowsnest rig checks the receiver against an independent model that re-derives expected state from the spec after every step. A Jackson concept already has state, actions and invariants, which is most of such a model. Conjecture: an executable concept, stepped alongside the browser, would be a stronger oracle than the per-state LTL sentences, and it would cover the entity-relationship constraints that do not fit propositional LTL.
  • Shrinking and pin-on-find. With no violation, there was nothing to shrink or pin. When one fires, the shrink discipline carries over with Bombadil's substitutes: the recorded trace and --reproduce replace the seed, and the minimal trace becomes the pinned regression. Nothing in SPINACH changes that step.
  • Where it fits. As a generator of candidate properties for a human to review, SPINACH matches what these notes argue: the 94 and 161 sentences are a reading list for whoever writes the suite, and the share on authentication suggests where the model's attention goes. As an oracle that runs unattended, it does not yet fit: the properties are uncalibrated, the extractors are unverified, and the exploration that would exercise them does not reach the states they constrain.

Daikon infers likely invariants from observed traces (Ernst et al. 2001); SPINACH infers them from a model of what the application should be. Conjecture: the failure modes mirror each other. Trace mining fits too closely to what happened; prior-driven inference fits too closely to what usually happens in apps of this class. The first misses properties the traces never violated, the second asserts properties the site never had. The review gate exists for the second.

8.1.6. References

Ernst, Michael D., Jake Cockrell, William G. Griswold, and David Notkin. 2001. “Dynamically Discovering Likely Program Invariants to Support Program Evolution.” Ieee Transactions on Software Engineering 27 (2): 99–123. https://doi.org/10.1109/32.908957.
Greenman, Ben, Sam Saarinen, Tim Nelson, and Shriram Krishnamurthi. 2023. “Little Tricky Logic: Misconceptions in the Understanding of LTL.” The Art, Science, and Engineering of Programming 7 (2). https://doi.org/10.22152/programming-journal.org/2023/7/7.
Jackson, Daniel. 2021. The Essence of Software: Why Concepts Matter for Great Design. Princeton University Press.
O’Connor, Liam, and Oskar Wickström. 2022. “Quickstrom: Property-Based Acceptance Testing with LTL Specifications.” In Proceedings of the 43rd Acm Sigplan International Conference on Programming Language Design and Implementation (Pldi 2022), 1025–38. San Diego, CA, USA: Association for Computing Machinery. https://doi.org/10.1145/3519939.3523728.
Ravi, Savitha, and Michael Coblenz. 2026. “SPINACH: Inferring Properties of Web Applications for Property-Based Testing.” In Proceedings of the 1st International Workshop on Specification-Driven Development Life Cycle (Specops ’26), 10–13. Oakland, CA, USA: Association for Computing Machinery. https://doi.org/10.1145/3842652.3843198.
Zhou, Zhe, Ankush Desai, Benjamin Delaware, and Suresh Jagannathan. 2026. “Trace-Guided Synthesis of Effectful Test Generators.” Proceedings of the Acm on Programming Languages 10 (PLDI). https://doi.org/10.1145/3808264.

Seen at

  • BugBash 2026 — Oskar Wickström's talk, Old Tom Bombadil is a merry fuzzer!.