wal.sh
home > research > crowsnest-pbt-grinder · Jun 25, 2026
crowsnestproperty-based testinghypothesispbtmodel-based testingwire-contractsightingsgrinder

Property-testing the crowsnest client
Hypothesis and a model-based oracle keep the v2 wire contract honest, and caught the 2.3.0 drift on their own

Table of Contents

This is a dated record. It describes the property-based test rig in aygp-dr/crowsnest-py as it stood on 2026-06-25, the day the canonical receiver in this repo (src/wal_sh/cn/server.clj) moved from 2.0.0 through 2.3.0 to 2.4.0. The spec it tests against is crowsnest spec v2.4.

1. The rig, and its job

crowsnest-py is not a product. It is a minimal compliant client for the crowsnest v2 wire contract, a mock-data grinder, and the point: a property-based test rig whose job is to whine, to surface gaps between the spec and the receiver by handing Hypothesis enough rope to find a counterexample.

A client emits a sighting (a flat event) to the receiver on :8127/sightings; the dashboard reads the log via GET /sightings. The contract is small and unforgiving: required {name, service, duration, status}; status coerced to unknown (never silently ok); flat attrs capped at depth 64; finite duration/start/count; a newest-first ring of 100; typed errors {ok:false, error:{code, message}}. Everything below is in service of proving the receiver actually does those things, against not five hand-picked inputs but the whole shape of the wire.

2. What the tree holds

The numbers in this note are what the test tree shows at a named commit, not what a run log remembered. At HEAD e1a267d (2026-06-26) there are 43 test functions across four files; 24 of them are the P1 to P14 properties (letter suffixes are sub-cases of one property), 16 carry a @given decorator, and the stateful machine contributes two rules and two invariants. 

$ git -C ~/ghq/github.com/aygp-dr/crowsnest-py rev-parse --short HEAD
e1a267d
$ grep -c 'def test_' tests/*.py
tests/test_grinder.py:8
tests/test_openapi.py:7
tests/test_properties.py:24
tests/test_simulator.py:4
$ cat tests/*.py | grep -c 'def test_'
43
$ grep -cE '@given|hypothesis' tests/*.py        # nonzero files only
tests/test_properties.py:14
tests/test_simulator.py:5
$ cat tests/*.py | grep -cE '^\s*@given'
16
$ grep -hoE 'def test_p[0-9]+[a-z]?_' tests/*.py | sort -u | wc -l
24

pytest --collect-only was not run for this record (it needs the package's dependencies installed), so the collected-item count is not reproduced here. The "68 of 69" figure quoted later is the count from the original 2026-06-25 run as recorded in the repo's STATUS.md; it is larger than 43 because tests/test_grinder.py parametrizes three of its tests over the four grinder modes.

3. Four ways property-based testing is applied

3.1. Falsifiable claims, one per property (P1 to P14)

Each property is a single falsifiable statement about the receiver, run over 80 Hypothesis-generated inputs (max_examples=80 in tests/test_properties.py) plus the boundary values seeded by hand. Twenty-four test functions cover the fourteen properties, e.g.:

  • test_p1_round_trip: a well-formed sighting survives ingest unchanged.
  • test_p2_status_coerced_to_unknown: any out-of-enum status string lands as unknown, never ok. This is the correctness fix the whole v2 cutover existed for, asserted over arbitrary text rather than a fixed list.
  • test_p5b_nan_inf_duration_rejected / test_p5c_non_number_duration_rejected: R3: NaN/Infinity and non-numbers are 400, and nothing non-finite ever reaches a GET.
  • test_p9b_overdeep_attrs_rejected: R2: attrs past depth 64 are rejected atomically.
  • test_p10_arbitrary_object_never_drops_connection: throw a random JSON object at the receiver; it must answer, never drop the socket. (14% of generated inputs are "invalid", rejected by the strategy filter, which is itself a signal the generator is probing the hostile region.)

The strategies seed the edges deliberately: "OK", "warning", "", " ", "0", "true", "ok " for status; NaN/Infinity literals for duration; unicode for the string fields. Fixtures sample what you thought of; Hypothesis samples that and what you didn't.

3.2. Model-based stateful testing: the contract vs an oracle

The strongest test is not a single property but a machine. ContractMachine (a Hypothesis RuleBasedStateMachine in tests/test_simulator.py) drives long random sequences of two moves:

  1. emit: a well-formed sighting (with an explicit unique id so ordering is checkable), expecting {ok:true, stored:1}.
  2. emit_malformed: a contract-violating body (bare scalar, non-object attrs, 200-deep nesting, NaN duration, a bad item inside a batch), expecting a typed BadSighting to 400 and the log left untouched (rejection is atomic).

After every step, two invariants must hold. First, simulator_matches_oracle: the receiver's in-memory log equals an independent oracle that re-derives the expected log from the spec rules: coerce status, insert newest-first, cap at 100. The receiver and the oracle are two implementations of the same contract; if they ever disagree, Hypothesis shrinks the divergent run to its minimal form. Second, contract_invariants_hold: every stored row has a string id, an enum status, a finite duration, an object attrs. Green-by-default is structurally impossible.

This runs against the receiver's pure ingest core, not over HTTP, so it tests the contract logic the spec actually constrains, with no server, at 200 sequences of up to 40 steps each (max_examples=200, stateful_step_count=40).

3.3. The grinder as a realistic-load fixture

The same client doubles as a load generator with a pluggable mode adapter: one mode-agnostic emit loop, and a registry of mock vocabularies selected by name (crowsnest-grind --mode {ecomm,itil4,llm,monit}). Each mode is a pure random_sighting(rng): e-commerce traffic, an ITIL4 service-management org, local LLM-agent telemetry, monit-style systems checks over a faked server fleet.

Two properties hold across every mode, parametrized:

  • spec-validity: whatever the vocabulary, each sighting carries the required fields with sane types and flat attrs (no nested objects/lists). 400 samples per mode.
  • coercion bait: each mode deliberately plants out-of-enum statuses (in_progress, breached for ITIL4; rate_limited, truncated for LLM; connection failed for monit; degraded, throttled for e-commerce). Grinding any mode through a live receiver, the stored log must coerce all of them to unknown, visible as amber on the board, never green.

So the grinder isn't just demo data; it's a fourth PBT surface that exercises the contract under believable, adversarial-by-design traffic.

3.4. A drift sentinel

test_vendored_matches_canonical_if_reachable (tests/test_openapi.py, line

  1. is opportunistic: if the canonical receiver is up on :8127, it diffs the

vendored mirror's /openapi.json against it, paths, schema components, per-schema properties, and skips cleanly when it is not. A red here is a finding, not a flake: the vendored receiver has fallen behind the canonical one. 

def test_vendored_matches_canonical_if_reachable(clear_log):
    """If the canonical receiver is up on :8127, the vendored OpenAPI should
    agree on paths and schema shape (xfail-on-drift would be a finding)."""
    try:
        canonical = json.load(urllib.request.urlopen("http://127.0.0.1:8127/openapi.json", timeout=1))
    except (urllib.error.URLError, OSError, json.JSONDecodeError):
        pytest.skip("canonical receiver not reachable on :8127")
    vendored = c.fetch_openapi()
    assert _doc_routes(vendored) == _doc_routes(canonical), "path/method drift vs canonical"
    assert set(vendored["components"]["schemas"]) == set(canonical["components"]["schemas"])
    for name, schema in canonical["components"]["schemas"].items():
        v = vendored["components"]["schemas"][name]
        assert set(v.get("properties", {})) == set(schema.get("properties", {})), \
            f"{name} property drift vs canonical"

4. What it has caught

The v1 suite surfaced six real gaps (F1 to F6), each a shrunk counterexample: non-dict meta crashing the receiver, deep-nesting DoS, NaN/Inf violating RFC-8259, status case/defaulting, host-attribution noise, sha precedence. (F7, a count-default "bug", turned out to be a spec misread: the default is client-side.) The v2 cutover closed all six; the same properties now stand as conformance assertions, green.

Then, on 2026-06-25, the sentinel earned its keep. That day the canonical receiver src/wal_sh/cn/server.clj in this repo shipped 2.3.0 (5bc29385), then 2.4.0 (71dd8eed), while the vendored mirror in crowsnest-py was still 2.0.0. Running the suite isolated on a spare port, 68 of 69 tests passed and one went red: test_vendored_matches_canonical_if_reachable. The repo's STATUS.md (eae94e0, 2026-06-25) records it: 

## PBT status

68/69 pass on the spare port. The 1 red — `test_vendored_matches_canonical_if_
reachable` — is the **drift sentinel** firing because live is now 2.3.0 while the
vendored mirror is 2.0.0. Leave it red; it's the signal until we reconcile. The
drift is 100% additive (Sighting +`state`/`run`/`after`; attrs +`env`/`priority`/
`region`/`threshold`/`value`; Info +`capabilities`) — confirms the minor-ladder.

The diff the test produced was exactly the new contract surface:

surface 2.0.0 → 2.3.0 delta
Sighting fields + state, run, after
attrs reserved + env, priority, region, threshold, value
Info + capabilities ["state","typed-attrs","correlation"]
paths / schemas / codes nothing removed, nothing renamed

The PBT rig detected a version bump in a separate repo and characterized it, field by field, with no one telling it to look. And it confirmed the semver story empirically: the delta is 100% additive, which is precisely why 2.3 is a minor bump and not a 3.0. A v2 client sees zero breaking change.

The canonical side of that day, from this repo's history. server.clj has not moved since: 

$ git log --format='%h %ad %s' --date=short -- src/wal_sh/cn/server.clj | head -6
71dd8eed 2026-06-25 feat(crowsnest): v2.4.0 — client→reporter rename + generic attrs.origin, e2e
30f5fd19 2026-06-25 feat(crowsnest): DELETE /sightings clears the ring ('clear' capability)
52a15e9c 2026-06-25 feat(crowsnest): env-configurable ring-cap + /info.retention.cap
5bc29385 2026-06-25 feat(crowsnest): v2.3.0 — flat correlation (run/after) + Info.capabilities (additive)
7e4a45a4 2026-06-25 feat(crowsnest): v2.2.0 — verbatim state label + typed reserved attrs (additive)
e94e5408 2026-06-25 fix(crowsnest): bump receiver version 2.0.0 -> 2.1.0 (align with spec)
$ git log -1 --format=%ad --date=short -- src/wal_sh/cn/server.clj
2026-06-25
$ grep -n 'def ^:const version' src/wal_sh/cn/server.clj
36:(def ^:const version "2.4.0")

The sentinel saw 2.3.0 because that is what was live on :8127 when the suite ran; by the end of the same day canonical was 2.4.0, so the gap the mirror has to close is wider than the table above shows. The 2.4.0 delta (reporter rename, generic attrs.origin) is also additive.

5. Why property-based, not fixtures

A fixture suite samples five known-good inputs and asserts five known-good outputs. It tells you the receiver round-trips fixture 1. Hypothesis tells you the receiver round-trips fixture 1 and every other shape the wire admits except these three classes, and when a property fails, it shrinks the input toward the minimum. The shrunk input is the finding: not "something broke" but the smallest sighting that breaks it, which is usually a one-line spec clarification.

Model-based testing goes further still: by re-deriving the expected log from the spec rules and diffing against the real receiver after every step, it turns the spec itself into an executable oracle. The contract stops being prose you hope the code matches and becomes a thing two implementations are continuously checked against.

6. How it feeds the contract forward

Building four grinder vocabularies on top of v2 is also how the next version got specified. Every mode lost real detail to v2's status coercion: an ITIL breached and an LLM rate_limited both flattened to unknown. That information loss, surfaced by the modes, is the entire motivation for 2.3's state field: preserve the verbatim label, keep the coarse health bucket for the color. The PBT rig didn't just verify the contract; it generated the pressure that grew it.

As of this record the vendored mirror is at 2.0.0 and canonical is at 2.4.0. The sentinel stays red until the mirror is reconciled; per STATUS.md, that red is the signal, not a failure to suppress.