Experiments

Table of Contents

The site runs a client-side feature-gate SDK modeled on Statsig's client payload shape. There is no server: the gate registry is a static EDN file served from /static/experiments.json (exported from data/experiments.edn by gmake experiments-json) and evaluated in the browser.

The user id is a persistent uuid in localStorage under wal-sh.exp.uid — the local analogue of Statsig's analyticsAnonymousId.

Why

Two use cases pulled the same shape:

  • Tool rollouts. Ship a new /tools/ SPA behind a gate, ramp the percentage, and check nothing regresses before flipping to 100%.
  • Content variants. Split-test a page's layout, copy, or ordering with the same primitives, and record which arm the reader saw.

The 5 tools scaffolded from RubyConf 2026 (message-format, mutation, fib, gc-viz, fair-queue) each ship as a feature gate. Turn one off and its dropdown/link in site/tools/index disappears at render time.

Shape

Two record types, both keyed by string id:

  • feature-gates — return a boolean. Wrong id = false (fail-closed).
  • dynamic-configs — return a structured value. Wrong id = nil.

Both types share a :rules list. Each rule has:

Field Meaning
:id rule name (opaque; used for exposure attribution)
:percentage 0-100. Omitted = 100.
:salt changes the bucketing partition without renaming gates
:value the value returned when this rule matches
:secondary-exposures list of { :gate id :value bool } dependency checks

Rules are matched top-to-bottom; the first that matches wins. A :default rule (no percentage) at the bottom is the fallthrough.

Bucketing

Bucket assignment is deterministic on djb2(user-id ":" salt ":" gate-id) mod 10000. Djb2 was picked to match Statsig's hash_used: "djb2" — same-user-same-gate always land in the same bucket, which means a 50% rollout is stable across page loads.

(require '[wal-sh.experiments.core :as e])
(def user {:analytics-anonymous-id "aaaa-bbbb"})
(e/bucket (:analytics-anonymous-id user) "tool.gc-viz" "gc")

That 8642 falls above 5000, so the 50% gc-viz gate returns false for this user. Change the salt from gc to gc2 and the bucketing partition rotates — the user's arm changes without renaming the gate.

Currently gated

Sourced from data/experiments.edn; gmake deploy-experiments-json ships it. Change there, no bundle rebuild required: reload the page and the browser client refetches.

Gate Rollout Depends on Purpose
tool.message-format 100% — RubyConf 2026 — Fung
tool.mutation 100% — RubyConf 2026 — Schirp
tool.fib 100% — RubyConf 2026 — d'Oliveira
tool.gc-viz 50% tool.mutation RubyConf 2026 — Zhu (canary)
tool.fair-queue 100% — RubyConf 2026 — Baygeldin
tool.display 100% — 16-colour grid sink, spec v0.3.1
tool.pasted-file-editor 100% — editor with a paste threshold (after Simon Willison)
tool.dinner-split 0% — scaffold: core and tests, no UI yet
tool.gas-town-tla 0% — scaffold: core and tests, no UI yet
tool.mono-watch 0% — scaffold: core and tests, no UI yet
tool.order-state-flow 0% — scaffold: core and tests, no UI yet
tool.spect8r 0% — scaffold: core and tests, no UI yet
ui.tools-nav-experiments-section 100% — Show RubyConf-adjacent bucket

The tool.gc-viz canary demonstrates two features at once: percentage-based rollout (50%) and secondary exposure (only users already in the tool.mutation arm are eligible).

Flags

A second, simpler mechanism sits next to the gates above: a flat per-browser on/off switch, not a rollout. src/wal_sh/flags/core.cljc + src/wal_sh/flags/browser.cljs declare each flag once in a registry and resolve it by one fixed precedence: url-param > localStorage > env > default. A decisive ?param=on|off also writes itself into the flag's localStorage key, so it sticks on later visits without the param.

Flag Param localStorage key Default Purpose
ecommerce ?shop wal_shop_flag off site-wide commerce/checkout
verdict-badges ?verdict wal_verdict_badge_flag off show verification badges (verdict · checker · date) under headings

verdict-badges is hand-written vanilla JS (site/static/js/verdict-badge-flag.js), not the compiled flags.js bundle the ecommerce flag uses — same precedence and naming convention, but the compiled bundle is ~21KB gzipped (Closure/cljs runtime) against every other sitewide postamble script being under 3KB, and this flag needs no fetch, so there was nothing to gain from the bundle.

To turn verdict badges on for yourself: visit any page once with ?verdict=on appended — it sticks via localStorage, no need to repeat the param. Equivalent from the devtools console: localStorage.setItem('wal_verdict_badge_flag', 'on'), then reload. Reverse with ?verdict=off or the matching setItem(..., 'off'); remove the key entirely to fall back to the site default (hidden).

Exposures

The client records an exposure event whenever gate? or config-value resolves. Events are POSTed to the local crowsnest receiver at localhost:8127/sightings when it's running; otherwise buffered in sessionStorage. See crowsnest for the receiver.

Statsig shape reference

The record shape came from watching the wire on events.atlassian.com/teameurope/agenda — Atlassian ships Statsig's Java server SDK (v3.1.3) with djb2 hashing and analyticsAnonymousId as the primary id. We keep the same primitives but replace the service call with a static file, which is enough for a single-author site.

Related