A/B engine — spec
Table of Contents
1. Purpose
The A/B engine is the single source of bucket assignment on wal.sh.
Every adtech module that segments users — sponsored-research
archetype, attribution-audit holdout, exit-intent gating, plate A/B —
consults this module directly or reuses its hash. One reader in
variant-a sees variant-a across the whole surface.
Two hashes exist in the tree: fnv1a-32 in ab-engine.core (the
primary engine, host-portable to JS) and djb2 seed 5381 in
attribution-audit.core and sponsored-research.core (a smaller,
faster hash inherited from an early port of the JS ab-engine).
djb2 callers are deliberately kept in sync with the ab-engine's
djb2 sibling so a variant-a assignment agrees across both hashes
for the four production experiments.
ORACLE: src/wal_sh/adtech/ab_engine/core.cljc (pure), browser.cljs
(window._abTesting), server.clj (replay).
Rationale for research: A/B is the smallest measurement unit in the
adtech stack. Formalising it as a pure function with a :cljs=/:clj=
parity test surface lets us replay real traffic through the engine on
the JVM and compare against the browser assignment — closes the
"assignment drift" class of bug where a hash difference silently
splits the audience.
2. Requirements
- A1
- Assignment is a pure function of
(user-id, experiment)— same input, same variant, forever. No global state read at assignment time. - A2
- Bucket in
[0, 1)is(fnv1a-32(user:exp) mod 10000) / 10000. - A3
- If
bucket >traffic=, the user is not in the experiment;assignreturnsnil. - A4
- Otherwise
normalized = bucket / trafficwalks the variant weights; first cumulative weight strictly greater thannormalizedwins. Variants must have weights summing to 1.0. - A5
fnv1a-32must produce the same value on JVM and JS for the same string. Achieved via explicit 32-bit masking after every multiply (Math.imulon JS;unchecked-multiply-inton JVM).- A6
- Exposure and conversion are set-valued per user — recording the same user twice is a no-op. Prevents double-counting.
- A7
- Two-proportion z-test returns a coarse Gaussian
approximation and a
:reasonstring when either sample is zero. For real inference the Wilson interval is provided separately. - A8
- The four production experiments (
exp-001throughexp-004) are named literals incore— the JS module is the historical source of truth and this namespace matches it 1:1. - A9
- A parallel
djb2hash exists inattribution-audit.core=/=sponsored-research.corefor the fourexp-101..104sponsored experiments. Its seed (5381), its key format (userId:expId), and its bucket space (mod 10000) matchfnv1a-32; only the arithmetic differs.
3. Contract signature
Pure (core.cljc):
(fnv1a-32 s) ; -> uint32, host-portable
(bucket s) ; -> double in [0,1)
(assign user-id experiment-map) ; -> variant-map or nil
(evaluate user-id [exps]) ; -> {exp-id {:experiment :variant :variant-id}}
(exposed state exp-id variant-id user-id) ; -> state'
(converted state exp-id variant-id user-id) ; -> state'
(counts state exp-id variant-id) ; -> {:exposed :converted :rate}
(lift state exp-id ctrl-id treat-id) ; -> relative lift or nil
(absolute-lift state exp-id ctrl-id treat-id) ; -> pT-pC or nil
(z-test cc cs vc vs) ; -> {:z-score :p-value :significant? :reason}
(required-sample-size mde alpha power p) ; -> n or :infinity
(wilson-half-width p n) ; -> half-width or nil
Browser surface:
window._abTesting = {
evaluate: (fn [user-id] -> {exp-id ...}) ; assign every exp for user
exposed: (fn [exp variant] -> nil)
converted: (fn [exp variant] -> nil)
report: (fn [] -> nil) ; console.log lines
}
window._ab_uid ; localStorage key holding the id
4. Bucketing invariants (property tests)
- Determinism:
(assign u e)is idempotent across invocations. - Distribution: for a fair coin split, the empirical proportion is
within the Wilson 95 % interval over N samples for
N ≥ required-sample-size.wilson-half-widthis the standing test bound. - Cross-host parity:
fnv1a-32on a fixed dictionary produces identical results on JVM and JS. Verified with a REPL fixture. - Traffic slicing:
P(assign u e ≠ nil) = experiment.trafficwithin the Wilson interval.
5. Related literature
- Vickrey on second-price auctions and truthful bidding (Vickrey 1961) frames why a bucketed A/B is the cheapest possible auction proxy — the "bid" is a per-user random assignment, the "price" is the observed lift.
- Edelman, Ostrovsky & Schwarz on the generalized second-price
auction for ad slots (Edelman, Ostrovsky, and Schwarz 2007) motivates the
auction-disclosure experiment (
exp-104) — showing bid arithmetic is a variant we test explicitly. - Google's 2019 move from second-price to first-price is the industry-scale experiment we cannot run (Google Ad Manager Team 2019); ours is the smaller analogue on a sponsored slot.
- FNV-1a hash choice: the reference implementation and its 32-bit
variant are ubiquitous in ad-server load balancers because
mod 10000gives a stable bucket without a keyed HMAC. FNV-1a's distribution over short strings has been characterised in the hash-function benchmarks maintained by https://github.com/aappleby/smhasher. - djb2 (Bernstein) is documented at http://www.cse.yorku.ca/~oz/hash.html; retained for parity with the earlier JS module.
- Ali et al. on discrimination through delivery (Ali et al. 2019) motivates why an audit-visible A/B hash matters: if the delivery is opaque, discrimination is undetectable by design.
6. Cross-references
- attribution-audit/spec.org — reuses
djb2for arm assignment (exposed=/=holdout) with the same key format. - sponsored-research/spec.org — reuses
djb2for archetype-per-user and per-slot mulberry32 seeding. - exit-intent/spec.org — reads
arm; suppressed inholdout(R31). - plates/spec.org — plate A/B (not yet built) will consult this engine.
- ORACLE:
src/wal_sh/adtech/ab_engine/core.cljc(pure)src/wal_sh/adtech/ab_engine/browser.cljs(window._abTesting)src/wal_sh/adtech/ab_engine/server.clj(JVM replay)
- [BROKEN LINK: No match for fuzzy expression: *2007–2013: Programmatic exchange era] on how bucketing hashes became universal in DSPs.
7. Open questions
- Should
djb2be retired in favour offnv1a-32across the board? The two hashes are kept in sync for the current experiment set; divergence risk is low but nonzero. A property test running both hashes over the same key space and asserting equivalence-modulo- bucketing would close this. - Consent-gating. The engine assigns unconditionally today. Under the IAB TCF v2 profile (IAB Europe 2020), experiments that affect content presentation may require Purpose 1 consent; the scaffolding for a consent-gated engine is not written.