A/B engine — spec

Table of Contents

1. Purpose

The A/B engine is the single source of bucket assignment on wal.sh. Every adtech module that segments users — sponsored-research archetype, attribution-audit holdout, exit-intent gating, plate A/B — consults this module directly or reuses its hash. One reader in variant-a sees variant-a across the whole surface.

Two hashes exist in the tree: fnv1a-32 in ab-engine.core (the primary engine, host-portable to JS) and djb2 seed 5381 in attribution-audit.core and sponsored-research.core (a smaller, faster hash inherited from an early port of the JS ab-engine). djb2 callers are deliberately kept in sync with the ab-engine's djb2 sibling so a variant-a assignment agrees across both hashes for the four production experiments.

ORACLE: src/wal_sh/adtech/ab_engine/core.cljc (pure), browser.cljs (window._abTesting), server.clj (replay).

Rationale for research: A/B is the smallest measurement unit in the adtech stack. Formalising it as a pure function with a :cljs=/:clj= parity test surface lets us replay real traffic through the engine on the JVM and compare against the browser assignment — closes the "assignment drift" class of bug where a hash difference silently splits the audience.

2. Requirements

A1
Assignment is a pure function of (user-id, experiment) — same input, same variant, forever. No global state read at assignment time.
A2
Bucket in [0, 1) is (fnv1a-32(user:exp) mod 10000) / 10000.
A3
If bucket > traffic=, the user is not in the experiment; assign returns nil.
A4
Otherwise normalized = bucket / traffic walks the variant weights; first cumulative weight strictly greater than normalized wins. Variants must have weights summing to 1.0.
A5
fnv1a-32 must produce the same value on JVM and JS for the same string. Achieved via explicit 32-bit masking after every multiply (Math.imul on JS; unchecked-multiply-int on JVM).
A6
Exposure and conversion are set-valued per user — recording the same user twice is a no-op. Prevents double-counting.
A7
Two-proportion z-test returns a coarse Gaussian approximation and a :reason string when either sample is zero. For real inference the Wilson interval is provided separately.
A8
The four production experiments (exp-001 through exp-004) are named literals in core — the JS module is the historical source of truth and this namespace matches it 1:1.
A9
A parallel djb2 hash exists in attribution-audit.core=/=sponsored-research.core for the four exp-101..104 sponsored experiments. Its seed (5381), its key format (userId:expId), and its bucket space (mod 10000) match fnv1a-32; only the arithmetic differs.

3. Contract signature

Pure (core.cljc):

(fnv1a-32 s)                                      ; -> uint32, host-portable
(bucket s)                                        ; -> double in [0,1)
(assign user-id experiment-map)                   ; -> variant-map or nil
(evaluate user-id [exps])                         ; -> {exp-id {:experiment :variant :variant-id}}
(exposed state exp-id variant-id user-id)         ; -> state'
(converted state exp-id variant-id user-id)       ; -> state'
(counts state exp-id variant-id)                  ; -> {:exposed :converted :rate}
(lift state exp-id ctrl-id treat-id)              ; -> relative lift or nil
(absolute-lift state exp-id ctrl-id treat-id)     ; -> pT-pC or nil
(z-test cc cs vc vs)                              ; -> {:z-score :p-value :significant? :reason}
(required-sample-size mde alpha power p)          ; -> n or :infinity
(wilson-half-width p n)                           ; -> half-width or nil

Browser surface:

window._abTesting = {
  evaluate:  (fn [user-id] -> {exp-id ...})       ; assign every exp for user
  exposed:   (fn [exp variant] -> nil)
  converted: (fn [exp variant] -> nil)
  report:    (fn [] -> nil)                       ; console.log lines
}
window._ab_uid                                    ; localStorage key holding the id

4. Bucketing invariants (property tests)

  • Determinism: (assign u e) is idempotent across invocations.
  • Distribution: for a fair coin split, the empirical proportion is within the Wilson 95 % interval over N samples for N ≥ required-sample-size. wilson-half-width is the standing test bound.
  • Cross-host parity: fnv1a-32 on a fixed dictionary produces identical results on JVM and JS. Verified with a REPL fixture.
  • Traffic slicing: P(assign u e ≠ nil) = experiment.traffic within the Wilson interval.

5. Related literature

  • Vickrey on second-price auctions and truthful bidding (Vickrey 1961) frames why a bucketed A/B is the cheapest possible auction proxy — the "bid" is a per-user random assignment, the "price" is the observed lift.
  • Edelman, Ostrovsky & Schwarz on the generalized second-price auction for ad slots (Edelman, Ostrovsky, and Schwarz 2007) motivates the auction-disclosure experiment (exp-104) — showing bid arithmetic is a variant we test explicitly.
  • Google's 2019 move from second-price to first-price is the industry-scale experiment we cannot run (Google Ad Manager Team 2019); ours is the smaller analogue on a sponsored slot.
  • FNV-1a hash choice: the reference implementation and its 32-bit variant are ubiquitous in ad-server load balancers because mod 10000 gives a stable bucket without a keyed HMAC. FNV-1a's distribution over short strings has been characterised in the hash-function benchmarks maintained by https://github.com/aappleby/smhasher.
  • djb2 (Bernstein) is documented at http://www.cse.yorku.ca/~oz/hash.html; retained for parity with the earlier JS module.
  • Ali et al. on discrimination through delivery (Ali et al. 2019) motivates why an audit-visible A/B hash matters: if the delivery is opaque, discrimination is undetectable by design.

6. Cross-references

  • attribution-audit/spec.org — reuses djb2 for arm assignment (exposed=/=holdout) with the same key format.
  • sponsored-research/spec.org — reuses djb2 for archetype-per-user and per-slot mulberry32 seeding.
  • exit-intent/spec.org — reads arm; suppressed in holdout (R31).
  • plates/spec.org — plate A/B (not yet built) will consult this engine.
  • ORACLE:
    • src/wal_sh/adtech/ab_engine/core.cljc (pure)
    • src/wal_sh/adtech/ab_engine/browser.cljs (window._abTesting)
    • src/wal_sh/adtech/ab_engine/server.clj (JVM replay)
  • [BROKEN LINK: No match for fuzzy expression: *2007–2013: Programmatic exchange era] on how bucketing hashes became universal in DSPs.

7. Open questions

  • Should djb2 be retired in favour of fnv1a-32 across the board? The two hashes are kept in sync for the current experiment set; divergence risk is low but nonzero. A property test running both hashes over the same key space and asserting equivalence-modulo- bucketing would close this.
  • Consent-gating. The engine assigns unconditionally today. Under the IAB TCF v2 profile (IAB Europe 2020), experiments that affect content presentation may require Purpose 1 consent; the scaffolding for a consent-gated engine is not written.
Ali, Muhammad, Piotr Sapiezynski, Miranda Bogen, Aleksandra Korolova, Alan Mislove, and Aaron Rieke. 2019. “Discrimination through Optimization: How Facebook’s Ad Delivery Can Lead to Biased Outcomes.” In Proceedings of the Acm on Human-Computer Interaction. Vol. 3. CSCW. https://doi.org/10.1145/3359301.
Edelman, Benjamin, Michael Ostrovsky, and Michael Schwarz. 2007. “Internet Advertising and the Generalized Second-Price Auction: Selling Billions of Dollars Worth of Keywords.” American Economic Review 97 (1): 242–59. https://doi.org/10.1257/aer.97.1.242.
Google Ad Manager Team. 2019. “Rolling out First Price Auctions to Google Ad Manager Partners.” Google Blog. https://blog.google/products/admanager/rolling-out-first-price-auctions-google-ad-manager-partners/.
IAB Europe. 2020. “Transparency & Consent Framework V2.0 Policies.” https://iabeurope.eu/tcf-2-0/.
Vickrey, William. 1961. “Counterspeculation, Auctions, and Competitive Sealed Tenders.” The Journal of Finance 16 (1): 8–37. https://doi.org/10.1111/j.1540-6261.1961.tb02789.x.