Pocket-es integration — spec
Table of Contents
1. Purpose
Pocket-es is the site's search UI — a ClojureScript BM25 client at
/search. The integration spec is how pocket-es consumes sponsored
inventory without hard-requiring the sponsored module: setting
USE_SPONSORED_RESEARCH=nil genuinely disables sponsored serving,
and pocket-es continues to work.
The integration point is a runtime window._sponsoredResearch
lookup via string-keyed aget (:advanced-compilation-safe). No
compile-time require, no top-level bindings. Interleave happens at
render time — organic results already in hand, sponsored slots
requested from the runtime provider, mixed into the same DOM.
ORACLE: src/pocket_es/ui.cljs (search UI), src/pocket_es/indexer.clj
(index + thumbnails), src/pocket_es/core.cljs (BM25).
2. Requirements
- P1
- Runtime interleave.
render-results!callswindow._sponsoredResearch.Format.interleaveat render time. If the object isn't present, results render organic-only. - P2
- 20 % ratio, non-displacing. Organic order is never
permuted; one sponsored slot after every
⌈n / ⌈n·0.20⌉⌉organic hits. Contract inherited fromsponsored-research.core/interleave-ratio. - P3
- Placeholder-then-swap. When the async idf-tail fetch hasn't
landed,
interleaveemits:placeholderentries so DOM slots are reserved and the swap-in onwal:sponsored-readydoesn't reflow. - P4
- String-keyed
aget. Everywindow._sponsoredResearchaccess uses(aget o "key"), not(.-key o), so:advancedcompilation cannot mangle the interop. - P5
__wal_tracker.rescan()called after render. Newly-mounted slots are picked up by the passive tracker's IntersectionObserver.- P6
- Thumbnails.
pocket-es.indexeremits:thumbper doc when the note's dir hasthumbnail.png/card.png/banner.png(in that order). Rendered at 48×48 with hover-zoom-to-4× on sponsored slots. Only DIR-form notes ship thumbnails. - P7
- Sponsored href. Sponsored slots render with
href"#", =rel"nofollow sponsored"= (WHATWG, n.d.), andclickispreventDefault-ed. The click event fires thesponsored.clicktracker event; the destination is the audit ledger, not the click'shref. - P8
- Prefetch on hover. Organic result hovers may trigger a
<link rel"prefetch">= for the destination (MDN Web Docs, n.d.-b); sponsored slots do NOT prefetch (their destination is#).
3. Contract signature
Pocket-es (ui.cljs):
(render-results! hits query) ; -> DOM mutation
;; -> calls window._sponsoredResearch.Format.interleave when present
;; -> calls window.__wal_tracker.rescan() after mount
(kind->class kind) ; -> CSS class
;; :organic -> "pes-hit"
;; :sponsored -> "pes-hit pes-hit--sponsored"
;; :placeholder -> "pes-hit pes-hit--placeholder"
;; Event dispatch
(js/window.dispatchEvent (js/CustomEvent. "wal:sponsored-ready" ...))
;; -> swap :placeholder entries with real cards
Indexer (indexer.clj):
(build-index) ; -> writes search-index.json
;; each doc entry includes :thumb when banner/card/thumbnail.png present
4. Sponsored slot markup
Every sponsored slot in pocket-es output carries:
data-track-id— stable per creativedata-track-kind—sponsored-product/chumbox-card/sponsored-placeholder(orsearch-resultfor organic)data-track-creative— creative IDdata-track-unit—product/listing/local/arbitrage/onsite_displaydata-track-sponsor— advertiser name if presentdata-track-page-idx/data-track-pool-idx— for infinite-feed positional analysisrel"nofollow sponsored"= on the<a>(WHATWG, n.d.)
The tracker's el-attrs reader picks these up verbatim; adding an
attr means updating tracker.core/element-payload to whitelist it.
5. Thumbnails
pocket-es.indexer walks site/**/index.org and, for each DIR-form
note (index.org inside a directory), checks for thumbnail.png,
card.png, banner.png in that order. First match is emitted as
:thumb in the doc's index entry.
Bare notes (<slug>.org at the top level of site/research/)
don't ship thumbnails. This is enforced at index-build; there is no
runtime fallback. The rule prevents a bare note from leaking its
parent directory's banner.
Rendered at 48×48 in the result list. Hover-zoom-to-4× is applied to sponsored slots only — the visual convention that sponsored slots have larger preview affordances than organic, deliberate.
6. Testing signals surfaced
Every organic hit and every sponsored slot emits impression, dwell, click, and hover events via the tracker. Downstream computable:
- Per-position CTR (organic vs sponsored, faceted by archetype)
- Dwell distribution per hit-kind
- Reverse-scroll rate as an intent proxy
- Sponsored displacement — must always be zero (non-displacement
invariant on
interleave-ratio)
7. Related literature
- BM25 ranking function is the industry-standard baseline for full-text search; the pocket-es implementation follows the reference in Robertson & Zaragoza's Probabilistic Relevance Framework. Not cited inline (not needed for the integration spec) but is the ranking substrate.
rel"sponsored"= HTML Living Standard (WHATWG, n.d.) for the outbound-link disclosure convention.<link rel"prefetch">= behaviour and cache semantics: MDN (MDN Web Docs, n.d.-b).- IntersectionObserver semantics (MDN Web Docs, n.d.-a) for the tracker's slot-visibility check post-mount.
- Sec-Fetch-Site headers (W3C 2021) would let a server-side ranking pipeline distinguish sponsored from organic clicks; currently unused because the destinations are same-origin.
- Cook et al. (Cook, Nithyanand, and Shafiq 2020) documents the practical measurement failure this integration deliberately avoids: no third-party ad server in the loop, no auction, no bid.
8. Cross-references
- sponsored-research/spec.org — supplies
Format.interleave+Format.sample-mixed. - sponsored-display/spec.org — attaches
Displayto the same object; would let pocket-es render display banners between results if the caller wanted them (currently doesn't). - tracker/spec.org —
__wal_tracker.rescan()called after render. - search-mount/spec.org — v7 spec that inverts this dependency (sponsored watches pocket-es via MutationObserver).
- beacon/spec.org — envelope for the search-derived events.
- ORACLE:
src/pocket_es/ui.cljs(search UI + interleave call site)src/pocket_es/indexer.clj(index + thumbnails)src/pocket_es/core.cljs(BM25 scorer)
- [BROKEN LINK: No match for fuzzy expression: *2000–2007: Contextual and search era] on the AdWords / AdSense contextual-search ancestry pocket-es sits in.
- pocket-es — Specification — the machine-readable contract for pocket-es itself, independent of the sponsored-slot integration.
- Building a Search Engine in One Session — the origin story of the pocket-es UI this spec calls into.
- pocket-es Search UX Spec — the dual-input redesign for the search UI where sponsored slots interleave.
9. Open questions
- Bare-
/searchdefault. Landing on/searchwith no?qshows an empty placeholder today. Proposed:match_all sort:date desc size:20labelled "Recent changes — type to search →", with sponsored interleave suppressed (no intent signal yet). Captured in the index.org open-work list. - Infinite-scroll chumbox feed. Spec'd but not landed; design constraints in search-mount. Cardinality is the payload — a "N words · M creatives" meter is the punchline.
- Sponsored slot ratio per query intent. The 20 % ratio is session-uniform; a per-query-shape ratio (higher for commercial- intent queries, lower for research queries) is not built.