Pocket-es integration — spec

Table of Contents

1. Purpose

Pocket-es is the site's search UI — a ClojureScript BM25 client at /search. The integration spec is how pocket-es consumes sponsored inventory without hard-requiring the sponsored module: setting USE_SPONSORED_RESEARCH=nil genuinely disables sponsored serving, and pocket-es continues to work.

The integration point is a runtime window._sponsoredResearch lookup via string-keyed aget (:advanced-compilation-safe). No compile-time require, no top-level bindings. Interleave happens at render time — organic results already in hand, sponsored slots requested from the runtime provider, mixed into the same DOM.

ORACLE: src/pocket_es/ui.cljs (search UI), src/pocket_es/indexer.clj (index + thumbnails), src/pocket_es/core.cljs (BM25).

2. Requirements

P1
Runtime interleave. render-results! calls window._sponsoredResearch.Format.interleave at render time. If the object isn't present, results render organic-only.
P2
20 % ratio, non-displacing. Organic order is never permuted; one sponsored slot after every ⌈n / ⌈n·0.20⌉⌉ organic hits. Contract inherited from sponsored-research.core/interleave-ratio.
P3
Placeholder-then-swap. When the async idf-tail fetch hasn't landed, interleave emits :placeholder entries so DOM slots are reserved and the swap-in on wal:sponsored-ready doesn't reflow.
P4
String-keyed aget. Every window._sponsoredResearch access uses (aget o "key"), not (.-key o), so :advanced compilation cannot mangle the interop.
P5
__wal_tracker.rescan() called after render. Newly-mounted slots are picked up by the passive tracker's IntersectionObserver.
P6
Thumbnails. pocket-es.indexer emits :thumb per doc when the note's dir has thumbnail.png / card.png / banner.png (in that order). Rendered at 48×48 with hover-zoom-to-4× on sponsored slots. Only DIR-form notes ship thumbnails.
P7
Sponsored href. Sponsored slots render with href"#", =rel"nofollow sponsored"= (WHATWG, n.d.), and click is preventDefault-ed. The click event fires the sponsored.click tracker event; the destination is the audit ledger, not the click's href.
P8
Prefetch on hover. Organic result hovers may trigger a <link rel"prefetch">= for the destination (MDN Web Docs, n.d.-b); sponsored slots do NOT prefetch (their destination is #).

3. Contract signature

Pocket-es (ui.cljs):

(render-results! hits query)                      ; -> DOM mutation
;; -> calls window._sponsoredResearch.Format.interleave when present
;; -> calls window.__wal_tracker.rescan() after mount

(kind->class kind)                                ; -> CSS class
;; :organic     -> "pes-hit"
;; :sponsored   -> "pes-hit pes-hit--sponsored"
;; :placeholder -> "pes-hit pes-hit--placeholder"

;; Event dispatch
(js/window.dispatchEvent (js/CustomEvent. "wal:sponsored-ready" ...))
;; -> swap :placeholder entries with real cards

Indexer (indexer.clj):

(build-index)                                     ; -> writes search-index.json
;; each doc entry includes :thumb when banner/card/thumbnail.png present

4. Sponsored slot markup

Every sponsored slot in pocket-es output carries:

  • data-track-id — stable per creative
  • data-track-kindsponsored-product / chumbox-card / sponsored-placeholder (or search-result for organic)
  • data-track-creative — creative ID
  • data-track-unitproduct / listing / local / arbitrage / onsite_display
  • data-track-sponsor — advertiser name if present
  • data-track-page-idx / data-track-pool-idx — for infinite-feed positional analysis
  • rel"nofollow sponsored"= on the <a> (WHATWG, n.d.)

The tracker's el-attrs reader picks these up verbatim; adding an attr means updating tracker.core/element-payload to whitelist it.

5. Thumbnails

pocket-es.indexer walks site/**/index.org and, for each DIR-form note (index.org inside a directory), checks for thumbnail.png, card.png, banner.png in that order. First match is emitted as :thumb in the doc's index entry.

Bare notes (<slug>.org at the top level of site/research/) don't ship thumbnails. This is enforced at index-build; there is no runtime fallback. The rule prevents a bare note from leaking its parent directory's banner.

Rendered at 48×48 in the result list. Hover-zoom-to-4× is applied to sponsored slots only — the visual convention that sponsored slots have larger preview affordances than organic, deliberate.

6. Testing signals surfaced

Every organic hit and every sponsored slot emits impression, dwell, click, and hover events via the tracker. Downstream computable:

  • Per-position CTR (organic vs sponsored, faceted by archetype)
  • Dwell distribution per hit-kind
  • Reverse-scroll rate as an intent proxy
  • Sponsored displacement — must always be zero (non-displacement invariant on interleave-ratio)

7. Related literature

  • BM25 ranking function is the industry-standard baseline for full-text search; the pocket-es implementation follows the reference in Robertson & Zaragoza's Probabilistic Relevance Framework. Not cited inline (not needed for the integration spec) but is the ranking substrate.
  • rel"sponsored"= HTML Living Standard (WHATWG, n.d.) for the outbound-link disclosure convention.
  • <link rel"prefetch">= behaviour and cache semantics: MDN (MDN Web Docs, n.d.-b).
  • IntersectionObserver semantics (MDN Web Docs, n.d.-a) for the tracker's slot-visibility check post-mount.
  • Sec-Fetch-Site headers (W3C 2021) would let a server-side ranking pipeline distinguish sponsored from organic clicks; currently unused because the destinations are same-origin.
  • Cook et al. (Cook, Nithyanand, and Shafiq 2020) documents the practical measurement failure this integration deliberately avoids: no third-party ad server in the loop, no auction, no bid.

8. Cross-references

  • sponsored-research/spec.org — supplies Format.interleave + Format.sample-mixed.
  • sponsored-display/spec.org — attaches Display to the same object; would let pocket-es render display banners between results if the caller wanted them (currently doesn't).
  • tracker/spec.org__wal_tracker.rescan() called after render.
  • search-mount/spec.org — v7 spec that inverts this dependency (sponsored watches pocket-es via MutationObserver).
  • beacon/spec.org — envelope for the search-derived events.
  • ORACLE:
    • src/pocket_es/ui.cljs (search UI + interleave call site)
    • src/pocket_es/indexer.clj (index + thumbnails)
    • src/pocket_es/core.cljs (BM25 scorer)
  • [BROKEN LINK: No match for fuzzy expression: *2000–2007: Contextual and search era] on the AdWords / AdSense contextual-search ancestry pocket-es sits in.
  • pocket-es — Specification — the machine-readable contract for pocket-es itself, independent of the sponsored-slot integration.
  • Building a Search Engine in One Session — the origin story of the pocket-es UI this spec calls into.
  • pocket-es Search UX Spec — the dual-input redesign for the search UI where sponsored slots interleave.

9. Open questions

  • Bare-/search default. Landing on /search with no ?q shows an empty placeholder today. Proposed: match_all sort:date desc size:20 labelled "Recent changes — type to search →", with sponsored interleave suppressed (no intent signal yet). Captured in the index.org open-work list.
  • Infinite-scroll chumbox feed. Spec'd but not landed; design constraints in search-mount. Cardinality is the payload — a "N words · M creatives" meter is the punchline.
  • Sponsored slot ratio per query intent. The 20 % ratio is session-uniform; a per-query-shape ratio (higher for commercial- intent queries, lower for research queries) is not built.
Cook, John, Rishab Nithyanand, and Zubair Shafiq. 2020. “Inferring Tracker-Advertiser Relationships in the Online Advertising Ecosystem Using Header Bidding.” Proceedings on Privacy Enhancing Technologies (Popets) 2020 (1): 65–82. https://doi.org/10.2478/popets-2020-0005.
MDN Web Docs. n.d.-a. “Intersection Observer Api.” https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API.
———. n.d.-b. “Link Rel=Prefetch.” https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel/prefetch.
W3C. 2021. “Fetch Metadata Request Headers.” https://www.w3.org/TR/fetch-metadata/.
WHATWG. n.d. “Html Living Standard: Link Types (Rel=Sponsored).” https://html.spec.whatwg.org/multipage/links.html#link-type-sponsored.