Arbitraging near-zero-cost inventory into the wal.sh sponsored surface
What sources exist, what they cost, what they leak, what they let you serve

wal.sh's sponsored surface currently serves out of its own /research/ corpus — inventory cost = $0 marginal (writing new notes has an authorship cost but not an ad-acquisition cost). That works but caps at 855 units and produces the pathologies documented in the retrospective: closed ecosystem, no counterparty, ACOS = ∞, 1.3% CTR that decomposes to 0 external clicks.

This document scopes the next arbitrage move: pulling third-party inventory into the same slot template, at zero or near-zero cost per unit acquired, and shipping it through the existing sponsored-research contract without inventing referents.

"Cheapest possible" carries two constraints:

1. Zero dollars. No paid API tier, no chumbox-provider account, no affiliate-program signup that costs money. Free tier only.
2. Zero authority claim. We do NOT tell the origin publisher we are serving them. We do NOT pay them. Every served unit points at the origin's canonical URL; the origin gets the click; we get nothing but observed CTR.

The result is a provably legal rebroadcast: reader sees a card, clicks through, lands on the origin. wal.sh gets no attribution credit and claims none — matches the retrospective §3.4 finding that provenance-at-write-time beats stats-at-audit-time.

Ten sources examined. Four ship criteria per source: shape (what one unit looks like), rate (items/day at zero cost), refresh (poll interval that stays within the free-tier), decay (how fast a unit becomes stale enough to stop serving).

Every source produces units that MUST normalise to the sponsored-research v-final unit shape (spec) before entering the auction. Normalisation is per-source; the contract downstream is shared.

R1. Each source ships a normaliser: (fetch, parse) → [Unit]. The normaliser MUST NOT invent fields. If a source doesn't publish :description, the unit's description is nil (not fabricated prose).

R2. Every produced unit carries :source-kind ∈ #{"rss" "github-api" "npm" "pypi" "arxiv" "wikipedia" "public-sitemap" "gov-wire" "creator-platform" "affiliate"}. Downstream (auditor, ledger) reads this to classify.

R3. Every produced unit carries :origin-url — the canonical URL on the source publisher. Every click routes there directly, no redirect through wal.sh (except the necessary tracker fire).

R4. Every produced unit carries :fetched-at (ISO-8601). Decay policy per source (§4) reads this.

R5. Per-source rate limiter: each fetcher records last-fetch-ts and refuses to fetch again inside the source's free-tier window. No "burst then throttle" — spread requests over the interval.

R6. Per-source cache: normalised units cached in data/arbitrage/<source>/<date>.jsonl. Same one-line-per-day pattern as data/event-flow/. Diffable, small, greppable.

Not all inventory ages the same way. Serving stale inventory is worse than serving nothing.

R7. The auction MUST filter out units with (:fetched-at + decay window) < now. Silent drop; not an error.

R8. Refresh cadence per source lives in config/arbitrage-sources.edn (§7). Adjustable without redeploy.

Not every source fits every query. The auction ranks sources per query so a dinner query doesn't get an arXiv paper.

R9. Per-source affinity vector: fixed weights over the tracker's query shape. Sketch:

R10. Query classification via the tracker's existing pocket-es tokeniser + a small keyword→category map (/site/static/query-categories.edn — to author). Category vector × source-affinity vector → per-source eligibility score. Top-scoring source wins the slot.

R11. Ties broken by freshness (newer fetched-at wins), never by random. Determinism is required for the auction proof to hold — same argument as retrospective §3.3 GSP over a single-bidder set.

We serve without paying. That constrains what we can claim.

R12. Every served unit's tracker event MUST carry :paid false — the ad system did not pay any counterparty to acquire this inventory. This is the load-bearing separator from a real retail-media network.

R13. Click-through routes DIRECTLY to :origin-url. wal.sh does not intermediate; the origin publisher sees a normal referer from https://wal.sh/. No cookie planting, no redirect, no affiliate parameter appended.

R14. No revenue attribution table. There is no revenue. The claimedCreditRatio (retrospective §3.4) is undefined; the ledger records the impression + click but not a synthetic $ value.

R15. Take-down: any origin publisher can request removal of their URL from the arbitrage index by adding a wal.sh-specific Disallow line to their robots.txt:

User-agent: wal.sh-arbitrage Disallow: /

The fetchers respect this exactly. Precedent: our own bot-compliance spec for how we expect to be treated when we're the crawler.

Single EDN file the operator edits. Every source has: :enabled boolean, :refresh-secs, :decay-days, :affinity map, :max-items-per-fetch.

{:sources
 {:rss-hn         {:enabled true  :refresh-secs 300  :decay-days 1
                   :feed-url "https://hnrss.org/frontpage"
                   :affinity {:dev 0.8 :research 0.5 :news 0.3}}
  :rss-lobsters   {:enabled true  :refresh-secs 300  :decay-days 1
                   :feed-url "https://lobste.rs/rss"
                   :affinity {:dev 0.9 :research 0.4}}
  :github-trending {:enabled true :refresh-secs 3600 :decay-days 7
                    :api-endpoint "/search/repositories"
                    :params {:sort "stars" :order "desc" :q "created:>2026-08-11"}
                    :affinity {:dev 1.0 :research 0.3}}
  :npm-updated    {:enabled true  :refresh-secs 1800 :decay-days 3
                   :feed-url "https://registry.npmjs.org/-/rss/updated"
                   :affinity {:dev 1.0}}
  :pypi-updated   {:enabled true  :refresh-secs 1800 :decay-days 3
                   :feed-url "https://pypi.org/rss/updates.xml"
                   :affinity {:dev 1.0}}
  :arxiv-cs-pl    {:enabled true  :refresh-secs 86400 :decay-days 30
                   :api-endpoint "http://export.arxiv.org/api/query"
                   :params {:search_query "cat:cs.PL" :sortBy "submittedDate"}
                   :affinity {:research 1.0 :dev 0.5}}
  :wikipedia-any  {:enabled true  :refresh-secs 3600 :decay-days 365
                   :api-endpoint "https://en.wikipedia.org/w/api.php"
                   :affinity {:research 0.8 :news 0.3 :cooking 0.5 :local 0.3}}
  :gov-uk         {:enabled false :refresh-secs 3600 :decay-days 30
                   :feed-url "https://www.gov.uk/government/publications.atom"
                   :affinity {:news 0.8 :financial 0.5}}
  :product-hunt   {:enabled false :refresh-secs 3600 :decay-days 14
                   :feed-url "https://www.producthunt.com/feed"
                   :affinity {:dev 0.6 :creator 0.9}}}

 :global
 {:user-agent "wal.sh-arbitrage/1.0 (contact: j@wal.sh)"
  :cache-dir  "data/arbitrage/"
  :respect-robots-txt true}}

Single per-note Makefile, tangled per §12, drives fetch + normalise

• deploy. Same pattern as

vegetarian-recipes-network.

# arbitrage-inventory Makefile
# Lives in site/research/browser-adtech-infra/arbitrage-inventory-sources/
# Fetches, normalises, deploys arbitraged ad inventory into wal.sh.

CACHE_DIR := ../../../../data/arbitrage
CFG       := config/arbitrage-sources.edn
BB        := bb

.PHONY: help fetch-all fetch normalise validate corpus deploy audit clean

help: ## Show targets
        @grep -E '^[a-zA-Z_-]+:.*## ' $(MAKEFILE_LIST) | \
                awk 'BEGIN{FS=":.*## "}{printf "  %-14s %s\n", $$1, $$2}'

fetch-all: ## Fetch every enabled source per its refresh cadence (respects rate limits)
        @$(BB) scripts/fetch-all.clj --config $(CFG) --cache $(CACHE_DIR)

fetch: ## Fetch one source: gmake fetch SRC=rss-hn
        @if [ -z "$$SRC" ]; then echo "Usage: gmake fetch SRC=<source-key>"; exit 1; fi
        @$(BB) scripts/fetch-one.clj --config $(CFG) --source $$SRC --cache $(CACHE_DIR)

normalise: ## Normalise cached JSONL into the sponsored-research unit shape
        @$(BB) scripts/normalise.clj --cache $(CACHE_DIR) --out $(CACHE_DIR)/normalised.jsonl

validate: normalise ## Run the sponsored-research contract validator on every normalised unit
        @$(BB) scripts/validate-units.clj --in $(CACHE_DIR)/normalised.jsonl

corpus: validate ## Emit arbitrage-corpus.json (browser-side auction inventory)
        @$(BB) scripts/emit-corpus.clj --in $(CACHE_DIR)/normalised.jsonl > \
                ../../../static/js/adtech/arbitrage-corpus.json
        @printf "✓ arbitrage-corpus.json: %s units, %s bytes\n" \
                "$$(jq '. | length' ../../../static/js/adtech/arbitrage-corpus.json)" \
                "$$(wc -c < ../../../static/js/adtech/arbitrage-corpus.json)"

deploy: corpus ## Push corpus + config to prod
        @rsync -avz ../../../static/js/adtech/arbitrage-corpus.json \
                www_wal_sh@vps47738.dreamhostps.com:~/wal.sh/static/js/adtech/arbitrage-corpus.json
        @echo "✓ arbitrage corpus deployed"

audit: ## Yesterday's arbitrage-serve breakdown by source-kind
        @$(BB) ../../../../scripts/tracker-intent.clj \
                --event sponsored.impression --format counts \
                | grep -E "source-kind=|source_kind="

clean: ## Wipe cache (forces re-fetch on next run — respect rate limits!)
        @rm -rf $(CACHE_DIR)
        @echo "✓ cache cleared (rate-limit clock unchanged)"

Every fetch appends to the audit ledger. Every serve does not — the serve is already logged by the tracker.

R16. On successful fetch: append verdict=attributed subject=arbitrage-<source>-<date> to .verify/chain.jsonl. Verifier arbitrage-fetcher. The attributed verdict marks the units as sourced-not-verified — we did not confirm the origin publisher meant to be indexed.

R17. On a take-down request (§6 R15): append verdict=corrected with :FINDING naming the removed URL.

R18. Weekly rollup: a gmake arbitrage-week-audit (deferred) would sum served-by-source and compare to fetched-by-source to surface fetch-serve gaps (a source we're pulling but never serving is waste; either boost its affinity or disable it).

Named boundaries. Adding a new source that violates one of these requires a spec bump.

• Paywalled content. Even if the paywall is bypassable via a cached-view trick, the served card MUST point at the paywall page, not the bypass. If the origin publisher can only monetise via subscription, we do not undercut them.
• Personal blogs without a public feed. A blog that publishes a URL but no RSS/Atom is not opting in to syndication; scraping is out.
• Content behind robots.txt Disallow. R15 respect goes both ways.
• Reader's own private data (Gmail, Google Drive, etc.). No.
• Chumbox provider inventory (§2.10). No.
• AI-generated content mirrors. If a source's feed is 100% LLM-shovelware, it's ad-shaped noise, not ad-shaped signal. Detection heuristic: >70% of the source's daily items have identical opening-3-tokens, or the source's URL structure follows a shovelware pattern. Silent drop.

The claim: arbitraged free-tier inventory can be served through the existing sponsored-research contract without inventing units, without paying the origin publisher, without misrepresenting the attribution, and at zero marginal cost per served card.

Fails if any of:

• CTR on arbitrage-served slots drops to 0% for a whole day (users ignore arbitraged content entirely — inventory quality gap)
• Any source produces >1 unit that fails the sponsored-research validate gate (contract leakage — normalisers are lying)
• Any served unit's :origin-url resolves to a wal.sh URL (self-loop — arbitrage is now cannibalising organic; kill)
• Any origin publisher requests removal AND we fail to honour it within 24h (R15 compliance breach — hard-fail; disable the source)
• The daily fetch cost (bandwidth + API calls) exceeds $0.01 cumulative across all sources (not "cheapest possible" anymore)

Test after 30 days of operation on a single-source config (start with :rss-hn only): if any of the above trip, cut scope.

1. Author fetcher for :rss-hn only. Zero rate-limit surface, single feed, well-understood shape. Ship the whole pipeline end-to-end for one source before adding more.
2. Wire the sponsored-research contract's unit constructor to accept :source-kind as a valid extra field (§3 R2). Update validate.
3. Add arbitrage-fetch-daily to the daily cron chain (after event-flow-snapshot).
4. Serve :rss-hn units in one of the two sponsored slots on /search; grade after 7 days against §11 refutation.
5. If green: add :lobsters, then :github-trending.
6. If red at 7 days: keep the doc, kill the pipeline. Ledger the finding as corrected against this spec.
7. Do NOT skip to §2.10 chumbox scraping; that's the failure mode this whole document exists to argue against.

• sponsored-research/spec.org — the contract every arbitrage unit ships through
• vegetarian-recipes-network/index.org — the opposite endpoint: closed-taxonomy, tightly curated inventory
• retrospective/index.org — §3.4 (provenance at write time), §3.8 (closed-ecosystem economics)
• attribution-audit/spec.org — the consumer of every served-arbitrage event
• bots/compliance-spec.org — how we crawl (mirrors R15 how we expect to be treated)
• event-flow-viz/index.org — the surface where arbitrage fill rate will show up