Vocabulary binding — the taxonomy is the targeting layer
controlled-vocabulary-v1 × sponsored-contract-v2
Table of Contents
- 1. 1. The relationship
- 2. 2. The two vocabularies are already correctly split
- 3. 3. The objective functions coincide
- 4. 4. What convergence does and does not threaten
- 5. 5. New invariants
- 6. 6. The card made the taxonomy user-visible
- 7. 7. Gate ordering
- 8. 8. What the ad system does not get to ask for
- 9. 9. Recursion
- 10. Cross-references
1. 1. The relationship
A PLP's ad targeting is bound to the retailer's merchandising taxonomy. Criteo's
delivery call carries filters=(price,le,500) — faceted filtering over the
retailer's own attributes — and its keyword targeting runs on the retailer's
category terms. The ad system does not own the taxonomy; it consumes it.
wal.sh has the same structure. The auction's matched (I6) is
doc.keywords ∩ query, so sponsored relevance is a function of vocabulary
quality and nothing else. Every pathology in the folksonomy propagates directly
into which units can serve.
This was not designed. It was discovered by building the auction on top of an unrevised vocabulary and watching it behave.
2. 2. The two vocabularies are already correctly split
controlled-vocabulary-v1 §3.2 gives #+FILETAGS and #+KEYWORDS different
contracts — validated against a term list versus unvalidated free text. That
split maps cleanly onto the two things retail media needs, and they are not the
same thing:
| Field | Vocabulary contract | Ad system role | Retail analogue |
|---|---|---|---|
#+FILETAGS |
controlled, 47 terms, build-validated | targeting keys — faceted eligibility | category tree + attributes |
#+KEYWORDS |
free text, convergence-normalized | matching — BM25 recall and matched display |
search relevance signal |
So: eligibility filters on facets, relevance matches on keywords. A sponsored slot
scoped to domain:formal-methods draws candidates by FILETAGS; which of those
candidates wins is decided by KEYWORDS overlap with the query.
The current implementation conflates them — Auction.quality reads keywords
for both jobs, which is why a slot cannot be scoped at all.
3. 3. The objective functions coincide
controlled-vocabulary-v1 §1 sets the operational goal from chip UX: chips that connect 2–25 documents each. A chip resolving to 61 documents is noise; one resolving to a single document is isolation.
The auction independently wants the same band, for a different reason:
| Term frequency | Chip behaviour | Auction behaviour |
|---|---|---|
| df = 1 (2,777 terms, 81%) | isolation | document can essentially never serve — I6 requires non-empty matched |
| df 2–25 | usable | discriminable candidate set; quality score separates |
df = 61 (clojure) |
noise | 61 candidates tie on relevance; GSP ranks on bid, which is deterministic noise |
Two independently derived targets landing on the same interval is worth noticing. The chip goal and the ad-targeting goal are the same objective function, because both are asking a term to partition the corpus informatively. That makes the vocabulary work structural for the ad system rather than adjacent to it.
4. 4. What convergence does and does not threaten
The arbitrage unit samples idf hapax (idf 6.343, ~3,320 alphabetic terms).
Reducing the #+KEYWORDS singleton ratio from 81% toward <15% sounds like it
would collapse that pool.
It does not. The idf map is over indexed tokens — predominantly body text —
while convergence normalizes the #+KEYWORDS field. squoze, folktrash,
reuleaux are body-text hapax and survive vocabulary work untouched.
Different consumers, different vocabularies:
| Consumer | Reads | Affected by convergence |
|---|---|---|
Auction.quality / I6 |
doc.keywords |
yes — directly |
| Slot eligibility (proposed) | #+FILETAGS |
yes — it is the target state |
| Arbitrage sampling | idf over body tokens |
no |
_srMarks |
_id |
no |
Worth stating explicitly, because the intuition runs the other way.
5. 5. New invariants
- I15 Targeting validity
- a slot's scope, if present, MUST be a
facet:termpair in the pinned vocabulary version. A slot scoped to a retired or misspelled term fails closed rather than serving unscoped. - I16 Vocabulary pin
- every unit MUST carry
vocabularyVersion, and itsmatchedterms MUST validate under that version. Without this, a vocabulary v2 silently changes what serves and no ledger event records why. - I17 Serviceability
- a document whose keywords are all df = 1 MUST NOT be admitted as inventory at B1. It cannot satisfy I6 for any realistic query, so admitting it produces a line item that never wins — inventory that is not inventory.
I17 is the one that changes the catalogue. Against the current corpus it would exclude a substantial fraction of the 841 eligible documents, and that exclusion count is a useful vocabulary-health metric in its own right: the number of documents too weakly tagged to advertise.
6. 6. The card made the taxonomy user-visible
The unit renders matched: retrieval, memory. That is an internal taxonomy
rendered as a public surface.
Consequences the vocabulary document did not have to consider before:
- Near-duplicate fragmentation now appears on cards. A unit showing
matched: agentic systembeside one showingmatched: agentic-systemsexposes the pathology directly rather than burying it in a chip list. - Sentence-length keywords ("best practices for jquery performance") break the card layout, not just the chip row.
- Uppercase variants render inconsistently where the BM25 tokenizer lowercased them anyway — §6.3's argument, now with a visual symptom.
Cheap mitigation, no vocabulary change required: constrain the rendered
matched to terms of ≤3 words and lowercase them at render. But it is a patch
over a corpus defect, and §6.3 already prescribes the real fix.
7. 7. Gate ordering
Vocabulary violations are upstream ad defects, so they must fail first. A contract check that passes because the vocabulary is broken is a false green.
.PHONY: gate
gate:
bb scripts/check-headers
python3 scripts/check_vocabulary.py # I15 term list validity
node bin/check-inventory.mjs # I17 serviceability
node bin/check-contract.mjs # I1-I7, I16 pin
node bin/check-plates.mjs # asset acceptance
check_vocabulary.py (controlled-vocabulary-v1 §7) becomes a dependency of the
contract gate rather than a sibling of it.
8. 8. What the ad system does not get to ask for
The taxonomy exists for the corpus, not for the auction. Three demands the ad system might make and should be refused:
- Do not add terms to improve fill. §9's four-part test governs new terms; "a sponsored slot would serve more often" is not one of the four.
- Do not retire hapax keywords to raise the singleton ratio. §6.2 phase 3 already forbids removing location keywords for search reasons; the same logic covers everything else. Unservable documents are I17's problem, not the vocabulary's.
- Do not version the vocabulary on the ad system's schedule. §11 states the vocabulary does not version in place — v2 is a new document plus a migration pass. I16 exists so the contract can pin a version and absorb that on its own cadence.
9. 9. Recursion
research/2026-keyword-vocabulary-convergence/controlled-vocabulary-v1 is unit
#8 in the sponsored inventory. The document specifying the targeting taxonomy
is itself advertised through it, and its own matched terms are drawn from the
vocabulary it defines. Its keywords — taxonomy, controlled-vocabulary,
folksonomy, information-retrieval, org-mode — are in the 2–25 band, so
it serves cleanly. It passes its own test.
Refutation condition. The claim is that chip density and auction discriminability are the same objective, so vocabulary work pays the ad system for free. It fails if the 2–25 band turns out to be right for chips and wrong for the auction — most plausibly at the low end, where a df = 2 term gives a perfectly good chip and a candidate set of two, which the quality score cannot rank meaningfully. Test by computing, for every keyword, chip utility (df in 2–25) against auction utility (candidate-set entropy after quality scoring). If the two curves peak at different df, they are separate objectives that happen to overlap, and I17's threshold must be derived from the auction curve rather than inherited from the chip target.
10. Cross-references
- retrospective/index.org — the build's own review
- sponsored-research/spec.org — v-final + v2 contract; I6 lives here
- controlled-vocabulary-v1 — the taxonomy this binds against
- creative-ops/diagrams.org — the pipeline the gate lives inside