Every request MUST send this exact User-Agent:

Mozilla/5.0 (compatible; Walsh-Research/1.0; +https://wal.sh/bot/)

The product token used for robots.txt matching is Walsh-Research (case-insensitive). The +URL MUST resolve to a public policy page.

(def ua-token "Walsh-Research")
(def default-headers
  {"User-Agent" "Mozilla/5.0 (compatible; Walsh-Research/1.0; +https://wal.sh/bot/)"})

The schema is reusable. If you are building a similar bot compliance system, the walsh-research-blocklist/v1 contract and its JSON Schema are available at the URLs above under no restrictions.

A tool MUST fetch /robots.txt for each host, cache it (24h SHOULD), and obey it for the Walsh-Research token. Group selection MUST follow RFC 9309:

• If one or more groups name our token, the single most specific group applies (longest matching User-agent value); the * group is ignored.
• Else the * group applies.
• Else nothing is disallowed.

A named group therefore overrides * in both directions: it can block us where * allows, and exempt us where * blocks. A tool MUST support Crawl-delay within the selected group (see R4). On any fetch/parse error a tool SHOULD fail open (allow).

;; select the most-specific matching group, then test the path prefix
(defn robots-allowed? [url]
  (let [g (select-robots-group (fetch-robots url) ua-token)]   ; nil | {:disallows [..] :crawl-delay s}
    (not (some #(str/starts-with? (path-of url) %) (:disallows g)))))

Opt-out via robots (what we honor):

User-agent: Walsh-Research
Disallow: /

Some opt-outs arrive out-of-band (email to j@wal.sh). These live in one published JSON contract that every Walsh-Research tool MUST consult, so an opt-out is honored uniformly across bots and languages. This is in addition to robots.txt and is checked first.

Contract walsh-research-blocklist/v1:

{
  "contract": "walsh-research-blocklist/v1",
  "updated":  "2026-05-23T00:00:00Z",
  "operator": "Jason Walsh",
  "contact":  "j@wal.sh",
  "policy":   "https://wal.sh/bot/",
  "refresh":  "PT6H",
  "blocked": [
    { "domain": "example.com", "added": "2026-05-23", "reason": "email opt-out" }
  ]
}

Requirements:

• A tool MUST fetch the blocklist and skip any request whose host equals or is a subdomain of a listed domain (example.com blocks www.example.com).
• A tool SHOULD validate the document against the published blocklist.schema.json before adopting it, and MUST NOT adopt a document that fails validation (retain the previous list instead).
• A tool SHOULD cache the list for the contract's refresh duration (ISO-8601). Timing is dictated by the data source, not hard-coded.
• On fetch failure or 404 a tool MUST retain the last-known list rather than clearing it — a transient outage MUST NOT silently un-block an opt-out. (A first-ever fetch failure yields an empty list = fail open.)

(defn blocked? [url]                          ; host apex + subdomain match
  (let [{:keys [domains]} (cached-blocklist)] ; validated against schema; TTL from :refresh
    (domain-blocked? (host-of url) domains)))

• Requests MUST be serial per process — no concurrent connections.
• A tool MUST NOT exceed one request per second per domain.
• A tool MUST honor a host's robots Crawl-delay: the spacing before the next request to a host is max(1 second, Crawl-delay).
• The first request to a previously-unseen host MUST NOT be delayed.

(def min-request-interval-ms 1000)            ; <= 1 req/s/domain

(defn throttle! [url]                         ; block until this host's slot is free
  (let [host (host-of url)
        interval (max min-request-interval-ms (crawl-delay-ms url)) ; robots Crawl-delay
        last (get @host-last-request host)    ; nil when unseen -> no wait
        wait (max 0 (- (+ (or last (now)) interval) (now)))]
    (when (pos? wait) (sleep wait))
    (swap! host-last-request assoc host (+ (now) wait))))

On HTTP 429 or 503 a tool MUST back off and retry with exponential backoff plus jitter, and MUST respect a Retry-After header when present (use it instead of the computed delay). A bounded number of retries SHOULD be used before giving up.

(defn with-backoff [thunk]                     ; thunk -> http response
  (loop [n 0]
    (let [resp (thunk)]
      (if (and (#{429 503} (:status resp)) (< n max-retries))
        (do (sleep (or (retry-after-ms resp) (* base-ms (Math/pow 2 n) (rand))))
            (recur (inc n)))
        resp))))

• A tool MUST NOT follow links or crawl recursively. It fetches a declared set of targets only.
• A tool MUST NOT download sub-resources (images, scripts, stylesheets).
• A tool MUST NOT train models on, index for search, or republish fetched content. Extract metadata, not content.
• A tool MUST NOT store content beyond what the analysis needs.

A tool SHOULD send conditional requests using stored validators (If-None-Match / If-Modified-Since) and treat 304 Not Modified as "no new data" without re-processing. A tool SHOULD dedup results by canonical URL (strip #fragment and trailing slash) so repeated runs accumulate first-seen items only.

A tool SHOULD run infrequently (daily or less) per source.

A tool MUST NOT extract content by scraping arbitrary HTML with per-site selectors – an unbounded maintenance burden, and easy to do impolitely.

For a feed-less source a tool SHOULD obtain a structured representation, preferring, in order:

1. Content negotiation — request Accept: text/markdown and use the body only when the server returns a markdown content-type (e.g. Mintlify-hosted docs, wal.sh).
2. llms.txt convention — fetch /llms.txt at the host root: a curated markdown index of the site's key pages (https://llmstxt.org).

Extract metadata generically from the markdown (links, headings); a tool MUST NOT rely on per-site structure. This keeps feed-less coverage clean and consistent with R6 (metadata, not content). A source that offers neither a feed nor markdown is left out rather than scraped.