Mozilla/5.0 (compatible; Walsh-Research/1.2; +https://wal.sh/bot/)

The robots.txt product token is Walsh-Research (matched case-insensitively, without the /1.2 version suffix).

Walsh-Research powers a daily research brief published at wal.sh/current/research. It reads a curated set of ~60 RSS/Atom feeds — research blogs, mainstream tech news, ML lab publications, formal-methods writing, aviation and radio — and produces a single morning digest of what changed. Sources are named and linked; nothing is republished in full.

The design intent is closer to a personal RSS reader than to a search crawler. It fetches feeds the publisher has already advertised, on a polite schedule, without following links.

• Polls advertised RSS/Atom feeds and a small set of government data endpoints on a daily schedule (one fetch per source per day)
• Curates the resulting headlines — title, URL, source, date — into a daily brief that links back to the original source
• Never follows links out of a fetched page and never crawls recursively

• Does not use fetched content as AI training data
• Does not pass article bodies into an LLM. The brief-writing step reads only the extracted metadata (title + URL + source + date) plus a short feed-provided summary where the feed itself supplies one
• Does not index content for full-text search
• Does not republish article content — only metadata, attribution, and a link back to the original source

The distinction between reads metadata into a curated newsletter and uses your writing as AI input or training data is intentional. If your site declares Content-Signal: ai-train=no or ai-input=no, that is already how this bot behaves; nothing changes on our side and nothing about your content ends up in a model.

Walsh-Research operates as two distinct actor classes with different compliance postures. This matters because the same tool acting as a scheduled feed subscriber and the same tool acting as an ad-hoc crawler are morally different things, and RFC 9309 was written for the latter.

• Feed poller. Scheduled fetch of a specific advertised feed URL (RSS / Atom / JSON Feed). This is what an RSS reader does. Publishing the feed is itself the invitation to poll it, so a generic User-agent: * / Disallow: / rule is not read as covering the advertised feed endpoint. Named Walsh-Research rules, Content-Signal, Cache-Control, ETag / If-None-Match, Retry-After, and the per-host rate limit are all honored.
• Crawler. Ad-hoc fetch of a page that was not advertised as a feed. The full RFC 9309 gate applies here, including User-agent: * groups. Used sparingly, only for pages a human operator specifically pointed at (e.g. verifying a link in the brief resolves).

If you want any Walsh-Research fetching gone, use the site-wide opt-out below; it covers both actor classes.

• Honors robots.txt (RFC 9309), including Crawl-delay, with the feed-poller distinction above
• Honors Content-Signal (ai-train, ai-input, search) as declared by the origin
• Maximum 1 request per second per domain
• Respects Retry-After headers (both integer seconds and HTTP-date)
• Serial requests only (no concurrency within a run)
• Sends conditional If-None-Match / If-Modified-Since so an unchanged feed returns 304 Not Modified and no body is transferred

Current status, 2026-10-18: requests are NOT signed. Walsh-Research identifies itself by User-Agent string alone, which is unverifiable — anyone can send that string. If you need to distinguish genuine Walsh-Research traffic from something claiming the name, the honest answer today is that you cannot do so from the request alone.

What exists so far: the request-signing half of Web Bot Auth is implemented and validated. Signed requests are accepted as well-formed by Cloudflare's test endpoint (HTTP 401 — correct construction, unregistered key). The Ed25519 signing, RFC 9421 signature base, and RFC 7638 key thumbprint all round-trip.

What is missing, and why:

1. A hosted key directory. Verification requires a signed JWKS at /.well-known/http-message-signatures-directory, where the response itself carries Signature / Signature-Input headers with per-request created / expires timestamps. wal.sh is a static site behind Apache and cannot mint those per request. Resolving this means either a signing edge worker or a nightly-regenerated pre-signed directory.
2. Registration. Cloudflare's Verified Bots program is an application, not a self-service toggle, and per their continuous accountability change it commits the operator to ongoing behavioral audit rather than one-time approval. That is a reasonable bar and we would rather meet it before applying than after.

No key is published and no key is in use, so there is nothing to trust yet. This section will name the key directory and thumbprint when there is one. Until then, treat the User-Agent as a courtesy label and use the opt-out paths below, which do not depend on identity verification.

Two paths, both honored. Pick whichever is easier for you.

Add a named rule to your robots.txt:

User-agent: Walsh-Research
Disallow: /

Or email j@wal.sh with the domain(s) you want blocked. You will be added to the published blocklist (checked before robots.txt, so an email opt-out works even if your robots.txt is temporarily unreachable) within a day. No verification, no reason required.

If Walsh-Research fetched something it should not have and you want it removed from the warehouse retroactively, mention that in the same email and it will be purged.

• Compliance spec — the full contract (walsh-research-compliance/v1.4), written to be reimplementable from that document alone
• blocklist.json and its schema — the live opt-out list
• test-fixtures.json — URLs any conformant implementation can fetch to self-verify the gate pipeline
• Reference implementation: jwalsh/tech-crawler (Clojure)