Mozilla/5.0 (compatible; Walsh-Research/1.2; +https://wal.sh/bot/)

The robots.txt product token is Walsh-Research (matched case-insensitively, without the /1.2 version suffix).

Walsh-Research powers a daily research brief published at wal.sh/current/research. It reads a curated set of ~60 RSS/Atom feeds — research blogs, mainstream tech news, ML lab publications, formal-methods writing, aviation and radio — and produces a single morning digest of what changed. Sources are named and linked; nothing is republished in full.

The design intent is closer to a personal RSS reader than to a search crawler. It fetches feeds the publisher has already advertised, on a polite schedule, without following links.

• Polls advertised RSS/Atom feeds and a small set of government data endpoints on a daily schedule (one fetch per source per day)
• Curates the resulting headlines — title, URL, source, date — into a daily brief that links back to the original source
• Never follows links out of a fetched page and never crawls recursively

• Does not use fetched content as AI training data
• Does not pass article bodies into an LLM. The brief-writing step reads only the extracted metadata (title + URL + source + date) plus a short feed-provided summary where the feed itself supplies one
• Does not index content for full-text search
• Does not republish article content — only metadata, attribution, and a link back to the original source

The distinction between reads metadata into a curated newsletter and uses your writing as AI input or training data is intentional. If your site declares Content-Signal: ai-train=no or ai-input=no, that is already how this bot behaves; nothing changes on our side and nothing about your content ends up in a model.

Walsh-Research operates as two distinct actor classes with different compliance postures. This matters because the same tool acting as a scheduled feed subscriber and the same tool acting as an ad-hoc crawler are morally different things, and RFC 9309 was written for the latter.

• Feed poller. Scheduled fetch of a specific advertised feed URL (RSS / Atom / JSON Feed). This is what an RSS reader does. Publishing the feed is itself the invitation to poll it, so a generic User-agent: * / Disallow: / rule is not read as covering the advertised feed endpoint. Named Walsh-Research rules, Content-Signal, Cache-Control, ETag / If-None-Match, Retry-After, and the per-host rate limit are all honored.
• Crawler. Ad-hoc fetch of a page that was not advertised as a feed. The full RFC 9309 gate applies here, including User-agent: * groups. Used sparingly, only for pages a human operator specifically pointed at (e.g. verifying a link in the brief resolves).

If you want any Walsh-Research fetching gone, use the site-wide opt-out below; it covers both actor classes.

• Honors robots.txt (RFC 9309), including Crawl-delay, with the feed-poller distinction above
• Honors Content-Signal (ai-train, ai-input, search) as declared by the origin
• Maximum 1 request per second per domain
• Respects Retry-After headers (both integer seconds and HTTP-date)
• Serial requests only (no concurrency within a run)
• Sends conditional If-None-Match / If-Modified-Since so an unchanged feed returns 304 Not Modified and no body is transferred

Two paths, both honored. Pick whichever is easier for you.

Add a named rule to your robots.txt:

User-agent: Walsh-Research
Disallow: /

Or email j@wal.sh with the domain(s) you want blocked. You will be added to the published blocklist (checked before robots.txt, so an email opt-out works even if your robots.txt is temporarily unreachable) within a day. No verification, no reason required.

If Walsh-Research fetched something it should not have and you want it removed from the warehouse retroactively, mention that in the same email and it will be purged.

• Compliance spec — the full contract (walsh-research-compliance/v1.4), written to be reimplementable from that document alone
• blocklist.json and its schema — the live opt-out list
• test-fixtures.json — URLs any conformant implementation can fetch to self-verify the gate pipeline
• Reference implementation: jwalsh/tech-crawler (Clojure)