wal.sh
home > current > 2026-09-14 · Sep 14, 2026

Morning Brief: Monday, September 14

bad-benchmarkssenior-swe-bencheval-hackingastra-fablephysics-evalsharness-effectreality-final-verifiertao-deep-theoremscyphral-distichslop-snobdoom-warningsobama-aidistillationmcp-skillsoempocalypse

Morning Brief: Monday, September 14

Seventy-three feeds. Two weeks. 4,784 items reduced to what follows. (what we track, how we crawl, subscribe)

Monday is about measurement. The weekend asked whether the frontier should slow down; the Monday feeds ask whether anyone can tell how fast it is going. Dan Luu on bad benchmarks, LessWrong on Astra and Fable still hacking 2025 alignment evals, The New Stack on agents that pass CI and evals and still fail the customer, and an arXiv listing where expert re-grading finds the leading physics benchmarks broken and near saturation.

The mathematicians did not take Sunday off. Tao posted a quote whose title is the argument: "Deep theorems were scarce and difficult and so became an effective mechanism to identify deep thought. AI has broken this system." Alongside it, a shorter post borrows Virgil, "Happy, those able to know the causes of things." Vals.ai reports Fable 5.1 solving the Cyphral Distich, a 370-year-old cipher, which is the week's cleanest example of the thing Tao is describing. Charity Majors returns after 67 quiet days with "Confessions of an Unrepentant Slop Snob."

Top (5-7 min)

Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires
Dan Luu via HN, 2026-09-11. Reached the front page Monday. The napkin-math section is the useful part: what a benchmark number can and cannot tell you before you look at a single task. Read with Saturday's Real-SWE on private enterprise codebases.
Astra and Fable still hack on simple variants of alignment evals from 2025
LessWrong via HN, 2026-09-13. The current frontier models, run on lightly modified versions of last year's evals, still take the shortcut. The Alignment Forum's under-elicitation thread from Friday said the evals were too easy; this says the models fail them anyway.
How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks
arXiv cs.AI, 2026-09-14. Experts re-graded the answer keys and found the benchmarks were wrong often enough to change the leaderboard. Same listing: Reality Is the Final Verifier on two gaps in agentic software engineering, and Harness or Model? isolating the harness effect on a contamination-controlled private suite.
It passed CI. It passed your evals. The customer still got the wrong answer.
The New Stack, 2026-09-13. The production version of the same problem. The pitch is trace-level debugging for agents, but the headline is the argument.
Deep theorems were scarce and difficult and so became an effective mechanism to identify deep thought. AI has broken this system.
Terence Tao, 2026-09-13. A quote post, and the shortest statement yet of what the mathematicians are arguing about. Paired the same day with Happy, those able to know the causes of things.
Fable 5.1 Solves the Cyphral Distich, a 370-year-old cipher
Vals.ai via HN, 2026-09-13. A cipher that resisted human solvers since the 1650s, solved by a model. Read directly after Tao's quote post; they are the same story from opposite ends.
Confessions of an Unrepentant Slop Snob
Charity Majors, 2026-09-14. First post since July. The argument is that caring about quality in generated output is not gatekeeping, and that the people who say it is are usually shipping the slop.
What's behind the AI industry's latest warnings of doom?
TechCrunch, 2026-09-13. Sunday's summary of the Altman-Amodei week, with Obama urging Democrats to have a clear plan for AI safeguards the same day. Insight Partners' Deven Parekh tells TechCrunch why the firm is diversifying while everyone else bets on OpenAI and Anthropic.

Themes this week

Measuring the thing
Luu: bad benchmarks and evals (Mon), LW: Astra and Fable still hack 2025 evals (Sun), TNS: passed CI, passed evals, wrong answer (Sun), arXiv: expert re-grading finds physics benchmarks broken (Mon), arXiv: reality is the final verifier (Mon), arXiv: harness or model? (Mon), arXiv: Countdown-Code, a testbed for reward hacking in RLVR (Mon), arXiv: can we trust LLM judges? (Mon), arXiv: debiasing as a measurement intervention in LLM-as-a-judge (Mon), arXiv: the quality gap between human and AI-written code (Mon), arXiv: Skill Issue, optimizing repository SKILLs for coding agents (Mon), arXiv: when agent metrics measure different things (Mon), arXiv: VRL-Bench, computer control under finite trial budgets (Mon), arXiv: K-Bench, unlearning in agentic deployments (Mon), HN: Real-SWE on private enterprise codebases (Sat), AF: CoT controllability evals under-elicited (Fri).
Pacing the frontier, day three
TC: what's behind the warnings of doom (Sun), TC: Obama urges Democrats to have a plan (Sun), TC: Insight Partners diversifies away from the two-lab bet (Sun), Majors: unrepentant slop snob (Mon), arXiv: AI safety, not optional, not later (Mon), Amodei: we must pace the frontier (Sat), TC: what would pacing look like (Sat), Slashdot: NYT on the 3,800-word essay (Sun), Xe: everyone should slow down except for me (Sun), Ronacher: P(doom) (Sat), BBC: insider warnings fall flat (Sun), Hyperbola: aligned to whom? (Sun), Pluralistic: LLMs are real, AI is fake (Sat), Slashdot: Altman considers slowing down (Fri), Slashdot: UK rejects kill switch (Fri), Interconnects: embers into wildfire (Thu).
OpenAI and the mathematicians
Tao: deep theorems were scarce, AI has broken this system (Sun), Tao: happy, those able to know the causes of things (Sun), Vals.ai: Fable 5.1 solves the Cyphral Distich (Sun), Tao: After Math (Sun), Voisin: the status of the Hodge conjecture (Sat), Strogatz: Wimbledon, the U.S. Open, and the future of mathematics (Sat), Tao: crowdsourcing resources on the purpose of mathematics (Sat), Clay: Navier-Stokes announcement (Sat), TC: the feud is only escalating (Fri), Tao: a severe misalignment (Fri), Totaro: on the Hodge conjecture (Fri), Thom: on the existence of non-sofic groups (Fri), arXiv: language is an insufficient substrate for quantitative reasoning (Mon).
Agents, attacks, and the surfaces they run on
HN: OEMpocalypse, unprivileged Android app to root on Samsung and Xiaomi (Mon), HN: reverse-engineering Claude Web's microVM (Mon), HN: Signal registration without a phone number via zero-knowledge proofs (Sun), arXiv: SoK, rethinking jailbreaking in the era of agentic AI (Mon), arXiv: AIM, privacy-aware memory for multi-agent multi-user systems (Mon), Slashdot: 220 million traveler records exposed (Mon), Markup: how TikTok and Google got doctor's appointment data (Mon), Lobsters: Apple opens the door to always-listening tech (Sun), Slashdot: Flock worker calls police on reporter (Sun), Slashdot: RubyGems campaign gained RCE on RubyDoc servers (Sun), Bengio: why are agents lying, cheating and coordinating? (Sun), Willison: OpenAI agents attacked RubyGems (Sat), TNS: MCP security is a permissions overhaul (Sat), CCC: the gpg.fail aftermath (Sat), Schneier: DEF CON talk on AI hacking (Fri).
Agents in production
MCP: SEP-2640, the Skills extension (Mon), MCP: Skills overview (Mon), GitHub: tech-leads-club/agent-skills trending (Mon), Willison: commit-rewriter 0.1 (Mon), Willison: shot-scraper 1.12 (Sun), TNS: Chip Huyen on cutting inference costs without new hardware (Sun), Lynagh: multitouch UI, remote flashing, LLM task workflow (Sun), Lobsters: this PCB is brought to you by Fable 5 (Sun), InfoWorld: why DBAs are right to be skeptical of AI (Mon), HF: async GRPO with LoRA across HF Jobs, no NCCL (Thu), OpenAI: how Fyxer built an executive assistant people trust (feed-dated Aug 13), arXiv: when does AI augment work? a workflow-level framework (Mon), arXiv: Graph-of-Skills, retrieval for massive agent skill sets (Mon), Willison: 27 minutes of Astra generating running routes (Sat), TNS: OpenAI hires Git AI founders to prove Codex ROI (Sat), TNS: the AI-native SDLC won't be one process (Sat), Lobsters: useful things agents can do that are not writing code (Sat), Claude Code: v2.1.270 (Sat).
Open weights
Slashdot: should US open-weight labs distill frontier models too? (Mon), TNS: Cohere builds non-reasoning translation for a reason (Sun), Latent Space: DeepSeek v4.1-Flash, return of the whale (Sat), TC: Garry Tan wants US labs to distill too (Fri), TNS: Cohere's translation model, open but non-commercial (Fri), Interconnects: open models reading list (Fri).
Money
TC: Ellison cancels $7.5 billion Oracle stock sale (Sun), TC: the 9 buzziest startups from YC Demo Day (Sun), HN: Nike exits the S&P 100 after a $200B wipeout (Mon), TC: fusion startups find defense partners (Sun), TC: Lyft has entered the robotaxi chat (Sun), Slashdot: California gig drivers certify a union (Sun), TC: Altman says going public in 2026 would be ill-advised (Sat), Economist: Nvidia is the central bank of AI (Sat), TC: Mullenweg is back as CEO (Sat).
Systems, languages, tooling
Lobsters: Homebrew 7.0.0 (Sun), HN: Julia 1.13 highlights (Mon), LWN: kernel prepatch 7.3-rc3 (Sun), LWN: subscription price change coming (Sun), Lobsters: switching to GNU Guix, a beginner's perspective (Sun), Lobsters: writing a Guix service from scratch (Sun), Hahn: anecdotally, programmers dislike "reduce" (Sun), Lobsters: what if my git host were a static site generator? (Sun), Old New Thing: why is the x86 undefined instruction called ud2? (Sun), Lobsters: purely functional operating systems (1982) (Sun), Lobsters: Go developers should try Odin (Sun), Lobsters: Singeli, high-level interface for low-level programming (Sun), HN: EterDB, a Postgres fork for incident recovery (Mon), HN: durable execution without history replay (Sun), LWN: stabilizing Rust's never type (Sun), HN: pkgsrc is cool (Mon), HN: the case against JPEG XL (Mon), Hackaday: CircuitPython goes turbo with precompiled functions (Mon), Clojurists Together: annually-funded developers' update, July and August (Sun), Babashka: v1.13.221 (Mon), nREPL: v1.7.0-antora docs fix (Sun), OCaml.org: more tree-sitter, more neocaml, more elisp (Mon), Planet Clojure: a REPL you can fork (Sat), Planet Clojure: def is not a function (Sat).

Scan (15 min)

Tail

The benchmark is the story, not the score
Luu's post is about napkin math and winter tires, which is to say about asking what a number could possibly mean before arguing over it. The arXiv physics paper did the work and found the answer keys wrong. The LessWrong post found the models still gaming last year's tests. The New Stack found the customer still getting the wrong answer after everything passed. Four independent sources, one conclusion: the instruments are less trustworthy than the leaderboards built on them.
Tao's Sunday quote is the mathematicians' thesis in one line
Scarcity of deep theorems was doing double duty as a filter for deep thinkers, and the filter is gone. The Cyphral Distich result is the same point stated from the other side: a 370-year-old problem cleared by a model on a weekend. Neither post says what replaces the filter, which is the open question the guest posts keep circling.
The slowdown week ends in politics and portfolio theory
TechCrunch's Sunday explainer closes the Altman-Amodei arc for now. Obama's advice to Democrats and Insight Partners' decision to stop betting the farm on two labs are the practical downstream: the people who allocate votes and capital have started to hedge.

Feed silences (>72h since last item)

Sources that publish frequently but have gone quiet:

  • Neel Nanda (391 days) — last item 2025-08-19.
  • Aphyr/Jepsen (94 days) — last item 2026-06-12.
  • Eugene Yan (85 days) — last item 2026-06-21.
  • Lilian Weng (72 days) — last item 2026-07-04.
  • Andrej Bauer (65 days) — last item 2026-07-11.
  • Julia Evans (55 days) — last item 2026-07-21.
  • Stephen Wolfram (55 days) — last item 2026-07-21.
  • Marc Brooker (47 days) — last item 2026-07-29.
  • AI Snake Oil (40 days) — last item 2026-08-05.
  • Antithesis (27 days) — last item 2026-08-18.
  • TigerBeetle (25 days) — last item 2026-08-20.
  • FreeBSD Foundation (21 days) — last item 2026-08-24.
  • Steve Yegge (21 days) — last item 2026-08-24.
  • Netflix Tech Blog (17 days) — last item 2026-08-28.
  • Bunnie Studios (15 days) — last item 2026-08-30.
  • METR (14 days) — last item 2026-08-31.
  • Microsoft Research (14 days) — last item 2026-08-31.
  • Hillel Wayne (13 days) — last item 2026-09-01.
  • Vicki Boykis (13 days) — last item 2026-09-01.
  • deepmind-blog (13 days) — last item 2026-09-01.
  • DuckDB (12 days) — last item 2026-09-02.
  • GitHub Engineering (12 days) — last item 2026-09-02.
  • Kenneth Payne (12 days) — last item 2026-09-02.
  • Fly.io (11 days) — last item 2026-09-03.
  • All Things Distributed (6 days) — last item 2026-09-08.
  • Klara Systems (5 days) — last item 2026-09-09.
  • Martin Fowler (5 days) — last item 2026-09-09.
  • Pydantic (5 days) — last item 2026-09-09.
  • Supabase (5 days) — last item 2026-09-09.
  • Google Research (4 days) — last item 2026-09-10.
  • Hugging Face Blog (4 days) — last item 2026-09-10.
  • Nature Machine Intelligence (4 days) — last item 2026-09-10.
  • Neon (4 days) — last item 2026-09-10.
  • cursor-blog (4 days) — last item 2026-09-10.

Back this cycle: Charity Majors (67 days), OCaml.org, The Markup.

Build provenance

build: 2026-09-14 | crawler-sha: 34c428f (Walsh-Research/1.2, compliance v1.4) | feeds: 73 core | items-considered: 4784 (14d, incl. 2623 arxiv-cs-ai) | warehouse: 45581 items | published: PUBLISHED_COUNT