Antithesis + AWS: Testing and Verification for Agentic Coding
Happy hour, SPIN Boston, 29 October 2026
Table of Contents
Event
| Field | Value |
|---|---|
| Event | Antithesis + AWS Happy Hour: Testing & Verification for Agentic Coding |
| Hosts | Antithesis, Amazon Web Services |
| Date | EDT (doors 16:30) |
| Venue | SPIN Boston, 30 Melcher St, Boston MA 02210 (Fort Point) |
| Format | In person; two talks, then food and drinks |
| Attending | registered |
| Time | Item |
|---|---|
| 16:30-17:00 | Arrival, check-in, food |
| 17:00-17:30 | AWS: the AWS AI platform, AI-DLC, and resilient architectures |
| 17:30-18:30 | Antithesis: agentic development, correctness, pre-production validation |
| 18:30-20:00 | Drinks and food |
The premise
The listing's claim is that agents have moved the bottleneck: code is cheap, and testing and verification are now the constraint. Its three symptoms are more and larger PRs eating review time, agent-written tests that cover the obvious path, and subtle bugs that survive both. The answer it promises is "a fundamentally different approach", which from Antithesis means deterministic simulation testing: run the whole system in a deterministic hypervisor, inject faults, explore schedules, and replay any failure exactly.
This site has argued the same premise from the other end. If an agent writes the code, the artifact that carries intent is the spec and the gates, not the code (The Oracle Is the Deliverable). A test an agent writes against its own implementation shares its blind spots; an oracle that has never been seen to fail is not yet an oracle (Four Controls for an Agent Skill). Deterministic simulation is one way to get an oracle the agent did not write: the properties are stated once, and the search for violations is done by the platform.
The AWS half is AI-DLC, AWS's name for an AI-driven development life cycle: AI plans and asks for clarification, humans validate in "mob" sessions, and work runs in short "bolts" across Inception, Construction and Operations. AWS pitched the same program at Agents of Impact in September.
Questions to bring
- Who writes the properties? If the agent writes both the code and the assertions Antithesis checks, the oracle has the same author as the bug. What does Antithesis recommend for keeping properties human-owned?
- A found bug arrives as a replayable trace. Can an agent consume that trace and propose the fix inside the same loop, and how is the fix then re-verified on the schedule that broke it?
- AI-DLC puts humans in "mob elaboration" at Inception. Is that where the properties get written, or are they left to Construction?
- For a browser front end, Bombadil is Antithesis's open property-based tester. How does it relate to the hypervisor product: same property language, or separate tools?
- Bombadil has no select-option action and records no network traffic; a spec has to add both itself, which the page-model run below did. Are either on the roadmap, and should one bad extractor value or action fail a step rather than end the run?
- Bombadil's
hash_currentis a SimHash of JS edges. Could a page model's coverage (controls, operations, transitions) guide exploration alongside it? - Would running Bombadil inside the deterministic hypervisor make timing findings, like states caught mid-navigation, replay exactly?
- Observability is listed as a theme. A simulated run has a perfect record of time; production does not. What carries over?
Prep: page models from Bombadil
The experiment behind the questions below runs OpenAPI codegen in reverse. One Bombadil exploration records every visible control per state, including the role and accessible name Playwright would locate it by, its form or repeated item, and the requests the page sent. A miner joins form submits to the OpenAPI request schema by value and generates Playwright page objects. Their field names and types come from the body key each field was seen to fill. A Playwright test written only against the generated classes passed. The same join also produced a drift report:
- fields the schema defines but the UI cannot set;
- request values the client hard-codes;
- six of seven selects with no accessible name, reachable only by position.
Bombadil 0.7.10 ended the run twice instead of failing the step:
- an extractor returned half a surrogate pair;
- a generated click had a negative coordinate.
The same model could carry several more tests. The aim is the same as codegen's: a UI or API change is a regenerate and a reviewed diff, not hand-fixed selectors.
| Test | Property | Matters most for |
|---|---|---|
| Model diff per change | regenerate the page model and review the diff (controls added or removed, locators changed, operations remapped) instead of screenshots | both; the cheap fast-tier artifact |
| Reload and deep-link equivalence | the model of a state reached by navigation equals the model after a reload of the same URL | SSR and hydration: a server render that disagrees with the client render |
| Route and DOM agree | after navigation, the URL and the page's h1 agree within n ms. The run caught pushState states still showing the previous page | SPA |
| CRUD laws from the operations | a create is eventually listed; a delete is eventually absent and a second delete is handled; an update is reflected. Generated per operation the model saw fire | both |
| UI and API validation agree | schema-driven values, including boundaries (quantity 0, an out-of-enum status): the form rejects them exactly when the API does | both |
| Error affordances under faults | with 500s or latency injected at the server, every failed write shows an error and no promise rejection goes unhandled | both; where Antithesis's fault injection fits |
| Model coverage as the budget | the share of controls clicked, operations fired and transitions taken, so the nightly tier spends its seed budget on what is uncovered | both |
| Locator stability | which locators survive a refactor or a locale change. Text-dependent role+name locators break under i18n; test ids and labels do not | both; also an accessibility metric |
| Per-route resources | DOM node, listener and heap growth per route, from Bombadil's CDP metrics | SPA long sessions |
| Variants | one model per feature flag or role, and the diff between them | both |
Server-rendered apps need one change to the recorder. A form that
submits by navigation never goes through fetch, so the spec would
record the FormData at the submit event instead. The form's
action and method give the endpoint without waiting for a request.
Notes
Notes pending.