The Harness for AI Agents — Boston Gen AI Meetup
Table of Contents
1. Logistics
- Event
- Boston Generative AI Meetup — The Harness for AI Agents
- When
- Thursday, 2026-07-23, evening
- Where
- Microsoft NERD Center, 1 Memorial Drive, Cambridge, MA
- Host
- Judah (Super Organizer)
- Hosts/partners
- Microsoft, Squark AI
- Admission
- $22 (catering + expenses)
- Link
- Event page
2. Thesis
The harness, not the model. Infrastructure that turns agent demos into systems with a contract: orchestration, memory/tools/MCP, governance, observability, human-in-the-loop, prototype→production.
3. Related notes crosslink
Direct predecessor — same thesis, four weeks earlier:
- AI Tinkerers Boston — GTM Agentic AI Launch (2026-06-29) — six talks that all
landed on "harness, not model": durable execution (Poliakov), secret-brokering
(Kumar), deterministic code + structured memory (Zhu), evidence-based harness
eval (Aune). Carries the harness-engineering literature (Osmani;
Böckeler/Fowler) and a full attribution teardown of
harness-eval.
Research to align against:
- terminal-bench — harness/agent benchmarking.
- agent identity attestation — governance / trust boundary.
- LangGraph checkpoints — checkpointing, resume, HITL interrupts.
- code-graph MCP survey — MCP tool surface.
- ObservabilityCon 2026 · AI Engineer World's Fair 2026 — observability and agent-infrastructure events.
4. Panel people
6. Notes by topic
6.1. Orchestration & multi-agent
See also: AI Tinkerers — Poliakov (durable execution over Postgres), Zhu (queue
between tool-call and memory; worker claims pending → active).
6.2. Memory, tools, MCP integration
See also: code-graph MCP survey; AI Tinkerers — Zhu (markdown memory → RDBMS).
6.3. Governance, security, compliance
See also: agent identity attestation; AI Tinkerers — Kumar (broker fronts the credential; approval is time-boxed).
6.4. Evaluation, monitoring, observability
See also: terminal-bench; ObservabilityCon 2026; AI Tinkerers — Aune (harness-eval: fix the model, vary the framework).
6.5. Human-in-the-loop
See also: LangGraph checkpoints; AI Tinkerers — Kumar (time-boxed approval window).
6.6. Prototype → production
See also: AI Tinkerers — Zhu ("what broke and how we fixed it").
6.7. Reliability / trust boundary
See also: AI Tinkerers overview — the throughline is "what is the contract around the model, and what is the blast radius when it's wrong."
7. Claims to test refutation
- "A decent model with a great harness beats a great model with a bad harness" (Osmani) :: a controlled run where swapping the model (harness fixed) moves the outcome more than swapping the harness (model fixed).
- a failure that reproduces identically across two materially different harnesses.
- an agent that retains or reuses a credential outside its approval window.
- a resumed run that re-invokes steps already recorded as complete.
8. Open questions / Q&A
- What is the control arm? If a harness claims a lift, lift over what baseline — bare model, no add-ons, run N times?
- Is the evaluator self-consistent — re-grade the same artifact twice, same score?
- Is the result spec-invariant, or conditional on the one PRD/task chosen?
- Where is the trust boundary — does the agent ever hold the credential, or only make a brokered, time-boxed request?
- On outage: do you resume (checkpointed) or restart — and do you re-pay for the LLM calls already made?
- What in the harness is deterministic code vs left to the model, and who decided?
9. Contacts to follow up network
| Name | Org | Contact | Thread |
|---|---|---|---|
10. Actions
[ ]
11. Scratch