Experience Report: Anthropic's Context-Compaction Cookbook Against a Beads Backend
Table of Contents
1. What this is
Anthropic ships a cookbook, automatic context compaction
(platform.claude.com/cookbook), that keeps a long tool-use loop under the
context window by summarizing and discarding old turns. Its demo drives a
mock support-ticket queue. We already have a real one: beads, 303 tickets,
133 ready. This report ports the cookbook to a Clojure Messages-API loop, wires
get_next_ticket to the real backlog, and runs the same baseline-vs-compaction
experiment the cookbook describes.
2. The correction: compaction here is client-side, not a server beta
Fetched the live cookbook page (platform.claude.com/cookbook/tool-use-automatic-context-compaction) directly. The quoted "How Compaction Works Under the Hood" text and the client-side-vs-server-beta distinction both check out verbatim — the cookbook's own intro recommends server-side compaction for newer models and frames itself as covering "SDK-based compaction" for older or cheaper-summarizer cases, matching this section's claim exactly. The =compact-2026-01-12= beta header and =compact_20260112= edit type are also correct per the current API reference. One number was wrong: "the whole runner is wal-sh.site.agent (~120 lines)" — =git show 8cf7af00:src/wal_sh/site/agent.clj= (the commit that introduced both the runner and this note; the file has never been touched since) is 186 lines, not ~120. Corrected in the body.
The Messages API does have a server-side compaction beta (compact-2026-01-12,
context_management: {edits: [{type: "compact_20260112"}]}). That is not what
this cookbook uses. Reading its "How Compaction Works Under the Hood" section,
the tool_runner(compaction_control…)= helper does the work on the client:
When the
tool_runnerdetects that token usage has exceeded the threshold, it […] injects a summary request as a user message […] Claude produces a summary […] the entire conversation history (including all tool results) is replaced with just the summary […] Claude continues with the compressed context.
So there is no beta header, no special endpoint — just orchestration around
POST /v1/messages. That made the port small: a manual tool-use loop that,
when the last request's input_tokens crosses a threshold, asks the model to
emit a <summary> and then restarts the message list with only that summary.
The whole runner is wal-sh.site.agent (186 lines), calling the API through
the JDK's java.net.http so it loads in both the CIDER REPL and babashka —
the same portability fix the cost tool needed.
3. The backend: beads, dry-run
Cross-checked the ticket counts against the repo's own history rather than the current backlog (which has since moved to 306 records). =git show 8cf7af00:.beads/issues.jsonl= — the exact commit that introduced this note and the agent — has 303 records (147 open, 155 closed, 1 in_progress). "133 ready" matches that same commit's message ("the real 133-ready backlog"), an independent, contemporaneous primary source.
get_next_ticket and update_ticket_status are backed by wal-sh.site.beads.
For the experiment they run through beads/queue-backend, an in-memory,
dry-run backend: get_next_ticket pops from a snapshot of the ready queue and
update_ticket_status just acks. Nothing touches issues.jsonl, so the loop
processes the real backlog without mutating a single ticket. (The live backend,
beads/handle-tool, does write — it is what a real grinder would use.)
The tickets the agent saw were genuine: "clean rebuild path for VPS", "use emacs –daemon for batch publish", and so on — the same P1s a human would pull.
4. The experiment
Same workload twice, 16 turns of Claude Sonnet 5, one ticket per turn: baseline with no compaction, then compaction at a deliberately low 3,000-token threshold (the window is ~200K; 3,000 just makes the mechanism fire in a short run). Run it yourself:
gmake ticket-agent compare # baseline vs compaction
gmake ticket-agent # a single run with compaction
4.1. Baseline: context ramps monotonically
turn 1 ctx-in 769 turn 9 ctx-in 3743 turn 5 ctx-in 2262 turn 12 ctx-in 4619 turn 8 ctx-in 3462 turn 16 ctx-in 5785
Each ticket adds ~250–300 tokens (its title, description, the tool_use, the
tool_result) and never leaves. Context climbs in a straight line to 5,785.
4.2. Compaction: a sawtooth
turn 7 ctx-in 3224 ↻ compaction #1 (ctx 3224 > 3000) — history replaced by summary turn 8 ctx-in 1352 <- reset turn 14 ctx-in 3200 ↻ compaction #2 (ctx 3200 > 3000) — history replaced by summary turn 15 ctx-in 1764 <- reset turn 16 ctx-in 2086
Twice, context crosses 3,000, the agent summarizes, and the next turn restarts at ~1,350–1,760 instead of ~3,200. The line becomes a sawtooth that never runs away.
4.3. The numbers
Did not re-run =gmake ticket-agent compare= (real API spend for a result already on record). Instead cross-checked this table against the commit message for 8cf7af00, written at the time, independently of this file's prose: "37% fewer tokens (57,929 -> 36,762) and 44% lower peak context (5,785 -> 3,224)" — an exact match to every cell below, confirming the numbers were not altered after the fact.
| Metric | Baseline | Compaction | Δ |
|---|---|---|---|
| Turns | 16 | 16 | = |
| Tool calls | 31 | 31 | = |
| Tickets processed | 15 | 15 | = |
| Compactions | 0 | 2 | |
| Peak context | 5785 | 3224 | −44% |
| Input tokens | 55470 | 34061 | |
| Output tokens | 2459 | 2701 | +10% |
| Total tokens | 57929 | 36762 | −37% |
Identical work — 16 turns, 31 tool calls, 15 tickets — for 37% fewer tokens and a 44% lower peak context.
5. Reading the result
The win is the integral, and the ceiling. Peak context is what keeps you under the window at all; 44% headroom means the loop can run roughly twice as long before it would have overflowed. Total tokens is the area under the curve — a sawtooth has less area than a ramp, so you pay 37% less for the same work. On a longer run both gaps widen, because the baseline keeps climbing and the compaction run keeps resetting.
Compaction is not free — output went up 10%. The two summary-generation calls cost output tokens the baseline never spent. It nets out to a large win only because input dominates (here 20–23x output); in an output-heavy workload the summary tax would eat more of the gain. Compaction trades a few generation calls for a much smaller running context — worth it when context is the cost, which for long tool loops it is.
It is lossy by construction. After a compaction the agent holds only its own
<summary> — the specific earlier tool_result bodies are gone. Ticket
processing survives this cleanly because each ticket is independent: the summary
("processed 6, here are the ids, continue") is a sufficient statistic. A task
that needs to reference a detail from turn 3 at turn 30 would not survive it.
That is the whole reason this sits where it does in the context thread:
- Compact (this note) — lossy, in-session, cheapest.
- Fork (Time-Travel Chat) — lossless, in-session, pay in storage.
- Remember (JITIR) — cross-session, pay in retrieval machinery.
6. Limitations and caveats
- Threshold is a demo value. 3,000 tokens is ~1.5% of Sonnet 5's window. At a realistic threshold (say 150K) compaction fires rarely and this loop would never trigger it. The experiment shows the mechanism, not a production cadence.
- Client-side ≠ the server beta. This port replicates the cookbook's
tool_runnerpattern. The server-sidecompact-2026-01-12path (the API summarizes and hands back compaction blocks you replay) is a different, untested implementation with its own trade-offs. - One model, dry run. Sonnet 5 (chosen for cost); no real tickets were closed. Summary quality — and therefore how much the agent "forgets" — is model- and prompt-dependent and was not stress-tested.
- Small n. One 16-turn run per arm. The numbers are a demonstration, not a
benchmark; re-run
gmake ticket-agent comparefor your own.
7. Claims to test  refutation
- Compaction cuts total tokens on this workload
- an output-heavy workload where the summary-generation tax exceeds the input saved, netting more tokens with compaction on.
- Peak context is bounded by the threshold
- a run where a single turn's addition (a huge ticket description) overshoots the threshold by more than one turn's worth, so the sawtooth peak drifts up.
- Ticket processing is lossless-enough under compaction
- a dropped or double-processed ticket after a summary, i.e. the summary was not a sufficient statistic for "what's left."
- The port matches the cookbook's behavior
- a divergence between this
client-side loop and
tool_runner(compaction_control…)= on the same inputs.
8. Reproduce
- Harness
src/wal_sh/site/agent.clj(the loop),src/wal_sh/site/beads.clj(queue-backend,tool-defs,handle-tool).- Run
gmake ticket-agent compare(needsANTHROPIC_API_KEY; dry-run, no ticket mutation).- Cost
- two 16-turn Sonnet 5 runs, ~95K total tokens, a few US cents.
9. Related
- The Agent Context Thread — this is its "compact" corner.
- Time-Travel Chat — the lossless-fork alternative to summarize-and-discard.
- Agent Memory Architectures (JITIR) — the cross-session axis.
- Claude Code Features: 2026 Q2 — where compaction shows up in the harness.