Experience Report: Anthropic's Context-Compaction Cookbook Against a Beads Backend

Table of Contents

1. What this is

Anthropic ships a cookbook, automatic context compaction (platform.claude.com/cookbook), that keeps a long tool-use loop under the context window by summarizing and discarding old turns. Its demo drives a mock support-ticket queue. We already have a real one: beads, 303 tickets, 133 ready. This report ports the cookbook to a Clojure Messages-API loop, wires get_next_ticket to the real backlog, and runs the same baseline-vs-compaction experiment the cookbook describes.

2. The correction: compaction here is client-side, not a server beta

corrected · pilot-annotation-pass · 2026-10-08

The Messages API does have a server-side compaction beta (compact-2026-01-12, context_management: {edits: [{type: "compact_20260112"}]}). That is not what this cookbook uses. Reading its "How Compaction Works Under the Hood" section, the tool_runner(compaction_control…)= helper does the work on the client:

When the tool_runner detects that token usage has exceeded the threshold, it […] injects a summary request as a user message […] Claude produces a summary […] the entire conversation history (including all tool results) is replaced with just the summary […] Claude continues with the compressed context.

So there is no beta header, no special endpoint — just orchestration around POST /v1/messages. That made the port small: a manual tool-use loop that, when the last request's input_tokens crosses a threshold, asks the model to emit a <summary> and then restarts the message list with only that summary. The whole runner is wal-sh.site.agent (186 lines), calling the API through the JDK's java.net.http so it loads in both the CIDER REPL and babashka — the same portability fix the cost tool needed.

3. The backend: beads, dry-run

correct · pilot-annotation-pass · 2026-10-08

get_next_ticket and update_ticket_status are backed by wal-sh.site.beads. For the experiment they run through beads/queue-backend, an in-memory, dry-run backend: get_next_ticket pops from a snapshot of the ready queue and update_ticket_status just acks. Nothing touches issues.jsonl, so the loop processes the real backlog without mutating a single ticket. (The live backend, beads/handle-tool, does write — it is what a real grinder would use.)

The tickets the agent saw were genuine: "clean rebuild path for VPS", "use emacs –daemon for batch publish", and so on — the same P1s a human would pull.

4. The experiment

Same workload twice, 16 turns of Claude Sonnet 5, one ticket per turn: baseline with no compaction, then compaction at a deliberately low 3,000-token threshold (the window is ~200K; 3,000 just makes the mechanism fire in a short run). Run it yourself:

gmake ticket-agent compare      # baseline vs compaction
gmake ticket-agent              # a single run with compaction

4.1. Baseline: context ramps monotonically

turn  1  ctx-in    769   turn  9  ctx-in  3743
turn  5  ctx-in   2262   turn 12  ctx-in  4619
turn  8  ctx-in   3462   turn 16  ctx-in  5785

Each ticket adds ~250–300 tokens (its title, description, the tool_use, the tool_result) and never leaves. Context climbs in a straight line to 5,785.

4.2. Compaction: a sawtooth

turn  7  ctx-in   3224
  ↻ compaction #1 (ctx 3224 > 3000) — history replaced by summary
turn  8  ctx-in   1352      <- reset
turn 14  ctx-in   3200
  ↻ compaction #2 (ctx 3200 > 3000) — history replaced by summary
turn 15  ctx-in   1764      <- reset
turn 16  ctx-in   2086

Twice, context crosses 3,000, the agent summarizes, and the next turn restarts at ~1,350–1,760 instead of ~3,200. The line becomes a sawtooth that never runs away.

4.3. The numbers

correct · pilot-annotation-pass · 2026-10-08
Metric Baseline Compaction Δ
Turns 16 16 =
Tool calls 31 31 =
Tickets processed 15 15 =
Compactions 0 2  
Peak context 5785 3224 −44%
Input tokens 55470 34061  
Output tokens 2459 2701 +10%
Total tokens 57929 36762 −37%

Identical work — 16 turns, 31 tool calls, 15 tickets — for 37% fewer tokens and a 44% lower peak context.

5. Reading the result

The win is the integral, and the ceiling. Peak context is what keeps you under the window at all; 44% headroom means the loop can run roughly twice as long before it would have overflowed. Total tokens is the area under the curve — a sawtooth has less area than a ramp, so you pay 37% less for the same work. On a longer run both gaps widen, because the baseline keeps climbing and the compaction run keeps resetting.

Compaction is not free — output went up 10%. The two summary-generation calls cost output tokens the baseline never spent. It nets out to a large win only because input dominates (here 20–23x output); in an output-heavy workload the summary tax would eat more of the gain. Compaction trades a few generation calls for a much smaller running context — worth it when context is the cost, which for long tool loops it is.

It is lossy by construction. After a compaction the agent holds only its own <summary> — the specific earlier tool_result bodies are gone. Ticket processing survives this cleanly because each ticket is independent: the summary ("processed 6, here are the ids, continue") is a sufficient statistic. A task that needs to reference a detail from turn 3 at turn 30 would not survive it. That is the whole reason this sits where it does in the context thread:

  • Compact (this note) — lossy, in-session, cheapest.
  • Fork (Time-Travel Chat) — lossless, in-session, pay in storage.
  • Remember (JITIR) — cross-session, pay in retrieval machinery.

6. Limitations and caveats

  • Threshold is a demo value. 3,000 tokens is ~1.5% of Sonnet 5's window. At a realistic threshold (say 150K) compaction fires rarely and this loop would never trigger it. The experiment shows the mechanism, not a production cadence.
  • Client-side ≠ the server beta. This port replicates the cookbook's tool_runner pattern. The server-side compact-2026-01-12 path (the API summarizes and hands back compaction blocks you replay) is a different, untested implementation with its own trade-offs.
  • One model, dry run. Sonnet 5 (chosen for cost); no real tickets were closed. Summary quality — and therefore how much the agent "forgets" — is model- and prompt-dependent and was not stress-tested.
  • Small n. One 16-turn run per arm. The numbers are a demonstration, not a benchmark; re-run gmake ticket-agent compare for your own.

7. Claims to test   refutation

Compaction cuts total tokens on this workload
an output-heavy workload where the summary-generation tax exceeds the input saved, netting more tokens with compaction on.
Peak context is bounded by the threshold
a run where a single turn's addition (a huge ticket description) overshoots the threshold by more than one turn's worth, so the sawtooth peak drifts up.
Ticket processing is lossless-enough under compaction
a dropped or double-processed ticket after a summary, i.e. the summary was not a sufficient statistic for "what's left."
The port matches the cookbook's behavior
a divergence between this client-side loop and tool_runner(compaction_control…)= on the same inputs.

8. Reproduce

Harness
src/wal_sh/site/agent.clj (the loop), src/wal_sh/site/beads.clj (queue-backend, tool-defs, handle-tool).
Run
gmake ticket-agent compare (needs ANTHROPIC_API_KEY; dry-run, no ticket mutation).
Cost
two 16-turn Sonnet 5 runs, ~95K total tokens, a few US cents.

9. Related