wal.sh
home > research > 2026-compaction-against-beads · Jul 26, 2026
context compactiontool useagent loopbeadsticket queueanthropic cookbookclojuretoken efficiencyclaude

Experience Report: Anthropic's Context-Compaction Cookbook Against a Beads Backend

Table of Contents

1. What this is

Anthropic ships a cookbook, automatic context compaction (platform.claude.com/cookbook), that keeps a long tool-use loop under the context window by summarizing and discarding old turns. Its demo drives a mock support-ticket queue. We already have a real one: beads, 303 tickets, 133 ready. This report ports the cookbook to a Clojure Messages-API loop, wires get_next_ticket to the real backlog, and runs the same baseline-vs-compaction experiment the cookbook describes.

The result is the cookbook's headline reproduced on real data — and one correction to how the feature is usually described.

2. The correction: compaction here is client-side, not a server beta

The Messages API does have a server-side compaction beta (compact-2026-01-12, context_management: {edits: [{type: "compact_20260112"}]}). That is not what this cookbook uses. Reading its "How Compaction Works Under the Hood" section, the tool_runner(compaction_control…)= helper does the work on the client:

When the tool_runner detects that token usage has exceeded the threshold, it […] injects a summary request as a user message […] Claude produces a summary […] the entire conversation history (including all tool results) is replaced with just the summary […] Claude continues with the compressed context.

So there is no beta header, no special endpoint — just orchestration around POST /v1/messages. That made the port small: a manual tool-use loop that, when the last request's input_tokens crosses a threshold, asks the model to emit a <summary> and then restarts the message list with only that summary. The whole runner is wal-sh.site.agent (~120 lines), calling the API through the JDK's java.net.http so it loads in both the CIDER REPL and babashka — the same portability fix the cost tool needed.

3. The backend: beads, dry-run

get_next_ticket and update_ticket_status are backed by wal-sh.site.beads. For the experiment they run through beads/queue-backend, an in-memory, dry-run backend: get_next_ticket pops from a snapshot of the ready queue and update_ticket_status just acks. Nothing touches issues.jsonl, so the loop processes the real backlog without mutating a single ticket. (The live backend, beads/handle-tool, does write — it is what a real grinder would use.)

The tickets the agent saw were genuine: "clean rebuild path for VPS", "use emacs –daemon for batch publish", and so on — the same P1s a human would pull.

4. The experiment

Same workload twice, 16 turns of Claude Sonnet 5, one ticket per turn: baseline with no compaction, then compaction at a deliberately low 3,000-token threshold (the window is ~200K; 3,000 just makes the mechanism fire in a short run). Run it yourself: 

gmake ticket-agent compare      # baseline vs compaction
gmake ticket-agent              # a single run with compaction

4.1. Baseline: context ramps monotonically

turn  1  ctx-in    769   turn  9  ctx-in  3743
turn  5  ctx-in   2262   turn 12  ctx-in  4619
turn  8  ctx-in   3462   turn 16  ctx-in  5785

Each ticket adds ~250–300 tokens (its title, description, the tool_use, the tool_result) and never leaves. Context climbs in a straight line to 5,785.

4.2. Compaction: a sawtooth

turn  7  ctx-in   3224
  ↻ compaction #1 (ctx 3224 > 3000) — history replaced by summary
turn  8  ctx-in   1352      <- reset
turn 14  ctx-in   3200
  ↻ compaction #2 (ctx 3200 > 3000) — history replaced by summary
turn 15  ctx-in   1764      <- reset
turn 16  ctx-in   2086

Twice, context crosses 3,000, the agent summarizes, and the next turn restarts at ~1,350–1,760 instead of ~3,200. The line becomes a sawtooth that never runs away.

4.3. The numbers

Metric Baseline Compaction Δ
Turns 16 16 =
Tool calls 31 31 =
Tickets processed 15 15 =
Compactions 0 2  
Peak context 5785 3224 −44%
Input tokens 55470 34061  
Output tokens 2459 2701 +10%
Total tokens 57929 36762 −37%

Identical work — 16 turns, 31 tool calls, 15 tickets — for 37% fewer tokens and a 44% lower peak context.

5. Reading the result

Two things the table makes concrete.

The win is the integral, and the ceiling. Peak context is what keeps you under the window at all; 44% headroom means the loop can run roughly twice as long before it would have overflowed. Total tokens is the area under the curve — a sawtooth has less area than a ramp, so you pay 37% less for the same work. On a longer run both gaps widen, because the baseline keeps climbing and the compaction run keeps resetting.

Compaction is not free — output went up 10%. The two summary-generation calls cost output tokens the baseline never spent. It nets out to a large win only because input dominates (here 20–23x output); in an output-heavy workload the summary tax would eat more of the gain. Compaction trades a few generation calls for a much smaller running context — worth it when context is the cost, which for long tool loops it is.

It is lossy by construction. After a compaction the agent holds only its own <summary> — the specific earlier tool_result bodies are gone. Ticket processing survives this cleanly because each ticket is independent: the summary ("processed 6, here are the ids, continue") is a sufficient statistic. A task that needs to reference a detail from turn 3 at turn 30 would not survive it. That is the whole reason this sits where it does in the context thread:

  • Compact (this note) — lossy, in-session, cheapest.
  • Fork (Time-Travel Chat) — lossless, in-session, pay in storage.
  • Remember (JITIR) — cross-session, pay in retrieval machinery.

6. Limitations and caveats

  • Threshold is a demo value. 3,000 tokens is ~1.5% of Sonnet 5's window. At a realistic threshold (say 150K) compaction fires rarely and this loop would never trigger it. The experiment shows the mechanism, not a production cadence.
  • Client-side ≠ the server beta. This port replicates the cookbook's tool_runner pattern. The server-side compact-2026-01-12 path (the API summarizes and hands back compaction blocks you replay) is a different, untested implementation with its own trade-offs.
  • One model, dry run. Sonnet 5 (chosen for cost); no real tickets were closed. Summary quality — and therefore how much the agent "forgets" — is model- and prompt-dependent and was not stress-tested.
  • Small n. One 16-turn run per arm. The numbers are a demonstration, not a benchmark; re-run gmake ticket-agent compare for your own.

7. Claims to test   refutation

Compaction cuts total tokens on this workload
an output-heavy workload where the summary-generation tax exceeds the input saved, netting more tokens with compaction on.
Peak context is bounded by the threshold
a run where a single turn's addition (a huge ticket description) overshoots the threshold by more than one turn's worth, so the sawtooth peak drifts up.
Ticket processing is lossless-enough under compaction
a dropped or double-processed ticket after a summary, i.e. the summary was not a sufficient statistic for "what's left."
The port matches the cookbook's behavior
a divergence between this client-side loop and tool_runner(compaction_control…)= on the same inputs.

8. Reproduce

Harness
src/wal_sh/site/agent.clj (the loop), src/wal_sh/site/beads.clj (queue-backend, tool-defs, handle-tool).
Run
gmake ticket-agent compare (needs ANTHROPIC_API_KEY; dry-run, no ticket mutation).
Cost
two 16-turn Sonnet 5 runs, ~95K total tokens, a few US cents.

9. Related