Agent Telemetry Systems: Token Spend, Model Routing, and the One-Operator Dashboard

Token spend accrues across three distinguishable surfaces — the interactive main loop, fan-out subagents, and unattended workflow/cron runs — and honest accounting must keep them separate, not report a single conversation total.

Refutation: false if a single session total reconciles to within ~5% of the sum of independently-captured per-surface counts (i.e. subagent/workflow spend is already folded into main-loop telemetry).

Open questions:

• Does the harness emit subagent usage under the parent session id, or as an orphaned child?
• Are cron/workflow runs attributed to a service identity or the human owner?
• Are cache-read vs cache-write counted separately (caching dominates long literate-essay contexts)?

A useful ledger is append-only JSONL, each block carrying {session_id, parent_id, task/custom_id, surface, model, input_tok, output_tok, cache_read_tok, cache_write_tok, cost_usd, source} — extending .beads/interactions.jsonl rather than inventing a parallel store.

Refutation: false if interactions.jsonl cannot be extended without breaking beads' dolt-backed schema (cf. the dotted-name issue), forcing a separate .verify/spend.jsonl.

Open questions:

• Mirror the .verify/chain.jsonl hash-chain so spend is tamper-evident, or is append-only enough?
• Store cost_usd at write time, or compute from model id + a versioned price table?
• Does a parent_id chain roll subagent spend up to the originating skill/command?

Spend can be attributed to concrete site work by keying ledger blocks to the same :CUSTOM_ID: heading slugs and bead ids used for verification, so a note's cost is as queryable as its review history via a wal-sh.site.token-cost REPL ns.

Refutation: false if a single agent turn routinely spans multiple CUSTOM_IDs/beads with no clean boundary (e.g. one fan-out generates DEPLOYMENT and TELEMETRY sections together — exactly this workflow), making per-unit attribution arbitrary.

Open questions:

• What is the key when an agent edits files with no heading CUSTOM_ID (Makefile, project-config.el)?
• Can token-cost join interactions.jsonl against org.clj for cost-per-published-page?
• Attribute LiteLLM-routed Ollama calls at $0 or an imputed bare-metal rate?

Per-task and per-day budgets are best enforced at the LiteLLM proxy via virtual keys with spend limits, since the proxy is the single chokepoint that holds the API keys and sees every cloud call — keeping budget policy out of agent code.

Refutation: false if any surface reaches Anthropic/OpenAI directly (a workflow reading ANTHROPIC_API_KEY from env), in which case proxy-side caps are not authoritative.

Open questions:

• Does LiteLLM v1.83.4 support per-virtual-key daily budgets via the /spend endpoints in spec/rfcs/litellm-openapi.json?
• Failure mode at budget hit — hard refusal, downgrade, or fall through to Ollama?
• Give each fan-out subagent its own ephemeral virtual key for isolated caps?

Three independent counters observe the same spend — harness/=interactions.jsonl=, the LiteLLM spend log, and OTEL metrics at 192.168.86.100:4317 — and the ledger's correctness check is whether the three agree, treating disagreement as a telemetry bug rather than picking a favorite.

Refutation: false if the sources are not independent — e.g. OTEL spend metrics are emitted by LiteLLM, so agreement is tautological.

Open questions:

• What tolerance counts as agreement, given tokenizer drift and rounding?
• Which counter is authoritative for billing when they disagree?
• Can a Monitor/cron raise an alert when daily harness totals and LiteLLM /spend diverge?

An agent-run site has four signal classes worth instrumenting — deploy events (sha + build time), token spend (per-session in/out/cache cost), model-routing decisions (which model served which task), and validate-prod results (post-deploy pass/fail) — and everything else is noise for a one-operator site.

Refutation: false if a fifth class with comparable decision-value emerges (content-drift / search-index freshness, per-page traffic from the Apache log tools) checked as often, or if any of the four never changes an operator decision over a quarter.

Open questions:

• Is search-index freshness a distinct fifth signal or a sub-case of deploy events?
• Do the Apache log analyses (wal-sh.site.logs) belong here or stay a separate read-path tool?
• What primary key joins all four — git sha, session id, or wall-clock window?

A local append-only metrics.jsonl (mirroring .verify/chain.jsonl) is the right substrate for this site, and OpenTelemetry / a collector is over-engineered for a single-operator, batch-cadence, JVM-and-Emacs workflow.

Refutation: false if event volume or distributed correlation (Mac proxy, nexus, DreamHost, Cloudflare edge all emitting) exceeds a flat JSONL queried by the REPL, or if a native sink (Cloudflare Analytics Engine) already captures two of the four for free, making the JSONL pure duplication.

Open questions:

• Can LiteLLM's spend log at :4000 be tailed directly instead of re-emitting token rows?
• Does the DreamHost-vs-Cloudflare split force two sinks?
• Git-track metrics.jsonl (auditable) or gitignore it (churn/size)?

Token-spend and routing telemetry should be read from the LiteLLM proxy rather than instrumented inside each agent, because the proxy is the single chokepoint every call already passes through.

Refutation: false if a material fraction of calls bypass the proxy (direct SDK, Ollama-direct, or Claude Code's own billing path the proxy never sees) so proxy-side numbers undercount, or if LiteLLM cannot attribute a call to the issuing task.

Open questions:

• What fraction of traffic flows through :4000 vs Claude Code's native billing?
• Can a per-call tag (skill name, bead id) make spend attributable to a unit of work?
• Is Ollama (zero marginal) a $0 row or omitted — and does omission bias the breakeven analysis?

Stamping every telemetry event with the deployed git sha (the value version.json publishes and deploy-check verifies) joins deploy events, validate-prod runs, and the spend of the producing session into one per-deploy record without a separate trace-id system.

Refutation: false if work spanning multiple commits (or dirty trees) makes the sha non-unique per logical change — then a session id is the real key and the sha is a coarse bucket.

Open questions:

• How are dirty-tree / multi-commit sessions attributed?
• Does the .verify/chain.jsonl block hash give a stronger join key for validation events?
• Can validate-prod emit the sha it tested so pass/fail binds to a deploy?

The minimal dashboard is not a web UI but a gmake telemetry (or REPL fn) that prints, for the last N deploys: sha, deploy time, validate-prod verdict, and the token cost of the session that shipped it — four columns, terminal-rendered, like deploy-check and audit-trail already report.

Refutation: false if the operator needs time-series visualization a static table can't convey, or if a Cloudflare-hosted edge dashboard becomes zero-effort enough that any local renderer is wasted.