• Who: Nate Aune · Jazkarta (CEO)
• One-line: Open, evidence-based sandbox for evaluating coding agents.
• Stack: Bun · TypeScript · React · Anthropic API · Docker
• Links: github · x · linkedin
• Adjacency: elenctic-spec, directive-proof-harness. Refutation surface: does the sandbox control for harness vs. model contribution, or conflate them? Resolved: it controls — fixes harness + model, varies only the framework (phase 2 swaps both). Repo: natea/harness-eval.
• Notes:
  • [X] Pulled the repo — natea/harness-eval (Bun/TS). Methodology: same spec + same harness + same model (Claude Code + Opus 4.6) in isolated sandboxes, so the framework is the only variable (Superpowers / Compound Engineering / Agent Skills / GSD). Two-instrument grading — evidence-based evaluator + blind code-quality judge → results.json → scorecard + dashboard.
  • AI credit + availability for other sites.
• Harness-engineering framing (the literature behind "measure the harness") — both define harness = everything except the model, which is exactly why Aune's eval holds the model fixed and varies the framework:
  • Osmani, Agent Harness Engineering — "a decent model with a great harness beats a great model with a bad harness." Harness = prompts, tools, context policies, hooks, sandboxes, feedback loops. Failures are "skill issues" (config), not model limits; AGENTS.md as a ratchet (every line earned by a past failure); planner/generator/evaluator split to avoid optimistic self-grading.
  • Böckeler (Thoughtworks), Harness Engineering for Coding Agents — harness as a cybernetic governor: guides (feedforward) + sensors (feedback); computational vs inferential checks; three dimensions (maintainability / architecture-fitness / behaviour); "keep quality left"; Ashby's law of requisite variety.
• Two axes of "measure it" — orthogonal benchmarks bracketing agent = model + harness:
  • Model axis — Artificial Analysis (independent): ranks models + providers. Metrics: Intelligence Index (composite of ~9 evals — coding, reasoning, real-world work), output speed (tokens/s), cost (per task / total), latency. Specialized indexes: coding agents, agentic knowledge work (AA-Briefcase), knowledge reliability (AA-Omniscience). Benchmark types: LLMs · coding agents · image/video · speech · API providers.
  • Harness axis — Aune / harness-eval: holds the model fixed, grades the framework on a weighted rubric (PRD adherence 40% · code quality 25% · speed 17.5% · token spend 17.5%).
  • Together: Artificial Analysis measures the model term, harness-eval the harness term — same equation, opposite variable held fixed.
• Wrap: the eval wraps the PRD — it grinds the spec for adherence (the 40% weight). But "how does this actually build?" isn't answered by PRD-adherence alone; it's the other axes — code quality (25%), speed (17.5%), and token efficiency (17.5%) — that say whether the build is any good.
• Eval of the evals (checked the repo):
  • Trials: default trialsPerCandidate: 3, but published results are n=1 (smoke); the full 3-trial matrix is operator-gated (cloud concurrency limited).
  • Distribution: the scorecard is built to report variance stats + inconclusive- ordering flags, but degenerate at n=1 ("speed/spend normalize degenerately"); real distribution needs the matrix.
  • Worker vs grader split: judgeModel: claude-sonnet-4-6, pinned independent of the worker (Opus 4.6) — "judge independence". Two instruments: evidence-based evaluator (frozen test plan) + blind code-quality judge.
  • Agent variance excluded: grading runs against a protocol-speaking stub, so worker/agent nondeterminism is kept out of the measurement.
  • Open meta-question ("same outcome → same eval result?"): grader self-consistency (re-grade the same artifact → same score) is not explicitly tested — variance is absorbed by N-trials + an independent judge, not by re-running the grader on a fixed artifact. The unmeasured axis.
  • Lineage: ViBench methodology (Replit / Georgian / CMU, CAIS 2026) — Pass@1 / Graded Score / Complete Failure Rate, one decoupled evaluator.
• Attribution — what is the score actually a result of? The design intends it to attribute to the skills/framework (the only free variable); worker, grader, and spec are held fixed. Term by term:
  • Skills (framework) — the intended signal: between-candidate variance. Clean only if the fixed terms are neutral + low-variance.
  • Worker (model) — fixed (Opus 4.6), but run-to-run model nondeterminism is real and only separable at N>1. At published n=1 the framework effect is conflated with a single worker roll.
  • Grader — fixed + independent (Sonnet 4.6), but self-consistency is untested, so grader noise (esp. the 25% blind-judge / inferential component) can masquerade as a framework difference.
  • Spec version — fixed, but not neutral: PRD adherence is 40% of the score, so the result is conditional on this spec (Symphony). A different / more ambiguous PRD could reorder the leaderboard — external validity, not estimated (though BRING-YOUR-OWN-PRD makes it testable).
  • Bottom line: intended f(skills); at n=1 actually f(skills, one worker roll, one grader pass, one spec). Clean attribution needs N trials (worker variance), repeated grading of fixed artifacts (grader variance), and multiple PRDs (spec).
• Clean-attribution design (the ablation ladder — each step a distribution, not a point):
  1. Baseline, no add-ons: fix model + harness + spec, run N iterations bare — does the spec even produce consistent results with no framework? This is the control arm (currently missing) and the floor everything else is measured against.
  2. + add-ons (frameworks): same fixed model + harness + spec, N iterations per framework — the framework's effect is the lift over baseline, not the raw score.
  3. × grader repeated: re-run judge / evaluator multiple times per artifact → a distribution of grades (grader self-consistency), so grader noise is subtracted, not assumed away.
  4. Only with all three layers is the outcome attributable: framework signal = (with-add-ons − baseline), net of worker variance (N) and grader variance (repeat), on a fixed spec — then repeat across specs for external validity.

Architecture (natea/harness-eval):

digraph harness_eval {
  rankdir=TB; bgcolor="transparent"; pad=0.2;
  node [shape=box style="rounded,filled" fontname="Helvetica" fontsize=11];
  edge [fontname="Helvetica" fontsize=9 color="#334155" fontcolor="#334155"];

  registry [label="registry\n(pinned frameworks)"  fillcolor="#f1f5f9" color="#334155" fontcolor="#334155"];
  target   [label="eval target\n(PRD + test plan)"  fillcolor="#f1f5f9" color="#334155" fontcolor="#334155"];
  config   [label="run config\n(budgets, weights)"  fillcolor="#f1f5f9" color="#334155" fontcolor="#334155"];
  { rank=same; registry; target; config; }

  orch    [label="orchestrator"                       fillcolor="#ede9fe" color="#6d28d9" fontcolor="#6d28d9"];
  sandbox [label="per-trial sandbox"                  fillcolor="#dbeafe" color="#1d4ed8" fontcolor="#1d4ed8"];
  build   [label="headless build\nsession(s)"         fillcolor="#dbeafe" color="#1d4ed8" fontcolor="#1d4ed8"];
  prov    [label="provenance + telemetry"             fillcolor="#fef3c7" color="#b45309" fontcolor="#b45309"];
  archive [label="archived workspace\n+ transcripts"  fillcolor="#f1f5f9" color="#334155" fontcolor="#334155"];
  grading [label="grading\nevaluator (test plan, evidence)\n+ blind judge (code quality)" fillcolor="#dcfce7" color="#15803d" fontcolor="#15803d"];
  results [label="results.json"                       fillcolor="#dbeafe" color="#1d4ed8" fontcolor="#1d4ed8"];
  out     [label="scorecard.md\n+ web dashboard"      fillcolor="#dcfce7" color="#15803d" fontcolor="#15803d"];

  registry -> orch;
  target   -> orch;
  config   -> orch;

  orch -> sandbox -> build;
  orch -> prov;
  build -> archive;

  prov    -> grading;
  archive -> grading;
  grading -> results -> out;
}