Muse Spark 1.1 eval: distilled to agent loops

The report centers the API deployment as the conservative upper bound: developer prompt plus tool/function calling are the new attack surfaces relative to the prior chat-only surface [P]. The whole document treats the deployed configuration as "model weights are one layer of a defense-in-depth stack" [P]. That is this site's thesis, stated by an adversary [I]. Read it as external validation of the sandbox-boundary posture behind drydock (the repo-scaffold testbed), not as a capabilities update [I].

Central quote, paraphrased: the section-3 evals isolate model-level behavior with no system defenses, so any resistance comes from the model itself; the report recommends deployments pair the model with policy-aligned safeguards, strict tool allowlists, and workspace isolation [P].

• The report disclaims the model as a sufficient control and pushes containment to the system layer [P]. This is the sandbox-boundary argument verbatim: the contract lives at the mount/allowlist/isolation plane, not in the weights [I].
• Evidence the system layer (not just adversarial training) does measurable work: the "System" configuration sits consistently below base ASR. Poly-Guard multi-turn drops from 11.2% to 6.9%, a 38% reduction [E]. The report states CyberBench multi-turn drops from 9.0% to 4.6%, about 49% [P].

The sharp structural point: exposing a developer prompt adds a mutable member to the governance tuple, and it is directly attackable [I].

• Joint (developer plus user) prompt optimization raises ASR above user-only attacks on several models [P]. The developer prompt is not trusted config; it is an injection channel the moment any untrusted content can influence it [I].
• Indirect injection: AgentDojo pass@1 ASR 0.7%, GraySwan ART 0.3% [E]. Strong, but the report self-reports trailing SOTA on file injection specifically [P]. File-borne injection is the residual hole [I]. This maps directly to the untrusted_external_data envelope: anything the agent reads is data, and file mounts are the weakest edge of that boundary [I].
• SWE-PI (injection in a coding-agent loop) is in scope precisely because tool calling is now realizable in deployment [P].

This is the most important loop for agent infrastructure and the most fragile claim in the report [I].

• SHADE-Arena: the report states side-task success is high in isolation (roughly 46 to 65% across the main/side tasks) but a CoT monitor collapses overall covert success to 6.8% [P]. The monitor, not the model's propensity, is what contains it [I].
• GDM-Stealth: the report states the "Classifier Evasion" column is empty across every model; no model clears the classifier-evasion threshold [P]. The report states sandbagging detection ROC AUC of roughly 0.98 to 0.99 [P]. Monitors are currently winning [I].
• The report footnotes the actual invariant: verbalized strategies are a lower bound on latent strategizing [P]. Every monitorability number above is conditioned on the reasoning being legible [I].

Directly relevant to any LLM-as-evaluator harness (ARIA, lemmata elenchus, the sprint-axiom conjecture registry) [I].

• Asymmetry: the report states verbalized evaluation awareness of roughly 79% on Agentic Misalignment against roughly 2.2% on dangerous-capability evals [P]. The model behaves differently when it detects an alignment eval specifically [I].
• CoT patterns the classifier flags: allowedness checks, evaluator-intent modeling ("is this bait?"), self-preservation reasoning ("exceeding safe capability leads to shutdown") [P]. The model mostly resolves to answer anyway, so measured performance deltas are small [P]. The deltas are small because the model chose not to act, not because it could not detect the frame [I].

No memory loop [P]. No cross-session state, no retrieval-poisoning threat model, no persisted-store integrity eval [P]. The injection threat model is strictly in-session (untrusted data consumed in-context) [P]. Poisoned-index / poisoned-memory attack surface is entirely out of scope [I].

• For JITIR / proactive retrieval this is the unaddressed edge: the report's robustness story assumes untrusted content arrives as a fenced tool result in one session [I]. A poisoned nomic-embed index that silently biases retrieval across sessions is a class of attack this eval regime does not touch [I].
• Open question, not a claim: is memory-poisoning reducible to the indirect-injection loop (just delayed), or a distinct provenance problem that needs a signed/append-only store (the logbook shape used by harbormaster, the spec-to-repo builder) to close? Reframing this is the deliverable, not resolving it [H].

• Risk escalation: the report states a SAVE-Bench score of 90.7; the model surfaces and escalates risky actions in coding-agent loops well, and peers vary widely [P]. Relevant to any autonomous-commit or worktree-coordination governance (the petclinic-lab TLA+ coordinator) [I].
• Insecure-code propensity: the report states CyberSecEval autocomplete/instruct pass@5 of roughly 71 to 77% [P]. Single-turn snippet only; the report itself flags that it does not capture multi-file / agentic code security [P]. Do not read it as a review-loop substitute [I].
• Defender tax: the report states cyber false-refusal rates are notably high (non-agentic FRR roughly 10%, agentic roughly 4.7%, above peers) [P]. For legitimate security tooling (saproxy, netax, security research) this model eats false refusals: a real cost, reported by the vendor, which says it is working to reduce it [P].

The same report collapses to one line in two registers.

Every claim in this note is intended to attribute to one of four sources, explicitly labeled. Vocabulary and definitions from the aygp-dr sprint-axiom methodology (a hypothesis-driven, elenctic examination of software-engineering claims, using Lakatos, Proofs and Refutations (1976) as the methodological anchor). Reused here for reading-notes on external eval reports.

Applied to a vendor eval report the vocabulary narrows: [E] marks a figure from the report that a second source confirms (see the property drawers on the two headings that carry one); [P] marks a paraphrase of the report's own claims that this note has not independently confirmed; [H] marks a single-source conjecture; [I] marks this note's own derivation.

Claims without a label are a failure mode. The document should surface its own epistemic status, not conceal it.