Mozilla.ai ran a tutorial titled Contextual Evaluation of LLM Guardrails Across Languages and Agentic Systems. Its claim, in the organizers' words:

evaluating guardrails is as important as evaluating the LLMs they protect, and context- and language-specific evaluation results should inform guardrails that move beyond static taxonomies of harm toward dynamic policies

The framing is that evaluation results are not a report card issued after the fact but an input to guardrail design, and that a guardrail built from a fixed list of harm categories cannot absorb what an evaluation finds. They place this alongside the EvalEval Coalition's work on evaluating evaluations themselves.

A community and language-informed evaluation of 120 refugee and asylum scenario pairs across English, Farsi, Arabic, Kurdish-Sorani and Pashto, scored by native-speaker evaluators from Respond Crisis Translation against six rights-based criteria. The results were published as open MHRE evaluation data on Mozilla Data Collective.

Two things about this design are worth separating from the result.

The scenarios are pairs, and the languages are matched. That makes the evaluation capable of catching a self-consistency failure — the same policy, two languages, and a guardrail that scores them differently is wrong on its own terms, with no need to adjudicate which score was correct. A benchmark that only asks "did the guardrail get this right" cannot detect that; one that asks "did it give the same answer to the same question" can.

The scorers are native speakers recruited through a translation organization that works with asylum seekers, not crowdworkers or a model. For a domain where the harm is defined by context — what is safe to tell someone about their asylum claim depends on jurisdiction, status, and what a wrong answer costs them — that is the expensive part of the method and the part hardest to reproduce.

The reported failure modes, verbatim: text-only guardrails

rubber-stamped responses they had no way to verify, hallucinated terms, and scored identical English and Farsi policies differently

Three different failures, and they are not equally interesting.

Hallucinated terms is the familiar one. Rubber-stamped responses they had no way to verify is the structural one: a guardrail asked whether a claim about asylum procedure is accurate, given only the text of the claim, has no mechanism for answering and approves by default. That is a gate whose green result carries no information, which is the same defect as a test that passes because it never ran its assertion.

Scored identical English and Farsi policies differently is the one that gives the other two their force, because it is the failure the paired design was built to catch. It is a demonstrated inconsistency rather than a disagreement about ground truth, and it is what makes the hypothesis — that guardrails need search, retrieval and fact-checking — an inference from evidence rather than an assertion about architecture.

Thirty-five participants exercised an agentic guardrail demo. The reported numbers:

Tools are reported to have moved verdicts in both directions, upgrading and downgrading.

This section is my reading, not the organizers'.

The two figures in tension are the 90 percent agreement and the twenty-fold spread in tool use. If giving a guardrail search and retrieval changes its verdict one time in ten, then nine times in ten the tools were either unnecessary or ignored, and the case for tool-enabled guardrails rests entirely on the tenth. Agreement rate is therefore the wrong summary statistic for the question being asked: a new gate that agrees with the old one 90 percent of the time has only been shown to differ. Whether it is an improvement depends on the direction of the 10 percent, and "bidirectional" is an honest way of saying that direction is not yet reported.

The model spread sharpens this. A model averaging 0.2 tool calls per run is running in something close to non-agentic mode most of the time, so part of the 90 percent agreement is not evidence that tools rarely matter — it is evidence that one of the two models rarely used them. Pooling the two into a single agreement figure mixes "the tools did not change the answer" with "the tools were not called". Those need separating before the headline number means anything, and the per-model tool-call averages are exactly the data that would separate them.

None of this is a criticism of the session, which was a tutorial with 35 participants rather than a study, and which the write-up does not oversell. It is a note about which follow-up would settle the question the work raises. The stated next step — testing whether tool access makes guardrails more reliable across humanitarian, financial and social-engineering use cases — is that follow-up, and the measure to want from it is not agreement with the baseline but accuracy on the cases where the two disagree.

• any-guardrail — a unified open-source interface for swapping guardrail implementations behind one API, with custom policies. The swap-in-place design is what makes a comparison like the one above cheap to run.
• Otari — an open-source LLM gateway for selecting and switching models, which is how the same guardrail policy gets exercised against Sonnet 4.6 and GPT-5 Nano without changing the harness.

Two threads already open in this corpus meet in it.

The first is calibration. The recurring rule here is that a gate never run against a known-bad input is uncalibrated, and the multilingual pair design is a clean instance of the same idea from the other direction: construct inputs whose correct handling is knowable a priori — two renderings of one policy must score alike — and the gate's failure becomes visible without an oracle. That is a cheaper form of calibration than labelling ground truth, and it transfers to any checker where an invariant is easier to state than a correct answer.

The second is LLM evaluation and the SmallCon 2024 guardrails sessions, which treated guardrails as a product feature to configure. The shift here is to treating the guardrail as a component under test with its own failure modes, which is a different activity and a harder one.

• Roya Pakzad and Daniel Nissani, From Evaluation to Guardrails: What We Brought to ACM FAccT 2026, Mozilla.ai blog, 22 July 2026.
• ACM FAccT 2026 conference site, for dates and venue.