From Evaluation to Guardrails: What We Brought to ACM FAccT

From Evaluation to Guardrails: What We Brought to ACM FAccT 2026

What happened

The ACM Conference on Fairness, Accountability, and Transparency took place in June in Montreal.
A tutorial titled 'Contextual Evaluation of LLM Guardrails Across Languages and Agentic Systems' was presented at ACM FAccT 2026.
An evaluation was conducted across 120 refugee and asylum-focused scenario pairs.
The evaluation covered five languages: English, Farsi, Arabic, Kurdish-Sorani, and Pashto.
The MHRE evaluation data was published as open data on Mozilla Data Collective.
Native-speaker evaluators from Respond Crisis Translation scored the evaluation scenarios.
Thirty-five participants ran the demo during the hands-on session.
Ninety percent of verdicts agreed across agentic and non-agentic judge modes.
Claude Sonnet 4.6 used web search with 4.1 tool calls per run on average.
GPT-5 Nano made 0.2 tool calls per run on average.
Mozilla AI released any-guardrail as an open-source tool providing a unified interface for choosing and swapping guardrails.
Otari, Mozilla's new open-source LLM gateway, allows seamless switching of LLMs behind judges.
Future testing is planned for humanitarian, financial, and social-engineering use cases in English-Farsi and English-Spanish language pairs.

An evaluation was conducted across 120 refugee and asylum-focused scenario pairs. (120)
Thirty-five participants ran the demo during the hands-on session. (35)
Ninety percent of verdicts agreed across agentic and non-agentic judge modes. (90)
Claude Sonnet 4.6 used web search with 4.1 tool calls per run on average. (4.1)
GPT-5 Nano made 0.2 tool calls per run on average. (0.2)