Does an AI agent's safety guard change with language?
A controlled 1,220-step benchmark found many small and purpose-built agent guards changed decisions when only a user's request was translated. The largest measured harms were concentrated and model-specific, and the 15 October paper remains an unreviewed workshop preprint.
The short version
3 key points
- 1.Researchers translated only the user instruction in 1,220 labelled agent steps while keeping the trajectory, proposed action and safety label unchanged.
- 2.Across 19 guards and five languages, many small and specialised guards moved beyond their own noise floor; frontier guards moved by roughly a point or less in the reported tests.
- 3.The benchmark is a company-authored, unreviewed preprint; low-resource translations lacked fluent human review and statistical intervals may be too narrow.
The answer: for many smaller guards, yes
Changing the language of a user's request can change whether an AI agent's safety guard allows exactly the same proposed tool action, according to a controlled preprint submitted on 15 October. The effect was largest in smaller open models and four purpose-built agent guards, while three frontier systems changed little in the authors' setup. The result identifies a deployment test that multilingual services can run now; it does not prove that every guard or language fails in the same way.
The benchmark begins with 1,220 labelled tool-invocation steps from 589 AgentDojo trajectories in banking, workplace messaging, travel and productivity environments. There are 868 safe and 352 unsafe steps drawn from 58 unique user instructions. The researchers translated only the instruction into German, French, Turkish, Swahili and Amharic. Tool calls, observations, reasoning histories, environment data and labels remained byte-identical, so a changed verdict can be attributed to the altered instruction condition rather than a different action.
Across 19 guard configurations, the authors collected 489,000 verdicts. They added six English rewrites and a whitespace placebo, and repeated managed guards three times to estimate nondeterministic noise. Among nine guards below 13 billion parameters, the average placebo-adjusted verdict-flip rate rose from 1.5 percentage points in French to 4.4 points in Amharic. Four specialised guards averaged a 6.3-point adjusted flip rate in Swahili and Amharic. Those are pooled shifts; the practical damage depended on which guard and error type changed.[1]
Over-blocking and missed attacks are different failures
The reported pattern split broadly by model scale. Several open guards in the 12–30 billion parameter range blocked more benign actions after translation, with false-positive shifts reaching five percentage points in the examples reported. Every statistically significant loss of attack recall appeared in a guard below ten billion parameters, reaching a 7.4-point decline for one model on Amharic. Purpose-built guards were not automatically robust: their verdicts moved the most as a class, though their adapted inputs make their absolute error rates hard to compare with step-level guards.
This distinction matters to real workers. Over-blocking can stop legitimate bookings, transfers, searches or internal tasks, especially for people who do not write in English. Under-blocking is quieter: an unsafe action that an English request would have stopped may pass in another language. A procurement claim such as 'the guard passed the English benchmark' therefore says little about equal service quality or attack resistance for multilingual users.
The damage was concentrated. One translated instruction moved a guard's blocked share from 12% to 92% across its related steps. A post-hoc analysis found excess flips were more common when the agent's reasoning restated the original English request, suggesting the guard may have reacted to inconsistency between the translated trusted request and an English trajectory. This is plausible diagnostic evidence, not a causal demonstration of the model's internal mechanism.[1]
How to test a deployed agent without accepting the headline on trust
Teams deploying tool-using agents should construct paired tests from their own workflows. Translate the user's task while freezing the account state, retrieved content, proposed tool call and ground-truth policy label. Measure both unsafe actions allowed and benign actions blocked, cluster results by underlying task, and repeat stochastic services. Long, enumerated requests and lower-resource languages deserve deliberate coverage because that is where this benchmark found concentrated failures.
A single aggregate accuracy score will hide the operational trade-off. Security staff need per-language recall for unsafe actions, false-positive rates for legitimate work and examples of high-impact flips. Product teams should also check whether the model silently translates, whether histories mix languages, and whether an English policy prompt conflicts with the user's language. Any translation-based mitigation needs its own evaluation: the authors did not test translate-then-guard or language-specific thresholds.
The finding is especially relevant for public services, banks and employers, where a false block may delay a necessary task and a missed block may expose personal data or money. Human fallback must be reachable in the affected language. Logging should preserve the guard version, request language, decision and appeal outcome without collecting more personal content than necessary. Those records can show whether benchmark differences become real service disparities.[1]
Why the evidence remains provisional
The two authors are affiliated with Kontext, a security company, and the paper is an unreviewed preprint accepted to a NeurIPS workshop. The translated conditions were generated by Gemini 3.8 Flash, which was also evaluated as a guard. Although the team used two model judges, back-translation and mechanical checks, neither author reads Swahili or Amharic. Four low-resource-language rows required repair, and some model-language combinations were not run.
The benchmark itself has structural caveats. Nearly 60% of steps contained substantial restatement of the request in the agent's history, allowing a guard to rely on the trajectory instead of the translated field. The original operating points for some open models already had high false-positive rates. Three specialised guards consumed adapted trajectory formats rather than native step-level inputs. Most importantly, bootstrap intervals clustered 589 trajectories even though translation was applied at only 58 unique instructions; the authors say this probably makes intervals too narrow.
Independent replication should use fluent reviewers for every language, more task families and languages, and guards configured exactly as production systems use them. Prospective red-team exercises should test whether attackers can intentionally exploit language switching, while service evaluations should measure unequal friction for legitimate users. Until then, the responsible conclusion is narrow: language belongs in agent-security testing, and English-only results should not be assumed to transfer.[1]
Explore the full source and evidence record
What this means for people
- Non-English users may encounter either more blocked legitimate actions or weaker protection, depending on the guard.
- Security teams need per-language false-positive and attack-recall measures rather than one English accuracy score.
- Public-service and workplace deployments need a language-appropriate human appeal path when automation blocks a task.
Global context
The benchmark spans five languages from high- to low-resource categories, but it is not representative of the world's languages or multilingual code-switching. Local services should test the languages and scripts their users actually employ.
What the evidence does not yet show
- This is an unreviewed company-authored preprint based on benchmark trajectories, not observed production incidents.
- Swahili and Amharic translations were not reviewed by fluent team members, and coverage differed across guards.
- The intervention occurred at 58 unique instructions, so intervals clustered at the 589-trajectory level may be too narrow.
- The study evaluated verdict changes and did not test a mitigation or execute tool calls.
What to watch next
- Independent multilingual replications with fluent reviewers and production guard configurations
- Paired tests of translate-then-guard and language-specific calibration
- Evidence from mixed-language histories, scripts and real user appeals
- Whether vendors publish per-language false-positive and attack-recall results
Living evidence record
Impact record IAI-0VSREAR
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
16 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Demirkiran and Ernstberger (arXiv) published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 16 October 2026
Research question, topics and regions
Language robustness of step-level AI agent guards
Topics: AI agents · Security controls · Multilingual AI · Prompt injection · Tool use
Regions: Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Security & Defence
Can a malicious instruction travel from one AI agent to another?
An unreviewed preprint explores prompt injection in single-agent and multi-agent systems. Its controlled tests raise questions about trusting another agent's output; they do not demonstrate a widespread real-world outbreak.
5 min · 1 source
Security & Defence
How often did prompt injection divert coding agents?
In a new sandbox benchmark, attack success averaged 19.6% by an AI judge and 15.4% by artifact checks across 26,220 undefended trajectories. Rates varied sharply by model and harness, and the strongest tested prompt defence traded security against task utility.
7 min · 1 source
Security & Defence
Can encoded instructions turn AI-agent skills into security exploits?
Yes, in a controlled benchmark: some models decoded and followed malicious payloads hidden in third-party skill files. Attack rates fell sharply with the original longer files, and the work does not demonstrate a live breach.
7 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.