Web agents worked far worse in Amharic and Urdu—what did the benchmark reveal?
Eight models attempted 3,000 localized task instances across six simulated sites. English-only evaluation overstated reliability, especially for open-weight agents and pages with pop-ups.
The short version
- What happened
- LoWAgent contains 600 unique tasks localized into English, Bengali, Hindi, Urdu and Amharic across six simulated websites, creating 3,000 task instances.
- Who it affects
- A person using an agent in Amharic, Urdu, Bengali or Hindi may receive a less reliable service even when the website offers an equivalent localized task.
- What remains unproven
- This unreviewed benchmark uses constructed sites, AI-assisted translation, eight models and binary task success; it does not measure satisfaction, accessibility or harm in live services.
Equivalent tasks produced unequal results across languages
A new benchmark suggests that testing web agents only in English can substantially overstate how well they will serve people using other languages. LoWAgent uses six simulated sites—shopping, a forum, used cars, rail tickets, government services and airline booking—with the same code and underlying state in English, Bengali, Hindi, Urdu and Amharic. Its 600 unique tasks become 3,000 localized instances, including 750 visually grounded cases.[1]
Who it affects
A person using an agent in Amharic, Urdu, Bengali or Hindi may receive a less reliable service even when the website offers an equivalent localized task.[1]
Keep in mind
This unreviewed benchmark uses constructed sites, AI-assisted translation, eight models and binary task success; it does not measure satisfaction, accessibility or harm in live services.
Source analysis · 1 linked source. Original source date: 19 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Equivalent tasks produced unequal results across languages
A new benchmark suggests that testing web agents only in English can substantially overstate how well they will serve people using other languages. LoWAgent uses six simulated sites—shopping, a forum, used cars, rail tickets, government services and airline booking—with the same code and underlying state in English, Bengali, Hindi, Urdu and Amharic. Its 600 unique tasks become 3,000 localized instances, including 750 visually grounded cases.
Eight models attempted the tasks. Gemini 3.1 Flash-Lite had the smallest reported gap, losing 1.5 to 7.3 percentage points outside English. GPT-5.6 Luna lost 16.6 points in Amharic, while Llama-3.1-8B and GPT-OSS-20B lost 34.5 and 33.3 points respectively. Visual tasks were harder: the best model reached about 60% in English versus 86.4% on text-only English tasks.[1]
Common pop-ups exposed another reliability gap
The team added a cookie notice and an advertising pop-up that agents had to dismiss. Gemini's success fell from around 80% across languages to below 20%; several open models lost more than half their baseline performance. Post-training two open models on 450 tasks improved held-out success by roughly five points overall, with larger gains in the four lower-resource languages.
For people using agents to book travel, find public services or shop, the result means language and interface conditions should be tested together before deployment. Native speakers reviewed the Fable-assisted localization and every task was manually executed, but the websites were artificial and did not include the full diversity of real layouts, dialects, assistive technologies or network conditions. Independent live-site evaluation and user studies would change the assessment.[1]
The distinction that matters
What changed while the task stayed the same
The benchmark held site logic and ground truth constant while varying language and interface obstacles.
Language
Same task localized five ways
Success gaps become measurable
Interface
Cookie and advert interruptions appear
Agents must recover before acting
Training
Small multilingual subset is used
Held-out performance improves modestly
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Evidence check
This unreviewed benchmark uses constructed sites, AI-assisted translation, eight models and binary task success; it does not measure satisfaction, accessibility or harm in live services.
Reporting basis: Source analysis. Original source date: 19 October 2026.
Continue the story
Read next
Continue exploring amharic: compare the finding and its evidence limits.
Can an AI tell whether an Amharic sentence is a warning, refusal or joke?
A new nine-class benchmark tested five approaches on 11,475 labelled sentences. AfroXLM-R led the held-out comparison, but its 76.93% macro-F1 leaves consequential errors unresolved.
Read the next reportExplore the full source and evidence record
What this means for people
- A person using an agent in Amharic, Urdu, Bengali or Hindi may receive a less reliable service even when the website offers an equivalent localized task.
Global context
The languages span South Asia and East Africa, but the study does not cover regional dialects, mixed-language use or many other lower-resource languages. Localized deployment needs local testing.
What the evidence does not yet show
- This unreviewed benchmark uses constructed sites, AI-assisted translation, eight models and binary task success; it does not measure satisfaction, accessibility or harm in live services.
What to watch next
- Native-speaker audits, more languages and dialects, real-site evaluations, accessibility testing and evidence that multilingual post-training improves live task completion without new errors.
Living evidence record
Impact record IAI-035YLWY
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
21 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 21 October 2026
Research question, topics and regions
How web-agent task success changes between English and four lower-resource languages, under both ordinary pages and common interface disruptions
Topics: Web agents · Language access · Amharic · Urdu · Bengali · Hindi
Regions: India · Bangladesh · Pakistan · Ethiopia · Africa · Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.