What breaks when an AI agent meets a messy customer brief?
Across 95 simulated delivery jobs, the leading configuration resolved only 14 without an incident. Finding the missing facts was necessary—but still not enough to deliver the right result.
The short version
3 key points
- 1.FDE-Gym contains 95 tasks derived from 20 delivery scenarios across 18 sectors, with production-like systems and scripted stakeholders.
- 2.The best of 13 tested configurations averaged 0.56 on a scale where the unchanged system is zero and an expert reference is one, resolving 14 tasks without an incident.
- 3.The benchmark is simulated and lacks a completed human baseline; its operator also developed and maintains the platform.
Agents could build the requested thing and still miss the real problem
An AI agent can complete a polished piece of work and leave a customer no better off. In a new benchmark built around ambiguous enterprise projects, the strongest tested configuration resolved only 14 of 95 tasks. The main failure was not simply coding. Agents struggled to discover unstated rules, decide when a sponsor's request targeted the wrong problem and turn what they learned into a working improvement.
FDE-Gym adapts 20 delivery projects from China, Singapore and the United States into 95 tasks across 18 sectors. Each workspace contains a production-like system, documents, historical data and more than 130 scripted stakeholders. The agent can ask questions and alter the system. Its output is then deployed unchanged and scored on held-out data using the customer's own measure, such as cleared reconciliation breaks or correctly handled support cases.[1]
The test rewards business improvement, not literal compliance
The unchanged system scores zero and an expert reference scores one. Contacting stakeholders, running trials and waiting can reduce the score; an operational incident wipes out positive uplift. A task counts as resolved only at 0.8 or above without an incident. Thirteen configurations from seven providers each ran 152 task-and-instruction combinations three times, capped at 200 turns, 150 minutes or eight US dollars per run.
At the hardest instruction level, the leading configuration averaged 0.56, with a 95% bootstrap interval of 0.50 to 0.62. Five configurations resolved no tasks. Giving leading models the relevant rules and constraints raised their scores to 0.73–0.82; requiring them to uncover those facts reduced them to 0.39–0.62. Yet discovery was not the whole answer. On 23 tasks where the leading model obtained every key fact, only 10 achieved at least 0.8 of the reference business uplift.[1]
The distinction that matters
Why a completed deliverable can still fail
FDE-Gym separates the sponsor's request from the customer's underlying result.
Brief
Receive an incomplete or misdirected request
The visible task may not describe the real problem
Discovery
Question stakeholders and inspect systems
Rules and constraints become available
Delivery
Deploy the work on held-out data
Only measured customer uplift earns credit
This is a warning against removing the delivery team too early
For workers, the result suggests that requirements gathering, stakeholder judgement and accountable sign-off remain central skills even when an agent can edit code and operate tools. A manager should not read benchmark success on tidy tickets as evidence that a system can safely interpret a vague operational goal. The benchmark also found weaker models making some simulated customers worse off than the unchanged baseline.
The evidence is still a simulation. Stakeholders followed scripts, companies were de-identified or fictionalised, and the 75 variants and reference solutions are held out. A human baseline is being collected but was not available, so the study cannot say whether agents are close to experienced engineers. It also cannot show whether mixed human-agent teams would outperform either side alone. DeepLethe runs the benchmark and leaderboard, creating a platform interest the authors disclose. Independent replication with real delivery teams, live organisational constraints and audited incidents would change the assessment. For now, the practical lesson is to evaluate agents on the customer's outcome—and keep people responsible for discovering what the brief leaves unsaid.[1]
Evidence check
The tasks are controlled simulations with scripted stakeholders and no completed human baseline. The benchmark operator is also an interested platform developer.
Reporting basis: Source analysis. Original source date: 15 October 2026.
Explore the full source and evidence record
What this means for people
- Workers may spend less time implementing routine changes but more time defining the real problem, checking constraints and owning deployment decisions.
- Customers need outcome-based acceptance tests because an agent can satisfy the written request without improving the underlying service.
Global context
The scenarios draw on projects from China, Singapore and the United States and span 18 sectors. Real organisations will differ in labour practices, regulation, data access and tolerance for automated action.
What the evidence does not yet show
- The tasks are controlled simulations with scripted stakeholders and no completed human baseline. The benchmark operator is also an interested platform developer.
What to watch next
- The promised expert human baseline, independent reproductions and audited field studies comparing agent-only, human-only and collaborative delivery.
Living evidence record
Impact record IAI-1KYVMRI
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
16 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Yang et al. (arXiv) published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 16 October 2026
Research question, topics and regions
Agent performance on ambiguous, end-to-end customer delivery tasks
Topics: AI agents · Work · Enterprise deployment · Customer requirements · Human judgement
Regions: China · Singapore · United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
UNESCO
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
AI, jobs & human choice · Background explainer
Published 2024-10-14 · 11 min 27 sec
UNESCO Talks: Interview with 2024 Nobel laureate Professor Daron Acemoglu
Economist Daron Acemoglu discusses how choices about AI affect workers, inequality and society. Pair the interview with our comparison of task exposure, hiring and skills evidence.
Evidence check: An expert interview from 2024, offering a perspective on policy and work rather than a measured forecast of job losses.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Work & Skills
Do agentic tools now make up most business AI use in the EU?
Within OpenAI's Enterprise and Business customer sample, yes: Codex and ChatGPT Work produced 55% of included output tokens at the end of September. That is not a measure of all EU firms, completed work or productivity.
10 min · 5 sources
Work & Skills
Can AI agents configure complex contract workflows?
OpenAI reports that GPT-6 Astra scored 55.0% across 11 Ironclad tasks versus 41.6% for GPT-5.6 Sol, while simulated time fell from 37.0 to 19.2 minutes. The company-run benchmark shows progress—not autonomous legal work or measured customer productivity.
7 min · 1 source
Work & Skills
Does an AI label bias code review—or does job rank matter more?
In a randomized vignette experiment, 447 Microsoft engineers did not penalize an AI-use disclosure on identical code, but rated the same code and author more favourably when the fictional author was labelled Principal rather than Level 1. One AI-normalized US workplace cannot settle how other teams respond.
9 min · 2 sources
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.