Can AI agents safely sign off real engineering work? This benchmark says not yet
Google and Mercor tested agents on 85 civil, structural, electrical and mechanical assignments. The strongest native setup averaged 48.3%, while only 41.3% of its deliverables cleared every hard safety and feasibility check.
The short version
- What happened
- The benchmark contains 85 long-horizon tasks: 30 mechanical, 27 electrical, 18 civil and 10 structural, with an average of 61 project files per task.
- Who it affects
- Engineers may gain drafting and checking assistance, but they need enough time, authority and traceability to challenge the tool rather than rubber-stamp it.
- What remains unproven
- The paper is an unreviewed preprint and the benchmark, release and leaderboard are operated by its creators.
Keep in mind
The paper is an unreviewed preprint and the benchmark, release and leaderboard are operated by its creators.
The 85 tasks are curated office deliverables, not a representative sample of all engineering practice or built projects.
Open-ended outputs use an LLM rubric judge; automated checks cannot encode every relevant professional judgment.
Only 29 tasks are public, while the 56-task hidden set cannot be independently inspected in full.
No matched human baseline, field safety outcome, productivity denominator or employment effect is established.
Multi-source analysis · 4 linked sources. Primary analysed source date: 3 November 2026. Earliest linked supporting source: 21 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 8 min The evidence, methods, implications and limitations
The direct answer: these agents are not ready to approve physical designs on their own
No. Physical-Engineering's Last Exam gives current AI agents something closer to an engineering project folder than a textbook question, and the strongest native model-and-tool setup still averaged 48.3% on the hidden test set. More importantly, a design earned cost-efficiency credit only after clearing every hard safety and feasibility check. The top configuration met 91% of individual constraints on the co-authored tasks, yet only 41.3% of its deliverables passed all hard checks. That difference is the practical finding. A system can look broadly competent while missing one requirement that prevents a drawing, calculation or technical memo from being acceptable. The benchmark does not prove that a 48% score corresponds to a 52% chance of unsafe construction, and none of the outputs became signed real-world designs. It does show why a high average, fluent explanation or nearly complete checklist cannot substitute for licensed review. For engineers, the plausible near-term role is bounded assistance—searching files, drafting calculations or checking revisions—with responsibility, verification and sign-off remaining with people who can inspect the complete project context.[1][2][3]
What the 85 tasks ask an agent to do
The benchmark covers 30 mechanical, 27 electrical, 18 civil and 10 structural tasks. Examples include choosing a foundation strategy from geotechnical records, checking roof diaphragms for wind or seismic loads, sizing electrical feeders, coordinating cable tray and chilled-water routes, reviewing a non-converging fire simulation and preparing an engineer's estimate. Each task runs in an isolated container and starts with a project folder containing an average of 61 files; the complete benchmark holds 5,170 files. Those files include PDFs, spreadsheets, emails, specifications, two-dimensional CAD drawings, three-dimensional building models and document revisions. The agent must decide which information governs, use local tools, run calculations and create a specified deliverable. It has no open internet access and may use up to six hours. This design improves on short-answer benchmarks because much professional work is information selection and document control, not just recall. But it still represents selected office deliverables inside a controlled environment. It excludes site visits, conversations with clients and contractors, changing conditions, formal professional liability and the full chain by which engineering decisions are checked, approved, built and maintained.[1][2][3][4]
The distinction that matters
Why an average score is not enough for engineering approval
PLE separates broad task performance from the requirement to clear every safety and feasibility gate.
Project folder
Agent searches an average of 61 files
Relevant inputs and revisions must be selected
Deliverable
Agent calculates, checks or drafts a design record
Output is scored by code, an LLM rubric judge or both
Hard gates
Every safety and feasibility check must pass
Only then can cost-efficiency points count
Human sign-off
Licensed reviewer verifies context and responsibility
Benchmark performance does not replace approval
How the tasks and graders were constructed
The creators used two routes. Practising engineers wrote 44 expert-authored tasks, including the starting files, prompt and grading rubric. For 41 co-authored tasks, engineers set the initial problem while an agent pipeline assembled the environment, reference deliverables and executable grader from licensable standards; engineers then validated the task and increased its difficulty. Twenty-nine tasks are public development material and 56 form the hidden test set. Quantitative outputs can be checked by software, while open-ended work is scored against an expert rubric by an LLM judge. Some tasks use both. Every tested model-harness pair is run three times and results are averaged, reducing but not eliminating run-to-run variation. This is more transparent than a single demonstration because the public repository exposes development tasks and evaluation code. Still, the denominator is 85 curated assignments, not the universe of physical-engineering work. Co-authored tasks may preserve assumptions introduced by the same class of systems being evaluated, and an LLM judge may reward or miss features differently from a licensed human reviewer. The paper reports sensitivity work with alternative judges, but independent reproduction on the hidden set is not yet available.[1][2][3]
How to read the 48.3% result—and the paper's different 50.6% figure
The public release ranks each model on its native harness, the software layer that gives a model tools and manages its work. On that comparison, Claude Opus 5.5 with Claude Code scored 48.3% across the 56 hidden tasks, followed by GPT 6 Astra with Codex at 45.8% and Gemini 4 Argon with Antigravity at 40.6%. The preprint reports a 50.6% strongest configuration across its broader matrix of model-harness combinations. These are not interchangeable league-table numbers: one is the native-harness headline used for the public competition, while the other reflects the paper's best evaluated pairing. The creators tested 14 frontier models across seven public harnesses. They say model choice produced spreads of up to 50 percentage points, while switching harnesses for the same model changed scores by a median of about three points, although harness choice affected cost. The strongest reported setup used about 27 million tokens and 63 minutes per task on average. That is evidence about benchmark resource use, not a comparison with an engineer's time, fees or quality. No human baseline with the same folders, time limit and scoring procedure is reported on the public page.[1][2][3]
What this means for engineers, firms and the public
For working engineers, the benchmark points toward task decomposition rather than replacement. A firm could test an agent on document search, drawing-change logs, first-pass calculations or specification checks, then require a named professional to verify inputs, reproduce the calculation and approve the output. Tasks with executable checks may be easier to bound than ambiguous judgments about constructability, unusual site conditions or acceptable risk. Employers should measure rework, missed constraints and review time, not merely how quickly an agent produces a polished file. Workers also need clarity about who is responsible when automation changes a calculation or selects the wrong revision. For clients and the public, a professional seal must continue to mean accountable human judgment under the applicable law. A benchmark score does not authorize a product for structural, electrical, civil or mechanical practice. Procurement teams should ask what task was evaluated, which standards and file formats were used, what failure rate occurred at the hard gates, how the tool logs its actions and how a reviewer can detect incomplete work. The evidence supports carefully supervised trials; it does not support unattended sign-off or staffing promises.[1][2][3][4]
Evidence limits, commercial interests and what would change the assessment
PLE is an unreviewed preprint and a benchmark built by Google researchers with Mercor, a company that supplies expert labour and operates the task environment. The public release, benchmark site and leaderboard are controlled by the creators. The 29 development tasks and code can be inspected, but the 56 hidden tasks cannot be audited in full without undermining the test. The leaderboard can change as models and harnesses change. Automated checks are valuable where requirements can be encoded; rubric-scored work inherits the limits of the selected rubric and judge. The tests do not measure safety after construction, rare environmental conditions, coordination with other disciplines, regulatory acceptance, liability, cyber security of project data, worker displacement or whether assistance improves a human team's outcomes. Confidence would rise with independent teams reproducing the public tasks, licensed engineers blind-reviewing model and human outputs, a matched human baseline, disclosed error categories and field trials that measure review time and caught versus missed defects. Evidence that an agent repeatedly clears all hard checks on unfamiliar projects and improves a team without weakening accountability would change the assessment. Until then, the result is consequential research, not Breaking news and not permission to automate professional sign-off.[1][2][3][4]
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 11 min 27 sec · Background explainer
UNESCO Talks: Interview with 2024 Nobel laureate Professor Daron Acemoglu
YouTube loads only when you choose; playback starts when you press play.
Economist Daron Acemoglu discusses how choices about AI affect workers, inequality and society. Pair the interview with our comparison of task exposure, hiring and skills evidence.
Evidence check: An expert interview from 2024, offering a perspective on policy and work rather than a measured forecast of job losses.
Published 2024-10-14. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the AI and jobs guidePhoto: Dylan Gillis / Unsplash · Licence
Continue the story
Read next
Continue exploring ai agents: compare the finding and its evidence limits.
Does OpenAI’s LifeVal show AI can manage everyday life?
No. OpenAI reports that its latest internal model scores 80.6% on an evaluation inspired by everyday tasks, up from 49.3% for GPT-5.2. But it has not published the task denominator, full scoring protocol, uncertainty or evidence that the systems work reliably in people’s homes and services.
Read the next reportExplore the full source and evidence record
What this means for people
- Engineers may gain drafting and checking assistance, but they need enough time, authority and traceability to challenge the tool rather than rubber-stamp it.
- Firms should measure missed constraints, rework and professional review time before changing staffing or approval processes.
- Clients and the public need professional sign-off to retain a clear, legally accountable human owner.
Global context
Engineering licensing, codes, liability and approval procedures differ by jurisdiction. A benchmark assembled around particular standards and file formats can reveal transferable failure modes, but its score cannot be treated as authorization to practise in any country. Local regulators, professional bodies, employers and workers must define acceptable use and responsibility.
What the evidence does not yet show
- The paper is an unreviewed preprint and the benchmark, release and leaderboard are operated by its creators.
- The 85 tasks are curated office deliverables, not a representative sample of all engineering practice or built projects.
- Open-ended outputs use an LLM rubric judge; automated checks cannot encode every relevant professional judgment.
- Only 29 tasks are public, while the 56-task hidden set cannot be independently inspected in full.
- No matched human baseline, field safety outcome, productivity denominator or employment effect is established.
What to watch next
- Independent reproduction of the public tasks and blind review by licensed engineers
- Matched human and human-with-agent baselines using the same folders and time limits
- Error categories showing which hard constraints models most often miss
- Field trials measuring defects caught, defects missed, rework and review time
- Professional and regulatory guidance on records, liability, confidentiality and sign-off
Living evidence record
Impact record IAI-0QO51T7
Evidence stage
Studied
Confidence
Supported
Reporting basis
Multi-source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
4 November 2026
Source trail
4 direct sources across 3 source types.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Evidence trail
Sources used for this report
Links checked 4 November 2026
Research question, topics and regions
Whether AI agents can complete realistic physical-engineering deliverables under safety, feasibility and cost constraints
Topics: Engineering work · AI agents · Professional safety · Benchmarking
Regions: United States · Global
This report is labelled multi-source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.