Can AI agents conduct publishable physics research? The best FrontierPhysics setup passed 46 of 168 rollouts
An unreviewed benchmark gave agents 56 research projects built by physicists, up to ten hours per attempt and both numerical and manuscript tests. The strongest configuration passed 27.4%, while expert audit found rigorous reasoning, physical intuition and literature synthesis—not basic tool use alone—were persistent failure points.
The short version
- What happened
- Forty-two physicists created 56 tasks across nine physics domains from research they had personally conducted. Twenty-two models in 26 configurations attempted each task three times, producing 4,364 valid rollouts under a ten-hour cap.
- Who it affects
- Physicists may gain useful drafting and computational support, but must still verify assumptions, edge cases and physical constraints independently.
- What remains unproven
- Unreviewed preprint developed and evaluated by the benchmark team, without independent replication reported.
Keep in mind
Unreviewed preprint developed and evaluated by the benchmark team, without independent replication reported.
The 56 tasks reconstruct contributors' completed research and are not prospective unknown discoveries.
Automated reviewer agents and task-specific rubrics produced nine false failures in one 168-rollout audit cohort.
Three rollouts per task, changing provider prices and different harnesses limit broad model and cost comparisons.
The accessible manuscript provides affiliations but no standalone funding or competing-interest statement.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Read the full report · 10 min The evidence, methods, implications and limitations
The answer: the agents sometimes completed real research-shaped work, but none was reliably publishable
FrontierPhysics found that current tool-using AI agents could complete some end-to-end physics projects, yet even the strongest tested model-and-harness configuration passed only 27.4% of its attempts. Claude Opus 5.5 running in Claude Code passed 46 of 168 rollouts: three attempts on each of 56 tasks. Every other configuration remained below that level. A pass was deliberately demanding. The agent had to satisfy deterministic tests for computed results and clear every blocking criterion written by a domain expert. A fluent manuscript could not compensate for wrong numbers, and correct numbers could not compensate for a scientifically unacceptable argument.
That result is more informative than saying an agent solved a physics question. The tasks ask for literature review, a research plan, coding or experiments, analysis and a manuscript. They were constructed by physicists from projects they had personally carried out rather than converted from textbook exercises with a known short answer. Even so, the benchmark is not evidence that an agent made a new discovery or ran safely in a live laboratory. It reconstructs prior research inside controlled sandboxes, with hidden reference values, expert rubrics and a fixed time limit. The correct conclusion is that research-shaped evaluation is becoming possible while dependable autonomous physics remains unproven.[1]
What was measured: 56 projects, nine domains and 4,364 rollouts
The benchmark covers condensed-matter and atomic, molecular and optical physics; quantum information; astrophysics; fluid dynamics; statistical, photonics and imaging, high-energy and chemical physics. Forty-two physicists contributed the 56 accepted tasks. Each underlying project had to represent at least two weeks of prior human research. Contributors supplied provenance, a project timeline, a high-level instruction and deterministic artifacts such as computed quantities, simulations or processed data. They also wrote task-specific rubrics that could test whether the agent found key literature, respected experimental constraints, used the correct physical model and supported the claims in its paper.
The study evaluated 22 models in 26 configurations using four agent harnesses. Models attempted every task three times at their highest available reasoning effort, with selected GPT systems also tested at a higher setting. Each rollout could run for up to ten hours. Infrastructure failures were retried, while agent-caused endings—including exhausting a budget or sandbox memory—stayed in the denominator. The complete evaluation contained 4,364 valid rollouts. This scale supports comparisons within the released benchmark, but the three attempts for a task are correlated. The authors therefore report approximate binomial intervals and a task-level bootstrap rather than presenting every rollout as an independent research project.[1]
The distinction that matters
How a FrontierPhysics attempt becomes a pass
A manuscript is counted only when the artifacts, blocking scientific criteria and weighted quality checks all survive review.
Project
Agent receives a high-level physics brief
56 expert-built tasks
Research
Search, plan, compute and write
Up to 10 hours per attempt
Verify
Run hidden deterministic tests
Wrong artifacts fail
Review
Apply expert blockers and rubrics
Only fully cleared work passes
Passing required both correct artifacts and a defensible scientific paper
FrontierPhysics separates numerical correctness from research quality. Deterministic verifiers check the artifacts that can be tested mechanically. A reviewer agent then applies expert-authored blocking and non-blocking criteria to the manuscript and trajectory. Blocking criteria can cover a fatal physical error, unsupported conclusion, missed real-world constraint or failure to follow the task. Non-blocking criteria receive importance weights and contribute to a quality score, but that score becomes zero if a deterministic check or any blocker fails. This design prevents polished prose from laundering incorrect computation and prevents a technically correct file from being counted as publishable when its reasoning is unsound.
There is an unavoidable measurement trade-off. Expert rubrics encode information that generic graders lack, but the benchmark team helped design the tasks and selected what counts as a fatal failure. Reviewer models can also misread a rubric. For 33 tasks with an author paper or manuscript available, the same rubric process gave the human reference material an average score of 96.2%, a useful sanity check but not proof that every criterion is fair. Because the evaluation mixes deterministic tests with model grading, the leaderboard reflects both agent capability and the quality of the benchmark's specifications, hidden tests, rubrics and reviewer implementation.[1]
The strongest result was still a minority of attempts, and price did not guarantee reliability
All tested configurations had an overall pass rate below 30%. The top result, 27.4%, means most attempts from the best configuration still failed at least one decisive requirement. The paper also reports Pass@3—the share of tasks with at least one success across three attempts—but a single successful rerun is not the same as operational reliability. A laboratory or research group needs to know which attempt is correct before seeing the hidden answer. Repeated sampling can increase the chance that one output is good while simultaneously increasing the review burden and the chance that a convincing but wrong manuscript is selected.
Higher cost generally accompanied higher pass rates, but the relationship was not monotonic. Some more expensive Claude configurations performed worse than a cheaper one, while GPT configurations were described as more cost-efficient within this setup. Those comparisons are sensitive to provider prices, harness overhead, rate limiting and tool accounting. Time excludes environment setup, verification and rubric review; output-token reporting varies by provider; and Claude Code resource counts include subagent work. The benchmark therefore supports a practical warning rather than a purchasing league table: more inference effort or spending cannot replace task-specific validation, and cost comparisons may move as models, prices and harnesses change.[1]
Expert audit found reasoning and intuition failures—and also mistakes in the benchmark
The team manually reviewed the 168 GPT-6 Astra rollouts, using task authors and administrators to inspect trajectories and files. Nine attempts had been incorrectly failed: eight because of task, verifier or rubric defects and one because the reviewer agent misgraded a correct result. Correcting those outcomes raised that configuration's pass rate from 14.9%, or 25 of 168, to 20.2%, or 34 of 168. That is a material shift. It demonstrates why benchmark scores should not be read as exact capability measurements, especially when open-ended work is converted into hidden tests and automated judgments.
Among 143 reviewed failed attempts, 48% carried a lack-of-rigour label, 34% lacked expert intuition and 27% had literature problems; attempts could receive multiple labels. The examples are concrete. Agents used their own assumptions to validate themselves, missed edge cases, ignored device calibration or realistic shot budgets, treated unsafe process settings as stable, or reached correct numerical output while omitting the scientific significance. Basic physics knowledge and tool execution were not the only bottlenecks. A system can know an equation and run code yet still fail to ask whether its method would detect its own error or whether the result respects the physical apparatus.[1]
What this means for physicists, research leaders and funders
For physicists, the benchmark suggests useful bounded roles before autonomous research. An agent may search literature, implement a candidate calculation, generate alternative analyses or draft a methods section, while a domain expert controls the question, checks assumptions and validates the artifacts independently. The failure taxonomy can become a review checklist: ask whether validation is circular, whether edge cases were tested, whether apparatus constraints were included, whether decisive literature was found and whether the paper's claim is narrower than its evidence. That approach treats the agent as an accelerant for inspectable work rather than an authority whose manuscript is accepted because it looks complete.
Research leaders should budget for verification rather than comparing only token cost or time-to-draft. Ten hours of autonomous runtime can still produce a result that a specialist must reconstruct. A low pass rate may be acceptable for generating hypotheses if outputs are clearly quarantined, but not for releasing safety-critical experimental settings or allocating funding. Funders and publishers also need disclosure rules for agent-produced code, literature searches and manuscripts, including model version, harness, tools, reruns and human corrections. FrontierPhysics does not measure whether AI improves a team's total productivity, training or creativity; those outcomes require prospective studies with human research groups and real deadlines.[1]
What remains unproven—and the next research question
The paper is an unreviewed preprint produced by the benchmark's own developers. The authors span universities, national laboratories and companies including Amazon, IBM, ZEISS and Prime Intellect; the manuscript notes that specified Amazon contributions were conducted outside the authors' roles, but the accessible paper contains no standalone funding or competing-interest statement. Tasks reconstruct the contributors' prior projects, which improves authenticity but may also align them closely with the chosen rubrics and reference solutions. English-language papers, cloud sandboxes and research-computing tasks do not represent every laboratory, institution or language.
The most useful next question is not whether a higher leaderboard score can be obtained. It is whether independent physicists can preregister tasks, run the released environments and select correct outputs without access to hidden answers, while measuring total human review time and scientific value. Stronger evidence would include independent benchmark governance, blind domain regrading across all model families, prospective projects whose answer is not already known to the task author, and comparisons between human-only, AI-assisted and agent-led teams. Until then, FrontierPhysics provides a demanding diagnostic of current agents. It does not establish publishable autonomous science, a replacement for physicists or a forecast of discovery rates.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring physics: compare the finding and its evidence limits.
Can AI agents complete end-to-end physics research? The best configuration passed 46 of 168 rollouts
In the unreviewed FrontierPhysics benchmark, 22 models attempted 56 research tasks three times each. Claude Opus 5.5 led at 27.4%, while a manual audit found nine false failures in one GPT-6 Astra configuration—evidence that both agent reasoning and benchmark grading remain difficult.
Read the next reportExplore the full source and evidence record
What this means for people
- Physicists may gain useful drafting and computational support, but must still verify assumptions, edge cases and physical constraints independently.
- Research managers should count expert review and rerun costs alongside inference cost; a fast manuscript is not a validated result.
- Students and early-career researchers need transparent disclosure and supervision so agent output does not obscure who understands or is accountable for the work.
Global context
The contributor list spans North America, Europe and Asia and the tested providers include models from several countries. However, the tasks and manuscripts are English-language and run in cloud research environments. Access to compute, literature, specialist reviewers and safe experimental infrastructure varies globally, so the leaderboard cannot be transferred directly to laboratories with different resources or governance.
What the evidence does not yet show
- Unreviewed preprint developed and evaluated by the benchmark team, without independent replication reported.
- The 56 tasks reconstruct contributors' completed research and are not prospective unknown discoveries.
- Automated reviewer agents and task-specific rubrics produced nine false failures in one 168-rollout audit cohort.
- Three rollouts per task, changing provider prices and different harnesses limit broad model and cost comparisons.
- The accessible manuscript provides affiliations but no standalone funding or competing-interest statement.
What to watch next
- Independent reruns of the released tasks, harnesses and grading pipeline.
- Prospective projects whose outcomes were not already known to task authors.
- Human-only versus AI-assisted comparisons measuring total time, corrections and scientific value.
- Blind expert regrading across all model families and explicit benchmark conflict disclosures.
- Multilingual tasks and tests in laboratories with different compute and safety constraints.
Living evidence record
Impact record IAI-1206TMI
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what FrontierPhysics research team published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
Whether tool-using language-model agents can complete open-ended physics research projects, produce correct numerical artifacts and write manuscripts that satisfy expert-defined publishability criteria
Topics: AI for science · Physics · Research agents · Benchmarks · Scientific reliability · Human oversight
Regions: United States · Europe · Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.