Can AI agents complete end-to-end physics research? The best configuration passed 46 of 168 rollouts

How 42 physicists turned their own work into 56 agent tasks

The task set began with 182 submissions from physicists. After a general quality review and a domain peer review, 56 were accepted. They came from 42 contributors at 26 institutions and cover nine domains spanning theoretical, computational and experimental physics. Contributors based tasks on their own research and supplied materials needed for evaluation while withholding information that would reveal a solution. This design gives the authors unusually strong knowledge of what a good answer should contain, but ties the sample to the research interests, task-writing choices and institutions represented by the contributor group.

Each of 22 models, arranged into 26 model-and-harness configurations, attempted every task three times. That produced 4,368 nominal task slots; the paper reports 4,364 rollouts after execution accounting. A ten-hour timeout applied to each rollout. Provider failures could be rerun, while endings caused by the model remained failures. The harnesses included Codex, Claude Code, Grok Build and Antigravity, with different inference settings. A model label therefore cannot be separated cleanly from the tools, prompts, sandbox, inference budget and agent loop around it.

The task mix is broad but not large within any one domain. With only 56 tasks divided across nine fields, differences between domains have wide uncertainty and should not be read as a league table of which areas of physics are easiest to automate. The benchmark also begins after experts have chosen and specified a task. It tests substantial execution and synthesis, but not the fully open-ended work of identifying a valuable scientific question, negotiating access to equipment, managing a laboratory or persuading a research community that a result is new and important.[1]

What passing measures—and why author manuscripts were used as a check

A rollout first faces deterministic verifiers designed for task-specific facts or outputs. It then receives rubric-based review for aspects that cannot be reduced to one number, such as justification, research quality and communication. The overall pass rate requires all deterministic tests and all blocking rubric criteria to pass. Non-blocking criteria contribute to a separate quality score. This layered scheme is sensible for open-ended work, but it transfers some uncertainty from the model to the judge: a rubric can be ambiguous, a verifier can encode a mistaken assumption and an automated reviewer can misunderstand a valid alternative.

To test whether the rubrics would accept expert work, contributors supplied their own papers or manuscripts for 33 tasks. When the same reviewer configuration graded manuscript-level criteria, those reference materials averaged 96.2%. That supports alignment between many rubrics and intended expert output. It is not a conventional human baseline: the authors' work helped define the tasks and the comparison covered only 33 tasks with suitable material. Nor does a 96.2% average show that every blocking criterion is correct.

The nine false failures reinforce the need for expert adjudication. They also make the benchmark more informative than a leaderboard that never inspects mistakes, because the paper discloses how grading error changed the result. Future versions need frozen tasks, versioned corrections and blind regrading by physicists who did not author the task. Publishing confidence intervals and task-level outcomes is helpful, but benchmark governance will matter as much as adding more models if researchers are to distinguish real capability progress from changed prompts, budgets or grading rules.[1]

The failure analysis points to reasoning and judgement, with a narrow denominator

For a closer look, contributors and administrators reviewed the trajectories and output files from the 168 GPT-6 Astra rollouts. The taxonomy tagged insufficiently rigorous thinking in 68 failures, or 48% of the 143 rollouts that failed before correction. Missing expert intuition appeared in 48, or 34%, and literature mistakes in 39, or 27%. Instruction-following problems appeared in 26%. Categories could overlap, so these percentages must not be added into a total. Examples of weak rigour included circular self-verification: an agent used its own result as evidence that the result was sound.

This is a diagnosis of one configuration on these tasks, not a universal explanation for every model failure. Claude Opus 5.5 led the aggregate table but did not receive the same complete manual taxonomy, and different harnesses may fail in different ways. The reviewers were also task contributors or benchmark administrators who knew the intended path. That gives them expertise, while making independent replication especially valuable. A blinded study could test whether outside physicists agree on which failures reflect reasoning, intuition, literature use or an underspecified task.

The distinction matters for research policy. If failure were mainly missing factual knowledge, retrieval or a larger context window might close the gap. If it is unstable reasoning and weak scientific judgement, giving an agent more tools or autonomy could instead let it produce a longer, more persuasive error. Laboratories considering agent assistance should therefore test intermediate claims, retain provenance for literature and code, and require domain experts to review the chain of evidence—not only the final manuscript.[1]

What this means for working physicists and research organisations

A 27.4% pass rate is high enough to make supervised assistance plausible and low enough to make autonomous reliance unsafe. Physicists may gain value from parallel literature searches, code drafts, sanity checks or first-pass analyses when failures are cheap to detect. The benchmark does not measure time saved by a human who can choose useful fragments from an unsuccessful rollout. Conversely, it does not measure the cost of checking confident but flawed work, information-security constraints around unpublished research or the opportunity cost of following a sophisticated wrong turn.

Institutions should resist converting one benchmark into a staffing forecast. The tasks were selected for evaluation, not sampled from all work done by physicists. They omit teaching, collaboration, equipment maintenance, grant work, ethical review and much of experimental practice. Model access and compute also differ sharply between well-funded laboratories and smaller institutions. If agent tools become a routine research input, reproducible records and access to verification resources will influence who benefits and who bears the review burden.

The immediate practical use is as a stress test. A team could compare its own workflows with the published task structure: define what must be deterministically true, specify blocking scientific criteria, save trajectories and assign an expert who can overturn automated grading. The assessment would change with preregistered, independently maintained benchmarks showing sustained improvement across new tasks, accompanied by evidence that scientists complete real projects faster or better without an increase in undetected error.[1]

What remains unproven

FrontierPhysics does not show that AI has or has not automated science. It shows that particular November 2026 model-and-harness configurations passed fewer than three in ten attempts on a contributor-built set of demanding physics tasks under a stated grading system. Results may change with model updates, more inference, a different scaffold or benchmark familiarity. Because the authors include researchers from universities, national laboratories and companies, replication should disclose model access, compute, prompts and any institutional or commercial support.

The decisive evidence would come from prospective use in research groups: new problems chosen before model access, independent expert review, a clear human-only or standard-tool comparator, and outcomes such as valid findings, time, cost, corrections and downstream publication. Until that exists, the responsible conclusion is narrower. Agents can sometimes assemble an acceptable end-to-end result, yet rigorous reasoning, expert judgement and evaluation reliability remain active bottlenecks. That is a useful research finding, not a Breaking event and not a warrant to remove human scientific responsibility.[1]