Can AI agents do theoretical computer science research? A 100-task benchmark verified proofs and constructions—but not independent solutions to long-standing conjectures
The unreviewed TCSExam benchmark gave agents 100 research-level tasks spanning formal proofs, solver engineering, mathematical constructions and open conjectures. One system completed 14 of 20 bare theorem tasks, yet the apparent conjecture resolutions reused already published ideas and the benchmark was built partly by the company whose model it evaluates.
The short version
- What happened
- TCSExam-Full contains 100 expert-reviewed tasks: 20 Lean theorem proofs, 30 solver-engineering problems, 20 mathematical constructions and 30 open conjectures. Outputs are checked by the Lean kernel or executable task checkers.
- Who it affects
- Researchers may gain useful formalisation, solver and construction assistance, but remain responsible for choosing questions, validating task fidelity and establishing novelty.
- What remains unproven
- The paper is an unreviewed preprint from a team with academic and Moonshot AI affiliations; no separate funding or competing-interests statement is provided.
Keep in mind
The paper is an unreviewed preprint from a team with academic and Moonshot AI affiliations; no separate funding or competing-interests statement is provided.
Attempt counts and some execution protocols differ between agents and between Full and Lite, limiting simple leaderboard comparisons.
Formal verification proves the encoded statement, not that the statement faithfully captures the source or that the result is novel.
Accepted open-conjecture submissions adapted already published counterexamples and proofs rather than independently settling the problems.
The benchmark measures bounded tasks with fixed interfaces and large compute budgets, not autonomous question selection or long-term scientific judgement.
Source analysis · 1 linked source. Primary analysed source date: 5 November 2026.
Read the full report · 10 min The evidence, methods, implications and limitations
The answer: agents produced verifiable research artefacts, but the benchmark does not show autonomous discovery at full scientific reliability
A new 100-task benchmark provides unusually concrete evidence that frontier AI agents can perform parts of theoretical computer science. Instead of asking models to explain textbook results, TCSExam asks them to submit complete Lean proofs, reusable solver programs, explicit mathematical constructions and proof-or-refutation attempts for open conjectures. Every final artefact is checked by a formal proof kernel or an executable verifier. That makes the strongest results more meaningful than an answer graded only by another language model.
The leading agent on the Full suite, GPT-6 Astra, completed 14 of 20 research-level theorem tasks from the bare statement and definitions, a 70% per-attempt success rate. Other systems improved solver outputs or construction references, and several produced verified special cases or bounds. Those are real benchmark accomplishments. They are not equivalent to independently settling a long-standing open problem, publishing a new theorem or replacing expert researchers. The paper explicitly notes that accepted open-conjecture submissions adapted counterexamples and proofs already published elsewhere.
The study is also an unreviewed preprint from a mixed academic-company team. Three authors list Moonshot AI affiliations, and Moonshot models are among those evaluated. The benchmark code, task sources and Docker recipes are public, but the paper gives no separate funding or competing-interests declaration. Readers should therefore treat the reported rankings as reproducible evidence from the benchmark authors, not an independent market comparison or proof of general research autonomy.[1]
What the 100 tasks measure—and why four tracks matter
Twenty theorem-proving tasks ask an agent to complete a Lean 4 proof of a result derived from recent theoretical-computer-science papers. The main 'bare' setting supplies the statement, definitions and mathematical assumptions without intermediate lemmas. Ladder and guided variants add checkpoint statements or proof ideas, allowing the researchers to distinguish direct proof construction from the ability to fill in a scaffold. A proof only counts as complete when Lean verifies the assembled result without unproved checkpoint assumptions.
Thirty solver-engineering tasks require programs that work across families of instances rather than one hand-picked example. Agents see public development instances and a naive baseline; final evaluation uses designated instance sets and task-specific resource limits. Another 20 tasks ask for explicit mathematical objects—codes, graph colourings or geometric configurations—with certificates that executable checkers or Lean can verify. Scores compare the submitted object with a documented reference, so exceeding 100% means beating that task reference, not proving a globally new record.
The final 30 tasks expose open conjectures in prove-or-refute form. Full credit requires a complete formal proof of the statement or its negation. Failed attempts receive zero on the resolution score, although the paper separately records verified partial bounds, finite cases and counterexamples to related claims. This separation is crucial: progress towards a conjecture can be scientifically useful, but it must not be presented as a solution when the central proof obligation remains open.[1]
The distinction that matters
Four different research outputs need four different checks
TCSExam separates artefacts that are often collapsed into a single claim that an agent can 'do research'.
Theorem
Submit a complete Lean proof
Lean kernel accepts or rejects
Solver
Write a reusable program
Hidden instances measure performance
Construction
Produce an object and certificate
Checker compares the achieved parameter
Conjecture
Prove the statement or its negation
Anything incomplete remains unresolved
How agents were compared: different repetition counts, fixed budgets and verifiable outputs
TCSExam-Full evaluates eight agents and TCSExam-Lite evaluates nine. Full has all 100 tasks; Lite selects five tasks from each track, cutting the number of tasks by 80% while retaining the same 12-hour development budget per attempt. Eight shared agents produced strongly aligned Full and Lite rankings, with a reported Spearman correlation of 0.93. That supports Lite as a screening tool within this sample, but the subset was selected after performance profiles from the original six agents were available, so it is not a prospectively frozen benchmark for every future system.
The comparison is not perfectly symmetric. Kimi K3, Kimi K3.1 Pro and two DeepSeek variants receive three attempts per task, while several other agents receive one. Scores are averaged within task before being averaged across tasks, which limits but does not erase protocol differences. Full and Lite also use different public self-test arrangements for solver tasks, although hidden judging, input splits and final limits are intended to match. A single aggregate index weights theorem proving, solvers and constructions at 30 points each and open-conjecture resolution at 10; those weights are an editorial choice by the benchmark designers, not a natural law of research value.
The authors reduce several common evaluation risks. Hidden solver instances are withheld during development, structure-preserving transformations limit direct reuse, and unexpected successes trigger renewed checks of statements and source fidelity. More than 40 theorem candidates were retired after agents solved them in pilot runs. Every retained task has expert approval, while outputs are checked by Lean or task-specific code. These controls strengthen correctness, but they cannot by themselves establish novelty, significance or whether a solver exploits an unintended but valid corner of a task definition.[1]
The results are uneven: strong formal proofs, mixed engineering and no clean leap to open-ended discovery
On Full, Astra's 70% bare-theorem success stands well above Kimi K3.1 Pro's 38.3% across repeated attempts and the 0% to 11.7% range for the remaining systems. Yet model strengths do not move together. A DeepSeek agent produced a verified construction better than the task reference despite completing none of its 60 theorem attempts in the tested version. Solver results also vary by instance family. A leaderboard number therefore compresses materially different abilities: formal proof completion, software search, optimisation and construction are not interchangeable forms of competence.
The open-conjecture column requires especially careful reading. Kimi K3.1 Pro and one DeepSeek version receive non-zero resolution scores, but the paper states that the accepted submissions adapt OpenAI's already published counterexamples and proofs. The benchmark still usefully checks whether an agent can reconstruct and formally validate that material under the task interface. It does not support a headline claiming that the evaluated agents independently solved major open conjectures. Other attempts yielded verified special cases and bounds, which may be promising starting points but need expert comparison with the literature.
Even a formally valid proof answers only the encoded statement. If a task omits a condition, translates a source incorrectly or permits a degenerate interpretation, the kernel can certify the wrong research question perfectly. The authors address this with agent-assisted fidelity reviews, boundary cases and human approval, and say tasks with unresolved source inconsistencies were removed. Independent domain experts should still audit any claimed advance before it is treated as new knowledge.[1]
What this changes for researchers, funders and universities
For theoretical computer scientists, the clearest near-term use is bounded assistance: formalising a theorem, searching a construction space, writing a solver or checking a finite case. Verifiable outputs can save time because a proof kernel or checker rejects many seductive mistakes automatically. Researchers still choose the question, inspect the formalisation, establish novelty and decide whether an improvement matters. The benchmark offers no evidence that an agent can independently select an important programme, understand its social value or maintain a reliable research agenda over months.
For universities and research organisations, procurement should focus on workflow evidence rather than the headline index. A useful tool should preserve the complete prompt, environment, tool calls, proof objects, code and failed attempts; allow independent re-execution; and make literature access and contamination controls explicit. The 12-hour per-task budget and substantial parallel infrastructure also mean that apparent autonomy may hide significant compute and engineering costs. Cost, review time and reproducibility belong beside task success when judging productivity.
For funders and publishers, machine-verifiable work could widen participation and accelerate checking, but it also increases the volume of plausible submissions. Policies should require disclosure of model and agent versions, human contributions, execution budgets and automated verification. Formal validity should not be confused with originality, and a benchmark team's company affiliation should be visible when commercial models appear in the comparison. The safest interpretation is that AI can now complete meaningful slices of formal research under tightly specified conditions—not that the scientific method has been automated end to end.[1]
What would change the assessment
Confidence would rise if independent theoretical-computer-science groups reproduced the results from the released containers, audited a representative sample of task translations and verified the claimed construction and solver improvements against the literature. Future versions should freeze tasks before model access, apply the same attempt counts and compute budgets to every agent, and report uncertainty across repeated runs. A blinded expert assessment of novelty would help separate valid rediscovery from genuinely new results.
The claim of autonomous research would require more than higher benchmark scores. An agent would need to identify a consequential unsolved question without benchmark leakage, develop a defensible approach, produce a novel result, survive adversarial expert review and support independent replication. It would also need to recognise when a formal target is unimportant or incorrectly specified. TCSExam is valuable because it makes several research outputs checkable; it is not yet a field trial of that broader capability.
The assessment would weaken if hidden-instance leakage, task-definition shortcuts, selective reporting or irreproducible vendor endpoints materially explain the results. It would strengthen if Lite continues to predict Full performance for future models and if versioned corrections remain transparent. For now, the benchmark is strong evidence that frontier agents can create verifiable artefacts in formal computer science, paired with equally strong evidence that verification, expert task design and literature checking remain indispensable.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring research agents: compare the finding and its evidence limits.
Can AI agents conduct publishable physics research? The best FrontierPhysics setup passed 46 of 168 rollouts
An unreviewed benchmark gave agents 56 research projects built by physicists, up to ten hours per attempt and both numerical and manuscript tests. The strongest configuration passed 27.4%, while expert audit found rigorous reasoning, physical intuition and literature synthesis—not basic tool use alone—were persistent failure points.
Read the next reportExplore the full source and evidence record
What this means for people
- Researchers may gain useful formalisation, solver and construction assistance, but remain responsible for choosing questions, validating task fidelity and establishing novelty.
- Universities and funders need complete execution records and equal-budget comparisons before treating benchmark scores as productivity evidence.
- Students should not infer that a formally checked output is automatically original, important or acceptable without attribution and expert review.
Global context
The benchmark was built by researchers affiliated with Tsinghua University, Moonshot AI and the Shanghai Qizhi Institute, but it evaluates models from organisations in China, the United States and elsewhere. Formal proof kernels and public containers make cross-border replication possible. Access to expensive agent runs and proprietary endpoints remains uneven, so strong scores do not imply that every university can reproduce the same workflow at comparable cost.
What the evidence does not yet show
- The paper is an unreviewed preprint from a team with academic and Moonshot AI affiliations; no separate funding or competing-interests statement is provided.
- Attempt counts and some execution protocols differ between agents and between Full and Lite, limiting simple leaderboard comparisons.
- Formal verification proves the encoded statement, not that the statement faithfully captures the source or that the result is novel.
- Accepted open-conjecture submissions adapted already published counterexamples and proofs rather than independently settling the problems.
- The benchmark measures bounded tasks with fixed interfaces and large compute budgets, not autonomous question selection or long-term scientific judgement.
What to watch next
- Independent replication of the released tasks, containers and claimed improvements.
- Equal-budget, repeated comparisons with frozen agent versions and prospectively selected tasks.
- Expert novelty reviews of verified special cases, bounds, constructions and solver outputs.
- Benchmark corrections or evidence of leakage, shortcuts or task-translation errors.
- A genuinely new conjecture result that survives public expert review and independent reproduction.
Living evidence record
Impact record IAI-1IIMJ06
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
Whether frontier language-model agents can produce verifiable theoretical-computer-science proofs, solver programs, constructions and progress on open conjectures
Topics: Research agents · Formal mathematics · Benchmarking · Theorem proving · Computer science · Verification
Regions: China · Asia & Middle East · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.