OpenAI’s legacy-code scores jumped sharply—does that make migration safe?
OpenAI reports large gains on a 20-language SQLite task and a hidden-test COBOL migration evaluation. The test suite is public, but the results remain company-run and do not establish that production modernisation is safe.
The short version
3 key points
- What changed
- OpenAI reports that GPT-6 Astra averaged an 87.4% test pass rate across 20 languages on MultiLangBench, compared with 26.8% for GPT-5.6 Sol.
- Why it matters
- Legacy-system teams have a stronger reason to run bounded pilots, but not to remove expert review or operational safeguards.
- What to watch
- Independent reproduction using the released MultiLangBench suite
A much stronger result on two bounded coding tasks
OpenAI reports a large jump in coding performance for languages with less public code and documentation than Python or JavaScript. MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in 20 languages. GPT-6 Astra averaged an 87.4% test pass rate, versus 26.8% for GPT-5.6 Sol. The company reports gains from 10.9% to 72.1% in COBOL, 21.4% to 83.5% in Fortran and 27.5% to 89.4% in Visual Basic.[1][2]
Who it affects
Legacy-system teams have a stronger reason to run bounded pilots, but not to remove expert review or operational safeguards.[1][2]
Keep in mind
OpenAI designed the evaluations, ran them and sells the evaluated models; there is no independent replication.
The article omits the COBOL migration denominator, per-target-language scores, repeated-run variation and uncertainty intervals.
The repository documents 24,000 converted SQLite tests and a recommended 272-test subset, but the article does not identify which produced the headline results.
Neither evaluation measures production defects, security, operational incidents, maintenance burden or worker outcomes.
Source analysis · 2 linked sources. Original source date: 19 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
A much stronger result on two bounded coding tasks
OpenAI reports a large jump in coding performance for languages with less public code and documentation than Python or JavaScript. MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in 20 languages. GPT-6 Astra averaged an 87.4% test pass rate, versus 26.8% for GPT-5.6 Sol. The company reports gains from 10.9% to 72.1% in COBOL, 21.4% to 83.5% in Fortran and 27.5% to 89.4% in Visual Basic.
The released repository makes the task inspectable. It converts 24,000 tests from SQLite's TCL suite into database inputs, queries and exact expected outputs. Because running all tests is expensive, the documentation recommends a 272-test subset for routine evaluations. OpenAI's article does not say whether its headline scores used that subset or all converted tests, how many complete model attempts were run, or whether uncertainty was calculated.[1][2]
COBOL migration is tested separately
A second evaluation asks models to translate COBOL programs into C++, Java, Kotlin, Python or Rust while preserving behaviour. Hidden tests require byte-for-byte parity on observable outputs; a migration passes only if every test passes. OpenAI reports 86.6% accuracy for GPT-6.1 Sol at low reasoning effort, against 5.4% for GPT-5.6 Sol, and 97.9% at high effort. It does not disclose the number or provenance of migration programs, per-target-language results, repeated-run variation or the hidden test suite, so outsiders cannot yet reproduce that headline result.[1]
The distinction that matters
What the two evaluations actually test
The company separates building a database engine from translating an existing COBOL program.
Build
Implement SQLite behaviour in 20 languages
Exact database tests measure partial functionality
Translate
Move COBOL into five target languages
All hidden output tests must pass
Deploy
Modernise a live production system
Not measured by either evaluation
Useful evidence, not permission to automate a cutover
For engineers maintaining banks, government services, weather systems or industrial software, the results support testing newer models on neglected languages. They do not show that an agent can understand undocumented business rules, live dependencies, security constraints or operational failure modes. The benchmark developer also sells the evaluated models, and no peer review or independent comparison is supplied.
A safe migration still needs a complete inventory, preserved reference behaviour, domain-expert review, representative tests, staged deployment and rollback. Evidence that would change this assessment includes independent reruns, a published COBOL denominator and per-language results, contamination checks, repeated trials with uncertainty, and prospective studies of real migrations measuring defects, review effort, incidents and total cost.[1][2]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Evidence check
OpenAI designed the evaluations, ran them and sells the evaluated models; there is no independent replication.
The article omits the COBOL migration denominator, per-target-language scores, repeated-run variation and uncertainty intervals.
The repository documents 24,000 converted SQLite tests and a recommended 272-test subset, but the article does not identify which produced the headline results.
Neither evaluation measures production defects, security, operational incidents, maintenance burden or worker outcomes.
Reporting basis: Source analysis. Original source date: 19 October 2026.
Continue the story
Read next
Also about software engineering
Is Gemini 4 Argon available now, and what do Google's benchmark claims show?
Google has announced a frontier model for coding, professional work and cyber defence, but access is initially limited to trusted defenders. Its 19-row comparison is broad and often strong; it is still a vendor evaluation, not an independent test or a public release.
Read the 7 min reportAlso about model evaluation
Does a cheaper small model make browser agents ready to scale?
Claude Haiku 5.5 sharply cuts Anthropic’s small-model token prices and posts large gains on the company’s selected computer-use tests. That makes high-volume automation more plausible—but the evidence is vendor-run and does not establish error-free performance in live workflows.
Read the 9 min reportExplore the full source and evidence record
What this means for people
- Legacy-system teams have a stronger reason to run bounded pilots, but not to remove expert review or operational safeguards.
- Workers who hold undocumented system knowledge remain central to defining reference behaviour and judging whether a migration is correct.
Global context
COBOL, Fortran and Visual Basic remain embedded in finance, government, science and industrial systems worldwide. Their risk comes from the surrounding organisation and application as much as the language, so evidence must travel beyond a single benchmark before supporting high-stakes modernisation.
What the evidence does not yet show
- OpenAI designed the evaluations, ran them and sells the evaluated models; there is no independent replication.
- The article omits the COBOL migration denominator, per-target-language scores, repeated-run variation and uncertainty intervals.
- The repository documents 24,000 converted SQLite tests and a recommended 272-test subset, but the article does not identify which produced the headline results.
- Neither evaluation measures production defects, security, operational incidents, maintenance burden or worker outcomes.
What to watch next
- Independent reproduction using the released MultiLangBench suite
- A documented COBOL corpus, denominator and per-target-language results
- Real migration studies reporting defects, review time, incidents and rollback outcomes
Living evidence record
Impact record IAI-0GA70MC
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
20 October 2026
Source trail
2 direct sources across 2 source types.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 2 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 20 October 2026
Research question, topics and regions
Whether new company-run coding evaluations establish that frontier models can safely build or migrate software in lower-resource programming languages
Topics: Software engineering · Legacy systems · COBOL · Model evaluation · Work & Skills
Regions: United States · Global enterprise technology
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.