A benchmark claim with enormous implied stakes
OpenAI's internal evaluation of its GPT-6.1 Sol model reports 86.6 percent accuracy migrating COBOL programs to modern languages at low reasoning effort, climbing to 97.9 percent at a higher reasoning effort setting. The reported baseline model scored just 5.4 percent on the same test, an enormous gap that, if it holds under independent scrutiny, would represent one of the more consequential practical breakthroughs in applied AI coding this year.
The stakes behind this specific benchmark are unusually high because COBOL modernization is not a niche technical curiosity, it is a persistent, expensive problem across banking, insurance, and government systems that run core operations on decades old code. A result this strong, if real, has direct implications for how enterprises budget and staff some of their most entrenched legacy modernization programs.
Why nobody should treat this as settled yet
No organization outside OpenAI has independently reproduced these results, and the report's naming of the comparison baseline model is unclear from what has been publicly disclosed. OpenAI also reportedly ran the low effort test for 39 percent less cost than the prior model's equivalent run, under conditions that have not been fully specified, adding another variable that independent evaluators would need to control for before treating the comparison as apples to apples.
This pattern, an internal vendor benchmark reported through secondary coverage rather than a peer reviewed or independently replicated study, has produced overstated claims before across the AI industry. Enterprise technology leaders have good reason to treat vendor reported coding benchmarks as a starting hypothesis to test against their own codebase, not as a result to plan a modernization budget around directly.
Why COBOL modernization has resisted easy fixes for decades
COBOL systems have proven resistant to modernization for reasons that go well beyond translating syntax from one language to another. The engineers who understand how these systems actually behave, including undocumented edge cases and business rules embedded implicitly in decades of accumulated code changes, are a shrinking population, and much of the original system documentation is incomplete or has not survived the system's operational lifetime.
That context matters for evaluating any AI translation benchmark in this space. A high accuracy score on translating code syntax says relatively little about whether the translated system preserves the subtle business logic and edge case handling that the original COBOL program encodes, often implicitly, after years of patches and workarounds that were never formally documented anywhere outside the code itself.
What a responsible pilot actually looks like
For any enterprise with a meaningful COBOL footprint, the sensible response to this claim is a small, deliberately scoped pilot on genuinely low risk modules, modules where a translation error would be inconvenient rather than catastrophic, paired with expert review and rigorous testing against the system's actual business rules. That is a meaningfully different commitment than greenlighting a large scale modernization program on the strength of one vendor's unverified internal test.
The pilot should specifically measure not just whether the translated code compiles and passes basic tests, but whether it correctly reproduces edge case behavior the original system handles, often in ways that are not obvious from reading the code alone. This is where legacy COBOL expertise remains essential even if AI assisted translation genuinely accelerates the mechanical part of the work, since validating behavioral equivalence requires understanding what the original system was actually supposed to do in every corner case.
The economics that make this worth testing regardless
Even with appropriate skepticism about the specific benchmark numbers, the underlying economic case for testing AI assisted COBOL migration is strong. Modernization projects have historically stalled or run over budget specifically because of the scarcity and cost of engineers with deep legacy system expertise, and any tool that genuinely reduces the mechanical translation burden, even if it does not fully replace expert review, changes the economics of projects that have sat stalled for years due to cost and risk concerns.
That argues for running the pilot regardless of how much confidence the specific 97.9 percent figure deserves. The downside of a small, well scoped pilot is limited, and the potential upside, meaningfully cheaper mainframe modernization at a moment when legacy system expertise keeps getting scarcer, justifies the investment of testing this claim directly against your own organization's actual codebase rather than dismissing it or accepting it uncritically.
What CIOs with legacy mainframe footprints should do now
Technology leaders overseeing COBOL dependent systems should treat this announcement as a prompt to scope a controlled internal evaluation, not as validated proof that large scale AI driven modernization is now safe to commit to broadly. Select a handful of genuinely representative modules, including at least one with known complex edge case behavior, and measure both translation accuracy and the time your own expert reviewers need to validate the output against the alternative of writing the migration by hand.
That internal data point, however it comes out, will be more useful for your organization's actual planning than any vendor benchmark, independently verified or not, because it reflects your specific codebase's complexity and your own team's review capacity. Given how much budget and risk tolerance ride on getting mainframe modernization timelines right, that direct evidence is worth gathering before committing to any particular vendor's tooling at scale.
