Factory’s Legacy-Bench exposes an uncomfortable gap between producing a plausible repair and knowing that the repair works. In its April 1 research release, the company reported that the best of its tested model-agent combinations passed 42.5% of legacy-software tasks. In September, Factory returned to the practical consequence: teams need acceptance evidence tied to their own codebase before delegating engineering work. Legacy-Bench research; September evaluation guidance.
The result worth paying attention to
Legacy-Bench spans six language families, with COBOL and Java 7 making up most tasks. Factory reports that agents believed they had succeeded in 97% of failed attempts. The benchmark is a vendor-created experiment, not a measurement of all deployed coding systems. Nevertheless, the discrepancy points to a specific development problem: confidence and completion checks can become disconnected.
The public repository provides ten sample tasks and an evaluation harness. That makes part of the method inspectable, while leaving the full reported result dependent on Factory’s larger task collection. Factory also discloses that some mainframe behavior is approximated. Those boundaries matter when considering production systems with different interfaces, restrictions, and business rules.
Generation and verification are different assets
Consider an illustrative migration that converts a payment calculation into another language. A competent model could preserve the ordinary arithmetic while changing how the system rounds a boundary value. The program might compile, produce a convincing example, and pass a happy-path test. A buyer who treats those signals as interchangeable with financial equivalence has evaluated the wrong property.
This is why a company’s old test cases, reconciled outputs, and records of exceptional behavior can become strategic assets in an AI modernization program. They provide a way to distinguish a useful candidate from an accepted replacement. That conclusion does not require assuming the model is incapable of learning the rule. It requires recognizing that the organization must have a defensible way to check whether the rule survived.
What should change in model development
The training implication is more precise than “add more legacy code.” A developer can ask which failure the model is making: misunderstanding the instruction, losing a formatting constraint, ignoring contradictory evidence, or stopping before it has gathered enough information. These failures suggest different interventions. More examples may help one; a better tool or a stronger stopping rule may help another.
Evaluation should therefore preserve the route to the answer as well as the final patch. A failed run that inspected the correct specification and misunderstood it is diagnostically different from one that never opened the relevant file. Combining both into a zero is useful for an acceptance rate but insufficient for deciding where the next training dollar should go.
The commercial opportunity is in demonstrable completion
For model providers, an attractive enterprise proposition would pair generation with evidence that a reviewer can inspect efficiently. That could include a narrowly scoped change, a record of affected interfaces, and a comparison against retained examples. These are proposed evaluation outputs, not capabilities this publication has independently tested in Factory’s product.
The economic calculation should include the cost of discovering a subtle mistake after an apparently successful run. A lower inference bill does little good if every result needs extensive reconstruction by a scarce specialist. Conversely, a system that makes its mistakes easy to detect could be useful before it can solve every task. Detectability and correction effort deserve measurements alongside pass rate.
There is also a limit to how much can be inferred from a single benchmark. A bank with comprehensive regression coverage may be better positioned to adopt coding models than a smaller firm with undocumented exceptions, even if both use the same language. Readiness is partly a property of the organization’s evidence, not simply the age of its software.
The evidence to watch
Watch for independent runs on the public tasks, fuller disclosure of hidden-task coverage, and customer evaluations that report accepted changes alongside specialist review time. The decisive improvement would be fewer undetected errors at a fixed review budget, rather than another increase in the number of patches produced.
References
Questions, news tips or corrections? Contact info@frontieraireview.com.
