Trail of Bits disputes 1Password's 26% AI patching figure

Security consultancy Trail of Bits published a rebuttal on 15 September 2026 to 1Password's FLAWED report, a benchmark of AI agents fixing software vulnerabilities released on 6 August 2026. The report's headline claim was that models produced clean fixes only 26% of the time. Trail of Bits says that number is a misleading guide to ordinary patching work and could discourage defenders from using tools that would help them fix more bugs.

According to Trail of Bits, which reviewed 1Password's published code and data, four design choices inflate the apparent failure rate. The sample was six vulnerabilities picked because their fixes were complex, and clean-fix rates across them ranged from 3% to 60%, so the average depends heavily on which bugs were chosen. Two of the prompts explicitly instructed agents to apply the wrong fix, and those account for 22% of the data. One evaluation mode stopped agents from building or running code at all, accounting for a further 36%. The two models tested, GPT-5.5 and Opus 4.8, ran at different reasoning settings — medium and high respectively — which were the tools' defaults, and neither was tested at its highest setting.

Trail of Bits re-analysed the patches and test results that 1Password published. Keeping only trials where agents could run code and were not told to apply the wrong fix, and excluding runs the study classified as having consulted the upstream fix, it found that 2,634 of 3,067 patches, or 86%, blocked the supplied exploit. The firm is careful to note that blocking an exploit does not prove a complete repair, but says the result shows useful capability that the headline does not convey.

The consultancy also criticises the grading. Agents given a proof-of-concept exploit were told to stop once their patch defeated it, but the grader then examined vulnerable paths the exploit never exercised. Agents were told not to change existing tests, even though a correct fix can require test updates; Trail of Bits found 8% of ActiveMQ verdicts penalised an intended behaviour change as a regression. Models grading their own patches matched human reviewers on the full five-category outcome in 65.9% of reviewed cases, and the two grading models disagreed with each other on 36.8% of the same patches. The report's own data, Trail of Bits notes, shows 248 generated patches repeated an off-by-one error present in the upstream Linux fix, and the automated grader caught it in only 24 of them. The Chromium grader marked patches clean that left a use-after-free bug in a callback.

Against that, Trail of Bits offered its own figures for human performance. It reviewed the first fixes submitted for 2,265 vulnerabilities across 236 of its security assessments between 2024 and 2026. Developers in those cases maintained the software, had detailed reports and knew their patches would be reviewed. Even so, 283 first fixes — 12.5%, or one in eight — failed to fully resolve the reported issue, with a 95% confidence interval of 10.5% to 14.5%. The firm says its data may undercount failures, because some mistakes are corrected informally before a formal fix review.

Alongside the analysis, Trail of Bits released two agent skills: post-patch-validation, to help agents test their fixes, and review-walkthrough, to help engineers review them.