Trail of Bits ships agent plugins to police AI-written patches

Trail of Bits published two agent plugins, patching-bugs and regression-hunt, on 21 October 2026. The tools let engineers delegate patch development and regression checking to AI agents, and were built after the firm audited its own agent-assisted security fixes and found that some of them broke working code.

According to the company, the plugins came out of Patch the Planet, a joint initiative with OpenAI under which Trail of Bits says it has co-authored hundreds of security patches for widely used open-source projects. The firm said it reviewed every patch one of its engineers authored and a maintainer merged upstream. That review confirmed 21 regressions across more than 130 merged pull requests: 16 functional, one performance and four build or test failures. None broke a security protection, the company said.

Trail of Bits defines a regression as a new defect introduced by a change that breaks previously correct and supported behaviour along a path a real user or attacker can reach. The firm said the audit itself was noisy. An agent searched each patch for possible regressions and tried to confirm each one with a proof-of-concept run before and after the change, before handing the results to a human engineer. In a single run, the 21 confirmed regressions were found among 58 candidates, a precision rate of about 36 per cent. Because agents take different approaches each time, the company said, every run throws up a fresh pile of candidates for people to sort through.

That led to two conclusions, according to the post: the firm needed a standard way to write patches with the same checks applied every time, and a standard way to check for regressions with written rules for what counts as one. Standardising both steps also makes runs repeatable and comparable, the company said.

The patching-bugs plugin reproduces reported bugs, develops fixes and tests them. The regression-hunt plugin checks whether changes break previously working behaviour. Maintainers can supply their own security assumptions and patching requirements so the agents work within a project's rules, and engineers review the output.

Trail of Bits said four principles shaped the design. Guidance to the agent is deliberately brief and opinionated. Both plugins are modular, so individual skills can be run on their own rather than end to end. Every step takes and returns a fixed schema covering the claim, the code revision, the required check, the raw evidence and open questions, which the firm said reduces variability and lets each step verify the previous step's work instead of trusting its summary. And every conclusion must rest on a proof of concept run against a live, running instance of the project, reached through the interfaces a real user would use. If a step cannot produce a live proof of concept, its result is marked inconclusive.

Regression-hunt works in four stages, according to the company: it traces the changed code through its callers to real interfaces to enumerate candidates, reproduces each candidate on a live deployment, replays the proof of concept using a fresh-context agent, then adjudicates. The adjudication step places the burden of proof on the agent to show a finding is not a false positive, an incomplete fix, a latent bug or a pre-existing bug. Where a threat model document exists, the orchestrator passes it to every step.

Trail of Bits tested the plugin against its set of 21 confirmed regressions. In preliminary evaluations, it said, regression-hunt produced 25 per cent fewer false-positive findings than a baseline Codex agent simply prompted to review the pull request for regressions.