AI Generated 6,080 Security Patches. Fewer Than One in Four Actually Fixed the Flaw.

Six thousand AI patches, and half of them left the door open

Two of the most capable AI models available were asked to write security patches for six real, recently disclosed vulnerabilities. Between them they produced 6,080. Just 26 percent fixed the flaw cleanly. Another 20 percent fixed it but changed how the software behaved. The remaining 53.9 percent failed to close the vulnerability, introduced a new one, or managed both at once.

The uncomfortable part is not the failure rate. It is what failure looked like. These patches compiled. They passed their tests. They blocked the exact exploit shown in the vulnerability report. And the underlying flaw was still sitting there, reachable through a slightly different input.

What the research actually did

The study was published on 6 August 2026 by Off-by-1 Labs, the security research team 1Password launched this year, under the title Frontier Models' Vulnerability Patches are Often F.L.A.W.E.D., short for Fix-Like Artifacts With Embedded Defects. Researchers ran OpenAI's ChatGPT-5.5 and Anthropic's Claude Opus 4.8, each with its vendor's cyber guardrails enabled, against six CVEs (a CVE being the public identifier assigned to one specific disclosed vulnerability). They generated 540 patches per model per vulnerability, across three environment configurations and nine prompt templates. The targets were chosen for recency, so the real fixes would not already be sitting in the models' training data: a Linux privilege escalation, remote code execution in Apache ActiveMQ, in the Exim mail server, in Spring AI and in Gemini CLI, plus a use-after-free bug in Chrome's File System Access API on macOS. Going in, the team expected a success rate above 67 percent.

The mechanism behind the misses has a name the researchers keep returning to: tunnel vision. A vulnerability report normally ships with a proof of concept, a short piece of code that demonstrates the flaw being exploited. The model reads it and optimizes for the thing it can see. On the Spring AI vulnerability, both models frequently produced patches that simply escaped the specific characters appearing in that proof-of-concept string. Doing so defeats the demonstration. It does nothing to the vulnerable code underneath, so if an attacker reaches the same code path through a different input, the bug is back, and the ticket that closed it still reads fixed.

• Roughly one in five patches resolved the vulnerability but altered application behavior, including cases where a model reimplemented a file parser or quietly flipped allow-list logic into deny-list logic.
• More than a third of the patches counted as successful were classified as fragile, meaning they guarded the vulnerable code with narrow input checks rather than removing it. With those filtered out, lead researcher Keith Hoodlet put the share of clean patches that addressed the actual root cause at about 17 percent.
• Prompt quality mattered enormously, and in one direction it backfired. Accurate remediation guidance lifted the fix rate to 65 percent. Wrong guidance dropped it to 15.2 percent, far worse than giving the model nothing at all, which scored 50.4 percent. Models followed bad instructions even when evidence generated during their own testing contradicted those instructions.

Why this lands on your patch pipeline

None of this is theoretical. OpenAI's Project Daybreak, announced in June 2026 with the consultancy Trail of Bits under the banner Patch the Planet, is already generating fixes for open-source software at scale, and Hoodlet told Dark Reading that his team confirmed and reported failing patches from one of its weekend patch sets. At roughly two to three dollars per generated patch, the economics push hard toward automation, and that is precisely where the risk sits. If your team is using an agent to burn down a vulnerability backlog, the danger is not that it produces obvious rubbish your engineers reject. It is that it produces plausible fixes your engineers approve, your scanner stops flagging, and your metrics record as remediated, while the exploitable code stays exactly where it was. Cursor's own telemetry shows 36 percent of AI code changes now going in with no manual review at all. Underneath sits an asymmetry that should shape how you plan: these models are considerably better at finding and exploiting flaws than at repairing them. Veracode arrived somewhere similar from a different direction this year, measuring a 56 percent security pass rate for AI-generated code across more than 100 models. Anthropic, reviewing the 1Password work, put the shift plainly. Progress used to be limited by how fast anyone could find new vulnerabilities. Now it is limited by how fast they can be verified, disclosed and patched.

What to do about it

1. Verify against the vulnerability, not against the proof of concept. Re-run the original exploit and then variants that reach the same code by another route, because a patch that only blocks the demonstrated input is the exact failure mode this study found most often.
2. Stop feeding the model remediation guidance you are not sure about. Incorrect hints cut the fix rate to well under a third of what the same model managed with no hints at all, so an uncertain steer from a busy engineer is worse than silence.
3. Benchmark AI patching on your own code before automating any of it. 1Password released its FLAWED tooling and full dataset publicly, so run it against bugs you have already fixed by hand, because success rates varied sharply between the six targets tested and your codebase will have its own profile.

Bottom line

Nobody was breached here. A research team spent real money generating six thousand patches and then did the unglamorous work of checking whether each one actually held, which is precisely the step the industry is currently automating away. Bring the number itself to your next security meeting. Fewer than one in four AI-generated patches closed the vulnerability without side effects, and a third of the ones that looked right were holding the door shut with a narrow check rather than removing the door. A closed ticket is not the same thing as a fixed flaw. The full research paper sets out the per-vulnerability breakdown.