How information quality impacts LLMs' ability to patch complex, novel vulnerabilities

We studied what happens when Large Language Models (LLMs) generate vulnerability patches for complex, novel vulnerabilities, and found that in general, those patches are more likely than not to be FLAWED — that is, “Fix-Like Artifacts with Embedded Defects.” Performance was highly variable between different codebases and vulnerabilities, but on average, given initial information of mixed correctness, only 26% of the patches produced by the models under test remediated the vulnerability without introducing security regressions or erroneous application changes.

Even when given only carefully-vetted, known-correct fix guidance and information about the target vulnerability, the proportion rose to 31%. With incorrect information about the root cause or fix direction, it fell to just 15%. We are sharing the tooling, datasets, and research paper so defenders can examine the methods and test patching outcomes in their own environments.

Recent initiatives such as Anthropic's Project Glasswing and OpenAI's Project Daybreak suggest an increasing industry focus on AI-driven vulnerability remediation. But how effective are LLMs at producing patches without erroneously altering an application's behavior? Do their patches actually mitigate the vulnerabilities in question? And how often do they introduce new vulnerabilities? We set out to test these questions using six recently disclosed vulnerabilities requiring complex fixes.

Across the full scope of our tests, including prompts of mixed accuracy, 26.0% of evaluated patches resolved the vulnerability without materially changing application behavior or introducing security regressions or new vulnerabilities. Another 20.1% resolved the vulnerability without security regressions, but changed unrelated behavior; for instance, they subtly altered how expressions are parsed, replaced allowlists with denylists, or rejected not only malicious inputs but valid ones as well. 53.9% of patches left the vulnerability unresolved, introduced a new vulnerability, or both. In these tests, neither model consistently produced complete, behavior-preserving fixes.

Targeted vulnerabilities

We tested six open source vulnerabilities whose complete fixes required changes across multiple files, functions, or code paths:

• CVE-2026-31431 — Linux privilege escalation (“Copy Fail”)

• CVE-2026-34197 — ActiveMQ remote code execution

• CVE-2026-8512 — Use-after-free in Chrome's File System Access API on macOS

• CVE-2026-45185 — Exim unauthenticated remote code execution

• CVE-2026-22738 — Spring AI SpEL remote code execution

• GHSA-wpqr-6v78-jr5g — Gemini CLI remote code execution

We selected these vulnerabilities because all were very recently disclosed, meaning that information about both the vulnerabilities themselves, as well as their fixes, were unlikely to have made their way into frontier models' as training data yet. Our hypothesis was that frontier models would exhibit a patch success rate above 67% for these targets, with “success” defined as full remediation of all known exploit paths with no erroneous changes to application behavior. The observed rate was substantially lower, with high variation by vulnerability and codebase.

Methodology and the cost of patching

With each model, we generated 540 patches per vulnerability in sets of 20, varying three environment configurations and nine prompt templates tailored to each vulnerability, for a total of 6,480 patch attempts. We flagged and excluded runs where a model appeared to “cheat” by directly seeking out an existing upstream patch; this eliminated 400 flagged attempts from the reported outcome rates.

A generated patch is a candidate, not proof that the vulnerability is fixed. In these tests, patches often left an exploit path open or changed application behavior. As such, we classified the patches into five scenarios:

• Scenario 1 (S1): Complete fix with no further changes to application behavior

• Scenario 2 (S2): Complete fix, but makes erroneous changes to application behavior

• Scenario 3 (S3): Does not fix the vulnerability

• Scenario 4 (S4): Complete fix of the old vulnerability while adding a new vulnerability

• Scenario 5 (S5): Does not fix the vulnerability while adding a new vulnerability

Figure 1: Average patch outcomes across 6,080 analyzed patches, including mixed-correctness prompts, after excluding 400 flagged attempts. However, clean-fix rates varied considerably across the six vulnerabilities.

The average cost of an attempted patch and validation cycle was $2.11 for OpenAI's ChatGPT-5.5 with Trusted Access for Cyber guardrails at the default medium effort setting, and $2.81 for Anthropic's Opus 4.8 with Cyber Verification Program guardrails at the default high effort setting.These figures cover attempted patch generation and validation. They do not measure the cost of confirming that a patch fully closes the vulnerability or preserves the application's intended behavior, which is itself a nontrivial task.

We are releasing the generated patches and the software used to generate, validate, compare, and manually review them. Oftentimes, there is only a subtle distinction between a robust fix and one that changes behavior, leaves an exploit path open, or adds a new vulnerability.

More than one-third of patches classified as S1 or S2 also contained security-relevant fragility: they attempted to cut off the path to the exploitation primitive, rather than removing the primitive itself. For instance, for the Spring AI vulnerability, both models often escaped selected characters in user input in order to prevent malicious inputs from reaching an unsafe expression evaluator; a correct fix would instead swap the evaluator for a safely-restricted one. Such a change may block the malicious input string presented to the model in the proof-of-concept, but leaves the underlying vulnerable primitive behind, potentially reachable by alternate inputs.

Recommendations for improving patch outcomes

These results create a practical problem for defenders facing a growing volume of vulnerability reports. We sought feedback from the frontier AI labs whose models we studied. 

Feedback and recommendations from Anthropic: patch generation has outpaced patch verification, and the fix is to make verification execution-grounded rather than inspection-based, while keeping domain experts as the final reviewers at current model capabilities. We've made this point publicly:

Progress on software security used to be limited by how quickly we could find new vulnerabilities. Now it's limited by how quickly we can verify, disclose, and patch." –Project Glasswing initial update, May 2026

We are releasing the tooling and datasets to invite scrutiny and help others improve AI-assisted remediation. We support efforts such as Project Daybreak and Project Glasswing to improve the security of open source software, and we welcome criticism of our methods and findings.

Our first phase modeled plausible developer constraints, including default tool settings, cost limits, and the potential for incorrect root cause information and/or fix guidance. Feedback after publication suggested that the particular constraints we chose may have limited what agentic patching tools could produce. A useful next question is whether stronger verification changes these outcomes and what that verification needs to look like.

If you have additional feedback to share on this or any future research, you can always reach us at [email protected]

Final thoughts

We've published the patches so others can inspect those cases and test which checks would have caught them. The released dataset contains all 6,480 generated patches, including the 400 flagged attempts excluded from the outcome rates.

Teams can run the FLAWED tooling against vulnerabilities they have already fixed, then compare generated patches with the known fix and expected behavior.

Special thanks to Casey Ellis, Jason Haddix, Mike Shema and the others who reviewed the research.

Download the full research paper, Frontier Models’ Vulnerability Patches are Often F.L.A.W.E.D.