Does the study mean AI cannot write patches?

No. About one quarter of evaluated patches were clean fixes. The problem is that teams cannot safely distinguish good and dangerous results without strong tests and expert review.

Which vulnerabilities were tested?

The selection covered six new flaws in Linux, ActiveMQ, Chrome, Exim, Spring AI, and a Gemini CLI action.

Why were 400 patches excluded?

The researchers flagged attempts in which a model appeared to search for an existing fix. Those were removed from the main analysis to better measure independent patch generation.

Can a company use FLAWED itself?

Yes. The code, methodology, and datasets are public. Because it uses Docker-socket access and executes third-party code, it should run only in an isolated test environment.

FLAWED study: Only 26 percent of AI patches were clean

What this is about

The new FLAWED study by 1Password’s Off-by-1 Labs asks an uncomfortable question: if AI finds more security flaws, can it also fix them reliably? The result published on August 6, 2026, is clear. Of 6,080 evaluated patch attempts, only 26.0 percent fully resolved the vulnerability without materially changing application behavior.

Another 20.1 percent closed the known flaw but changed the software in the process. In 53.9 percent of cases, the vulnerability remained, a new one was introduced, or both happened. The researchers call these results “Fix-Like Artifacts with Embedded Defects”: changes that look like repairs but contain hidden flaws.

What FLAWED actually does

FLAWED is an open test system for AI-generated security patches. It resets an open-source project to a known vulnerable revision, gives a model a description of the flaw, and asks it to create a patch. The model does not see the project’s real upstream fix.

The changes are then validated, audited, and graded in isolated containers using five outcome classes. They range from a complete fix without behavior changes to a patch that leaves the old flaw in place and introduces a new vulnerability.

The test covered six vulnerabilities disclosed in 2026, including flaws in Linux, ActiveMQ, Chrome, Exim, Spring AI, and a Gemini CLI action. Two capable models produced 6,480 patches in total. Four hundred attempts were removed from the main analysis because they showed signs of searching for existing fixes, leaving 6,080 evaluated patches.

Why it matters

Automated patching sounds like the logical response to a growing number of automatically discovered vulnerabilities. The study reveals an asymmetry: a model can identify suspicious code without reliably understanding the architecture and every side effect of a safe repair.

More than one third of the patches graded as complete or behavior-changing fixes also contained fragile security controls, according to Off-by-1 Labs. They often blocked only the known attack string instead of correcting the root cause. A slightly different input could therefore make the flaw reachable again.

Price does not solve the quality problem. The reported average costs were $2.11 and $2.81 per attempt and validation cycle. That is inexpensive compared with human engineering time, but useful only if testing and expert review reliably filter out defective results.

In plain language

An AI patch is like putting tape over a leaking water pipe. The outside may look dry even though the crack continues underneath, or the pressure creates a new leak elsewhere. Only a stress test shows whether the pipe was actually repaired.

A practical example

A team receives 100 AI-generated patches for a critical library. Applying the study’s averages as a rough illustration, about 26 would be clean fixes, 20 would fix the flaw while changing program behavior, and 54 would fail or introduce new risk.

The team therefore cannot merge the first patch it receives. It needs reproducible exploit tests, existing functional tests, checks for new vulnerabilities, and expert code review. FLAWED can make this process measurable: a company can first test previously fixed vulnerabilities from its own codebase and identify components where AI patching fails most often.

Scope and limits

The study examined six complex open-source vulnerabilities. Its rate cannot be transferred unchanged to small bugs, proprietary systems, or different models.
The grades depend on tests, evaluation criteria, and prompts. A missing test can make a defective patch look successful.
This was not a leaderboard comparison between individual models. The results describe the tested workflow and the state of August 2026.
Human review is not infallible either. It must be supported by automated tests, explicit security requirements, and independent checks.
FLAWED mounts the Docker socket for some workflows and executes third-party code. It belongs in a dedicated isolated environment.

SEO & GEO keywords

FLAWED, Off-by-1 Labs, 1Password, AI-generated patches, vulnerability remediation, software security, LLM security, code review, CVE, DevSecOps, patch verification, open source

FLAWED: Only 26 percent of AI patches passed the test