Bug Complexity
128 vulnerabilities scored by difficulty. Floor = every agent fixes it. Ceiling = no agent can.
Five difficulty bands
From floor (all agents pass) to ceiling (no agent passes). The wider the medium band, the better the benchmark discriminates between agents.
What the ceiling means
Ceiling samples are beyond current AI capability. Human review still wins for these. AI patching works best combined with human escalation for the hard tail.
How we score vulnerability complexity
128 CVE samples ranked by how many of the 15 agents fix them. Easy samples: most agents pass. Floor samples: no agent passes. The spread between easy and floor tells you how much headroom AI patching has left.
Difficulty is determined by how many agents succeed on a given bug. If most agents fix a bug, it is easy. If zero agents fix it, it is a floor sample — beyond current AI capability. Ceiling samples sit just above the floor: near-zero pass rate, with rare successes. This is objective and data-driven, not a subjective guess.
[KEY INSIGHT]
25 bugs no agent can fix
25 of 128 samples are floor samples — 0% pass rate across all 15 agents. Another 21 are ceiling samples with near-zero pass rates (0-20%). Oracle ceiling: 80.5% — even a perfect ensemble can only fix 80.5% of bugs.
The ceiling is important because it defines the realistic maximum. You cannot reach 100% with any agent or ensemble - some bugs require human expertise. Knowing the ceiling prevents teams from over-investing in agent optimization when diminishing returns have already set in.
Difficulty Distribution
Sample counts across 5 empirical difficulty tiers: easy, medium, hard, floor, ceiling. The distribution shows where agent performance varies. If most bugs cluster at the floor or ceiling, the benchmark does not discriminate - all agents are equally good or bad. But if bugs spread across the middle bands, that is where agent selection matters.
Your codebase will have its own distribution, which may differ from this sample. If you maintain legacy C code with buffer overflows, your bugs might cluster in the medium band. If you run modern Rust with dependency updates, your bugs might be mostly floor samples. Understanding your own difficulty distribution drives ROI modeling.
Easy
58
Most agents pass
Medium
17
Mixed results
Hard
15
Most agents fail
Very hard
21
Nearly impossible
Impossible
25
No agent passes
Hardest and easiest samples
The extremes. Easy samples are reliably fixed by most agents. Floor samples remain unsolved — no agent has ever fixed them.
Hardest bugs
| Project | Pass rate | Agents passed |
|---|---|---|
| stat-reader (stat-reader) | 0% | 0/15 |
| disassembly-engine (disassembly-engine) | 0% | 0/15 |
| disassembly-engine (disassembly-engine) | 0% | 0/15 |
| disassembly-engine (disassembly-engine) | 0% | 0/15 |
| disassembly-engine (disassembly-engine) | 0% | 0/15 |
| disassembly-engine (disassembly-engine) | 0% | 0/15 |
| js-engine (js-engine) | 0% | 0/15 |
| js-engine (js-engine) | 0% | 0/15 |
| data-compressor (data-compressor) | 0% | 0/15 |
| service-proxy (service-proxy) | 0% | 0/15 |
Easiest bugs
| Project | Pass rate | Agents passed |
|---|---|---|
| text-shaping (text-shaping) | 100% | 15/15 |
| text-shaping (text-shaping) | 100% | 14/15 |
| git-library (git-library) | 100% | 15/15 |
| network-switch (network-switch) | 100% | 15/15 |
| packet-analyzer (packet-analyzer) | 100% | 15/15 |
| image-processor (image-processor) | 93% | 14/15 |
| text-shaping (text-shaping) | 93% | 14/15 |
| text-shaping (text-shaping) | 93% | 14/15 |
| text-shaping (text-shaping) | 93% | 14/15 |
| text-shaping (text-shaping) | 93% | 14/15 |
Unlock full results
Enter your email to access the full methodology, per-sample analysis, and patch examples.
[NEXT STEPS]
See which agents handle the hard bugs
The behavior page shows how agents cluster by approach. The results page shows per-agent pass rates so you can match agent to difficulty.
Explore more
- Execution metrics
- how many turns agents take by difficulty
- Methodology
- how difficulty scores are computed
FAQ
How is bug difficulty measured?
Each of the 128 bugs is scored by how many of the 15 agents fix it. If all agents pass, it is a floor sample. If none pass, it is a ceiling sample.
What does complexity mean for my team?
If your codebase has mostly simple dependency bumps, expect higher fix rates than the benchmark average. Complex multi-file memory-safety patches will be closer to the hard band.
Benchmark Results
62.7% pass rate. $2.64 per fix. Real data from 1,920 evaluations.
Benchmark Results
62.7% pass rate. $2.64 per fix. Real data from 1,920 evaluations.
Agent Cost Economics
Fix vulnerabilities for $2.64–$52 with agents. 100x cheaper than incident response. Real cost data.
Agent Configurations
15 agent-model configurations benchmarked on real vulnerabilities. Compare pass rates and costs.
Benchmark Methodology
How XOR benchmarks AI coding agents on real security vulnerabilities. Reproducible, deterministic, and transparent.
Validation Process
25 questions we ran against our own data before publishing. Challenges assumptions, explores implications, extends findings.
See which agents produce fixes that work
128 CVEs. 15 agents. 1,920 evaluations. Agents learn from every run.