Snyk · SCA remediation benchmark
Fixed isn't
shipped.
Can an AI agent upgrade a vulnerable dependency without breaking the project? 307 real upgrades, four gates, one verdict.
90.2%
of the Evo Remediation Agent's upgrades passed all four gates
Leaderboard
Same model, three agents
#
Agent
01
Evo Remediation AgentSnyk agentic remediation
90.2%
97.1%
100.0%
96.7%
95.4%
02
Claude Code + Snyk scanGiven Snyk target + CLI
77.5%
83.7%
94.8%
98.0%
99.0%
03
Claude Code unaidedNo target, Snyk blocked
59.6%
63.5%
95.4%
98.4%
98.7%
Not ranked
Package manager bumpSnyk's version, no agent · non-agentic baseline
46.9%
62.2%
82.4%
87.0%
80.5%
Targeted vulns fixed: share of upgrades whose rescan shows every targeted vulnerability fixed (the rescan gate), not the share of vulnerability instances. 307 upgrades per agent. Click a column to sort. The package manager bump is a reference, not ranked.
Key results
Four gates, one verdict
| Metric | Package manager bumpnon-agentic baseline | Claude Code unaided | Claude Code + Snyk scan | Evo Remediation Agent |
|---|---|---|---|---|
| Dependencies resolve | 82.4% | 95.4% | 94.8% | 100% |
| Targeted vulnerabilities fixed | 62.2% | 63.5% | 83.7% | 97.1% |
| Build | 87.0% | 98.4% | 98.0% | 96.7% |
| Tests pass | 80.5% | 98.7% | 99.0% | 95.4% |
| All gates pass (PASS) | 46.9%(144/307) | 59.6%(183/307) | 77.5%(238/307) | 90.2%(277/307) |
| Vulnerability instances fixed | 56.5%(1,168/2,066) | 72.7%(1,503/2,066) | 83.1%(1,716/2,066) | 95.9%(1,982/2,066) |
Upgrades passing each gate
307 upgrades
→ 4 gates
→ 4 gates
StartAttempted
Evo Remediation AgentSnyk agentic remediation
90.2% PASS
Attempted: 307 of 307
Dependencies resolve: 307 of 307
Targeted vulns fixed: 298 of 307, 9 dropped
Build: 289 of 307, 9 dropped
Tests pass: 277 of 307, 12 dropped
Claude Code + Snyk scanGiven Snyk target + CLI
77.5% PASS
Attempted: 307 of 307
Dependencies resolve: 291 of 307, 16 dropped
Targeted vulns fixed: 243 of 307, 48 dropped
Build: 240 of 307, 3 dropped
Tests pass: 238 of 307, 2 dropped
Claude Code unaidedNo target, Snyk blocked
59.6% PASS
Attempted: 307 of 307
Dependencies resolve: 293 of 307, 14 dropped
Targeted vulns fixed: 186 of 307, 107 dropped
Build: 183 of 307, 3 dropped
Tests pass: 183 of 307
Package manager bumpSnyk's version, no agent
46.9% PASS
Attempted: 307 of 307
Dependencies resolve: 253 of 307, 54 dropped
Targeted vulns fixed: 175 of 307, 78 dropped
Build: 159 of 307, 16 dropped
Tests pass: 144 of 307, 15 dropped
Selected gate
Gate 2 · Targeted vulns fixed
A post-fix Snyk rescan of every target manifest no longer finds the targeted vulnerabilities through the upgraded package or any newly introduced dependency path.
Evo Remediation Agent
−9
vulnerability not fixed · 298 continue
Claude Code + Snyk scan
−48
vulnerability not fixed · 243 continue
Claude Code unaided
−107
vulnerability not fixed · 186 continue
Package manager bump
−78
vulnerability not fixed · 175 continue
View exact values
| Harness | Attempted | Dependencies resolve | Targeted vulns fixed | Build | Tests pass |
|---|---|---|---|---|---|
| Evo Remediation Agent | 307 | 307 (100.0%) −0 | 298 (97.1%) −9 | 289 (94.1%) −9 | 277 (90.2%) −12 |
| Claude Code + Snyk scan | 307 | 291 (94.8%) −16 | 243 (79.2%) −48 | 240 (78.2%) −3 | 238 (77.5%) −2 |
| Claude Code unaided | 307 | 293 (95.4%) −14 | 186 (60.6%) −107 | 183 (59.6%) −3 | 183 (59.6%) −0 |
| Package manager bump | 307 | 253 (82.4%) −54 | 175 (57.0%) −78 | 159 (51.8%) −16 | 144 (46.9%) −15 |
- 90.2%
- of upgrades passed all four gates with the Evo Remediation Agent (277 of 307), against 77.5% for Claude Code + Snyk scan and 59.6% for Claude Code unaided. →
- +17.9 pts
- more upgrades passed when Claude Code gets Snyk's scan and recommended upgrade: 77.5%, up from 59.6% on its own. Following Snyk's recommended version without an agent passes 46.9%. →
- 82.2%
- of the 163 upgrades that Snyk's recommended version alone does not solve were passed by the Evo Remediation Agent, against 61.3% and 36.2% for Claude Code. Same model, same findings: the harness makes the difference. →