Snyk Labs · SCA remediation benchmark
Fixed isn't
shipped.
Can an AI agent upgrade a vulnerable dependency without breaking the project? 307 real upgrades, four gates, one verdict.
90.2%
of the Evo Remediation Agent's upgrades passed all four gates
Upgrades passing each gate
307 upgrades
→ 4 gates
→ 4 gates
StartAttempted
Evo Remediation AgentSnyk agentic remediation
90.2% PASS
Attempted: 307 of 307
Dependencies resolve: 307 of 307
Vulnerabilities fixed: 298 of 307, 9 dropped
Build: 289 of 307, 9 dropped
Tests pass: 277 of 307, 12 dropped
Claude Code + Snyk scanGiven Snyk target + CLI
77.5% PASS
Attempted: 307 of 307
Dependencies resolve: 291 of 307, 16 dropped
Vulnerabilities fixed: 243 of 307, 48 dropped
Build: 240 of 307, 3 dropped
Tests pass: 238 of 307, 2 dropped
Claude Code unaidedNo target, Snyk blocked
59.6% PASS
Attempted: 307 of 307
Dependencies resolve: 293 of 307, 14 dropped
Vulnerabilities fixed: 186 of 307, 107 dropped
Build: 183 of 307, 3 dropped
Tests pass: 183 of 307
Package manager bumpSnyk's version, no agent
46.9% PASS
Attempted: 307 of 307
Dependencies resolve: 253 of 307, 54 dropped
Vulnerabilities fixed: 175 of 307, 78 dropped
Build: 159 of 307, 16 dropped
Tests pass: 144 of 307, 15 dropped
Selected gate
Gate 2 · Vulnerabilities fixed
A post-fix Snyk rescan covers every target manifest and no longer finds the upgrade's targeted vulnerabilities.
Evo Remediation Agent
−9
vulnerability not fixed · 298 continue
Claude Code + Snyk scan
−48
vulnerability not fixed · 243 continue
Claude Code unaided
−107
vulnerability not fixed · 186 continue
Package manager bump
−78
vulnerability not fixed · 175 continue
View exact values
| Harness | Attempted | Dependencies resolve | Vulnerabilities fixed | Build | Tests pass |
|---|---|---|---|---|---|
| Evo Remediation Agent | 307 | 307 (100.0%) −0 | 298 (97.1%) −9 | 289 (94.1%) −9 | 277 (90.2%) −12 |
| Claude Code + Snyk scan | 307 | 291 (94.8%) −16 | 243 (79.2%) −48 | 240 (78.2%) −3 | 238 (77.5%) −2 |
| Claude Code unaided | 307 | 293 (95.4%) −14 | 186 (60.6%) −107 | 183 (59.6%) −3 | 183 (59.6%) −0 |
| Package manager bump | 307 | 253 (82.4%) −54 | 175 (57.0%) −78 | 159 (51.8%) −16 | 144 (46.9%) −15 |
- 90.2%
- of upgrades passed all four gates with the Evo Remediation Agent (277 of 307), against 77.5% for Claude Code + Snyk scan and 59.6% for Claude Code unaided. →
- 82.2%
- of the 163 upgrades that Snyk's recommended version alone does not solve were passed by the Evo Remediation Agent, against 61.3% and 36.2% for Claude Code. →
- 46.9%
- of upgrades pass when only Snyk's recommended version is applied. This non-agent baseline trails every agent on every gate. →
Leaderboard
Same model, three agents
#
Agent
01
Evo Remediation AgentSnyk agentic remediation
90.2%
95.9%
100.0%
96.7%
95.4%
02
Claude Code + Snyk scanGiven Snyk target + CLI
77.5%
83.1%
94.8%
98.0%
99.0%
03
Claude Code unaidedNo target, Snyk blocked
59.6%
72.7%
95.4%
98.4%
98.7%
Not ranked
Package manager bumpSnyk's version, no agent · non-agent baseline
46.9%
56.5%
82.4%
87.0%
80.5%
307 upgrades per agent. Click a column to sort. The package manager bump is a reference, not ranked.
The Evo Remediation Agent passes 46 upgrades Claude Code + Snyk scan fails, and fails 7 it passes.
All findings →