About
Evidence beforerankings.
A Snyk benchmark initiative publishing transparent, versioned evidence about automated vulnerability remediation.
Snyk VulnBump measures whether an AI agent can upgrade a vulnerable dependency and leave the project working: dependencies resolve, the vulnerability is gone on a Snyk rescan, the build passes, and the tests still pass.
The first release compares Snyk's Evo Remediation Agent with Claude Code, with and without a Snyk scan, on the same 307 upgrades across 43 real repositories, all on the same model, alongside a non-agentic baseline that only applies Snyk's recommended version. Each release defines its repositories, agents, and gates up front and publishes the data behind every chart.
It is a companion to Snyk VulnBench, which studies how reliably AI systems find vulnerabilities.
The benchmark is described in VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green, by Gabija Pasiunaite, Ryan McMorrow, Brian Clark and Liran Tal (Snyk) and Luke Tu (Brown University and Snyk).
Questions and corrections are welcome via the website repository.
Principles
- 01
Fixed targets
Every configuration attempts the same vulnerabilities in the same repository state.
- 02
Verified outcomes
A verifier that never trusts the agent's report checks the fix in a fresh container.
- 03
A joint verdict
An upgrade passes only if the vulnerability is fixed and the build and tests still pass.
- 04
Honest uncertainty
Per-repository rates and repository-clustered significance tests sit beside the pooled rates; repeated runs are next.
- 05
Reproducible data
Versioned results and definitions remain downloadable.