Four gates. No partial credit.

The full method

Research question

Can an AI agent upgrade a vulnerable dependency without breaking the project?

Software composition analysis tools report far more vulnerable dependencies than teams fix. In Snyk Open Source production data from April–September 2026, 94.0% of findings have a known fixed version and 93.7% an upgrade path to it within the project, yet 89% are still open, about 10% resolved and 1% ignored. The bottleneck is upgrading safely: an upgrade that removes the vulnerability but breaks the build or tests is not a fix.

VulnBump is 307 upgrade tasks covering 2,066 fixable vulnerabilities in 43 repositories across six package managers, matched to Snyk Open Source production vulnerability distributions. An upgrade passes only if, in a fresh container, dependencies resolve, a rescan shows the targeted vulnerabilities fixed, the project builds, and every previously passing test still passes. All three agents use the same model, Claude Sonnet 5.5, and a non-agentic baseline that only applies Snyk's recommended version shows how much of the work that advice already does.

How VulnBump compares

A remediation benchmark needs real repositories with working builds and test suites, tasks driven by real vulnerability findings with known fix paths, and verification, independent of the agent, that the vulnerability is gone and the project still works. Existing benchmarks cover at most two: breaking-update benchmarks are not driven by vulnerabilities, so they cannot observe an agent that leaves the vulnerability in place, and CORAL scores against its own CVE-to-version mapping rather than a production scan.

The dataset

Grounded in production data

The target distributions come from Snyk Open Source findings in monitored projects from April–September 2026: about 300,000 organisations and 459 million unique issue–project pairs. A Snyk project is one monitored manifest, and a repository can contain several; because each project must be fixed separately, a vulnerability counts once for every project it affects. Java (62.7%), JavaScript (22.1%) and Python (9.9%) account for almost 95% of findings, and Maven, Gradle, npm, Yarn, pip and Poetry cover about 92% of all pairs, so VulnBump targets these six.

Candidates

Candidates come from 5 sources, which together cover all six package managers:

• Popular open-source projects (274). Curated applications, frameworks, starters and tutorials for Gradle, npm, Yarn, pip and Poetry.
• GitHub searches for vulnerable packages (280). Manifests that pin known-vulnerable versions of frequently vulnerable packages, plus searches for the less-represented package managers.
• BUMP projects (133). Maven projects from the BUMP benchmark, pinned at the parent of a breaking-update commit, where the project still builds and passes its tests.
• Gradle submodules (46). Modules of large multi-project builds, scanned separately so that one task does not span hundreds of modules.
• Popular Maven projects (34). Top-starred on GitHub, pinned at their last commit before 21 May 2022, CORAL's snapshot date.

Each candidate is scanned with snyk test --json, recording severity, fixability, direct or transitive status and fix difficulty per unique issue–project pair. Some repositories were collected at several commits, giving 805 candidates from 721 repositories; keeping the most vulnerable commit of each leaves 767: one per repository plus the 46 Gradle submodules.

Filtering for reproducible, well-tested projects

So that any later breakage can be attributed to the upgrade, the unmodified project must install, build and pass its tests from a clean checkout in its ecosystem's image (JDK 17, Node 20 or Python 3.11), with a 30-minute limit per step. Each candidate had to pass five filters in order; the first requires every declared dependency to download and install with the project's own package manager, without private registries. The largest drop is at the third: many projects have no tests that run in a clean environment, and most pip projects need external services such as Postgres or Redis. Earlier failures were mostly ecosystem-specific: multi-module resolution, private registries and plugins (Maven, Gradle), peer-dependency conflicts (npm), and lockfile or Python-version mismatches (Poetry). This leaves 99 qualifying candidates from 97 repositories (55 Maven, 9 Gradle, 19 npm, 8 Yarn, 3 pip, 5 Poetry).

Selection and final composition

Selected to match Snyk production data in direct versus transitive share, vulnerabilities per repository, severity and fix difficulty. Package managers are weighted so that each has enough cases to report on, rather than strictly in proportion to production. The dataset comprises 18 Maven, 3 Gradle, 11 npm, 3 Yarn, 3 pip, 5 Poetry repositories (21 Java, 14 JavaScript, 8 Python), with a median of 32 unique vulnerabilities (range 1–252) and 88 passing tests; 19 have at least 1,000 GitHub stars.

A few very large repositories, such as Apache Kafka and Spring Framework, were excluded because in earlier runs they ran out of memory or time, or had hundreds of upgrades that would take days to evaluate one by one. VulnBump therefore under-represents very large repositories and, since several are Gradle builds, Gradle.

Against production

The dataset matches production within a few points on severity, fix difficulty, and package-manager share except for Gradle, and on direct share for Maven, npm, Yarn and Poetry. Vulnerabilities per project span a similar range for Java and JavaScript, while the Python repositories cluster around a higher median. There are two deviations:

• Gradle is under-represented (3% of vulnerabilities against 17%), with a lower direct share (11% against 19%). Most Gradle candidates failed the filters, and the largest that qualified were too large to evaluate.
• pip vulnerabilities are mostly direct (84% against 56%), because only three pip repositories qualified.

The task

Each task asks an agent to apply one package upgrade (or pin) recommended by a baseline Snyk scan (snyk test --all-projects --ignore-policy) of the installed, pinned repository, which must first install, scan, build and pass at least one test at evaluation time. Snyk groups the vulnerabilities it can fix into fix groups: one upgrade or pin in one manifest, with the vulnerabilities it fixes. Vulnerabilities without a fix path are never counted against an agent.

Upgrade

A VulnBump task: one fix group of a pinned repository. A repository with 50 fixable vulnerabilities that require upgrading 20 packages yields 20 upgrades.

Fixable vulnerability instance

A (manifest, vulnerability id) pair in the baseline scan for which the scan offers an upgrade or pin. An instance reachable through several upgrades counts as fixed only if every covering upgrade fixed its own path.

An abridged baseline scan for one VulnBump repository (expressjs/generator) shows seven vulnerabilities across four upgrades. The minimist vulnerability is reachable through both the minimist and mkdirp upgrades, so it counts as fixed only if both fix their paths:

Tested 9 dependencies for known issues, found 7 issues, 8 vulnerable paths.

Issues to fix by upgrading:

  Upgrade ejs@2.6.2 to ejs@3.1.10 to fix
  ✗ Improper Control of Dynamically-Managed Code Resources [Medium][SNYK-JS-EJS-6689533]
  ✗ Remote Code Execution (RCE) [High][SNYK-JS-EJS-2803307]

  Upgrade minimatch@3.0.4 to minimatch@3.1.3 to fix
  ✗ Regular Expression Denial of Service (ReDoS) [Medium][SNYK-JS-MINIMATCH-3050818]
  ✗ Regular Expression Denial of Service (ReDoS) [High][SNYK-JS-MINIMATCH-15309438]
  ✗ Inefficient Algorithmic Complexity [High][SNYK-JS-MINIMATCH-15353389]

  Upgrade minimist@1.2.5 to minimist@1.2.6 to fix
  ✗ Prototype Pollution [Low][SNYK-JS-MINIMIST-2429795]

  Upgrade mkdirp@0.5.1 to mkdirp@0.5.5 to fix
  ✗ Prototype Pollution [Low][SNYK-JS-MINIMIST-2429795]
  ✗ Prototype Pollution [Medium][SNYK-JS-MINIMIST-559764]
    introduced by mkdirp@0.5.1 > minimist@0.0.8

Each upgrade is attempted from a fresh copy of the pinned repository and scored on its own (single-upgrade mode), so one upgrade's failure cannot affect another.

Harnesses

The package manager bump

The non-agentic baseline applies Snyk's recommended upgrade and nothing else: it uses no model, edits no code or tests, and never tries another version. It sets the named package's version with the ecosystem's own mechanism (npm install pkg@ver, yarn add, poetry add, or the version in requirements.txt, pom.xml or the Gradle build), and pins an undeclared package the same way, for example with an explicit Maven <version> or a Gradle constraints entry. A pin is managed when it overrides a version set by a Maven parent or BOM or by Spring's dependency-management plugin. If the lockfile refresh fails, so does the dependency gate; the only fallback is retrying npm with --legacy-peer-deps. Python packages are pinned exactly (==); pinning with >= changes the outcome of 2 of the 60 Python upgrades and leaves the total unchanged. It ran on 8 October 2026, three days after the agents' scans, on the same pinned repositories.

The agents

All three agents use the same model (claude-sonnet-5-5 on Vertex AI) and the same per-ecosystem container image. They differ in harness, prompt, tools and Snyk information, so VulnBump compares complete systems, as a user would get them, rather than ablating one component. All three agents use the same model and 110 agent turns per upgrade.

The Evo Remediation Agent is Snyk's agentic remediation engine, available through the Snyk CLI and a cloud service. It is invoked once per upgrade as snyk fix --agentic --sca --auto-approve --prepare-env --issue-ids <ids>, with the ids of the vulnerabilities that upgrade fixes. With --auto-approve, it applies every fix without prompting, including upgrades Snyk rates as likely to break the build, which a user could otherwise skip. It leverages Snyk intelligence, which evolved from Snyk's deterministic SCA fix engine for pull requests, and works from the same Snyk findings as Claude Code + Snyk scan. It runs its own scan, the build and tests, and repairs errors before finishing.

Sandbox and leakage controls

The agent runs in a restricted container modelled on the Evo Remediation Agent's production sandbox: a non-root user, a read-only filesystem except for the repository and a scratch directory, and 24 GB of memory. The ecosystem image (JDK 17 with Maven or Gradle, Node 20, or Python 3.11 with pip or Poetry) adds Node.js, a shell, jq and yq, and the project's dependencies are installed beforehand. Package registries are reachable (Maven Central through a read-through cache); Gradle wrapper distributions are pre-seeded because GitHub is blocked. Claude Code runs headless, without permission prompts, with its standard tools, including web fetch and web search; the Evo Remediation Agent has its own tools to read, edit and run commands, and no web tool. Three controls limit leakage of the eventual fix:

• No history. Each repository is a depth-1 clone of the pinned commit, with no remote, refs or tags.
• No GitHub. GitHub and mirrors that serve its repository files (jsDelivr, raw.githack, Sourcegraph) are blocked, and Claude Code has matching WebFetch deny rules and no gh CLI. Two repositories whose setup fetches from GitHub are exempt from the host block; only the WebFetch rules apply there.
• No Snyk for Claude Code unaided. It gets no Snyk CLI or token: snyk.io, security.snyk.io, api.snyk.io and app.snyk.io are blocked, and links to security.snyk.io are removed from its prompt.

The rest of the web, including changelogs and migration guides hosted outside GitHub, stays open; release notes hosted on GitHub are blocked with the rest of GitHub.

Verification

The verifier never trusts the agent's report. After the agent exits, it runs in a fresh container over the agent's working tree, as root with fresh caches and output directory, so nothing the agent left outside the repository (for example PATH shims or a local Maven repository) can affect the verdict. It runs all four gates in order, even when an earlier gate fails.

1. Dependencies resolve. The edited manifests and lockfiles install with the package manager's strict command, as CI would run it: mvn dependency:resolve, npm ci, yarn install --frozen-lockfile, poetry install, or pip install -r requirements.txt plus the project. Workarounds applied only during the agent's session do not count. Three npm repositories without a committed lockfile fall back to npm install. Gradle resolves dependencies during the build, so its resolution failures surface at the build gate; for Maven, which settles version conflicts itself, the gate is effectively a build precondition. If the strict install fails, npm and Yarn repositories fall back to a lenient install only so that the remaining gates can still run; the gate still fails.
2. Rescan: targeted vulnerabilities fixed. In a post-fix snyk test of every target manifest, the targeted vulnerabilities count as fixed when they are no longer reachable through the upgraded package or through any newly introduced dependency path. A rescan that fails or omits a target manifest fails the gate. Both scans use --ignore-policy, so a .snyk file cannot hide a vulnerability.
3. Build. Maven and Gradle projects compile (mvn compile, gradlew compileJava); JavaScript projects run their TypeScript type-check (tsc --noEmit) or build script; Python projects import their main package, catching removed modules or symbols. Seven JavaScript repositories have neither a type-check nor a build script, so only their tests catch breakage.
4. Tests. Every test that passed in the baseline still passes; a skipped or missing test counts as a regression. Tests are matched by name from JUnit XML (Maven, Gradle, pytest) or Jest JSON reports. For the six Mocha-based npm repositories without JUnit output, the passing count in the log must not drop below the baseline's, and a log with no count fails.

An attempt is a PASS only if all four gates pass. A gate that does not apply counts as passed. Agents may adapt tests to breaking changes (renames, API migrations, imports) but must not edit tests to make them pass.

Metrics

Gate pass rates

The share of the 307 upgrades that pass each gate individually, and all four gates (the PASS rate). A gate that does not apply counts as passed.

Vulnerability-fix rate

The share of the 2,066 fixable vulnerability instances that are fixed; an instance whose rescan failed counts as not fixed.

Per-repository (macro) rates

Repositories contribute 1 to 46 upgrades, so pooled rates, computed over all 307 upgrades together, lean towards a few large repositories. Macro rates average per-repository rates; they are reported for the PASS rate.

Paired comparisons

For two agents on the same upgrades, the macro PASS-rate difference is tested for significance with a cluster bootstrap over repositories (10,000 resamples), which accounts for correlation within a repository.

Snyk-advice-solvable split

The upgrades the package manager bump passes, a lower bound on what mechanical upgrading can solve. An agent's regressions are upgrades the bump passes and the agent fails; its gains, upgrades it passes and the bump fails. PASS rates are reported on both sides.

All four harnesses ran on the same 307 upgrades from 43 repositories. The three agents share identical baselines (2,066 fixable vulnerability instances in Snyk scans of 4–5 October 2026), with no infrastructure errors and no wall-clock timeouts. Four verification rescans failed (one for the package manager bump, one for Claude Code + Snyk scan, two for Claude Code unaided); those attempts count as failures. The package manager bump ran three days later, on 8 October 2026, and its 307 upgrades pair one-to-one with the agents': 305 have the same recommended upgrade, and two the same package with a newer target version.

The site also reports cost and latency for the agents, which the paper does not. The Evo Remediation Agent's cost is estimated from its token counts at $2 per million input and $10 per million output tokens; Claude Code's is what the tool reports. The two methods are not directly comparable, and token counts are not comparable either. Latency is the agent's run alone.

Threats to validity

Construct validity

• Snyk as task source and judge. Tasks and the rescan use Snyk's vulnerability database, which Claude Code unaided cannot consult, so a fixed version taken from another database may still be flagged. This penalises only Claude Code unaided; the Evo Remediation Agent and Claude Code + Snyk scan receive the same findings and target.
• Test strength. Breaking changes, especially changes in behaviour, are caught only as well as the project's tests exercise them. Six repositories (91 of the 307 upgrades) have at most four passing baseline tests, leaving most of the work to the build gate. VulnBump also relies only on each project's own tests, which agents can run, with no held-out tests or tests generated to target the upgrade. Held-out tests catch failures that visible ones miss: in DepBench, 61.8% of failed attempts had passed the visible tests or build. Even purpose-built tests catch only part: BreakGuard, which generates tests specifically to detect breaking dependency updates, catches at best 30.3% of those in the BUMP benchmark.
• Verifier noise. Test identities omit the module name, so same-named tests in different modules can mask regressions; three npm repositories without lockfiles let dependency trees float between baseline and verification; and the baseline is measured once, so a flaky test can be recorded as a regression (BUMP reruns tests three times).
• A deliberately simple baseline. The package manager bump never moves a BOM or parent or tries another version, so the Snyk-advice-solvable split is a lower bound on what mechanical upgrading can solve, and the agents' gains overstate what only an agent can do.
• New vulnerabilities not penalised. They appear in more Evo Remediation Agent attempts than Claude Code attempts, so this favours the Evo Remediation Agent.

Internal validity

• Leakage. The GitHub and Snyk blocks are host-file based, without egress filtering, so DNS-over-HTTPS, hard-coded IPs, unlisted mirrors, web archives and search snippets can bypass them; more simply, about 25 repositories are published libraries whose later releases are on package registries. Leakage would favour Claude Code, since the Evo Remediation Agent has no web tool and its prompt forbids such fetches. Claude Code transcripts were not kept, so leakage cannot be audited.
• Training-data exposure. 40 of the 43 public repositories are pinned at commits that predate Claude Sonnet 5.5's training data cutoff (June 2026), so the model may have seen these projects, their later upgrades or fixes for the same vulnerabilities. This may inflate absolute pass rates, but all three agents share the model.

External validity

• Case set selection. A few very large repositories, such as Apache Kafka and Spring Framework, were excluded, so VulnBump under-represents very large repositories and, since several are Gradle builds, Gradle.
• Scope. VulnBump covers the six package managers behind about 92% of production issue–project pairs (not Go, .NET, Ruby, PHP or pnpm), and scores upgrades one at a time rather than ordered and combined in one tree, as remediation is usually run in practice.

Conclusion validity

Each agent, and the package manager bump, ran once per upgrade, so the significance tests capture variation across repositories, not reruns. Upgrades are concentrated: the three largest repositories contribute 27% of them, and single repositories dominate some failure categories (one accounts for 28 of the 30 Claude Code dependency-gate failures), so pooled failure counts lean on a few projects, whereas macro rates weight repositories equally. Several package managers have only 3 repositories, so per-package-manager estimates are imprecise. All counts are a snapshot of the vulnerability database at scan time.

Future work

The planned re-run of VulnBump adds repeated runs to measure run-to-run variance; three baseline test runs to detect flakiness; held-out tests that agents cannot run, to catch changes in behaviour; a whole-repository mode that applies all of a repository's upgrades together instead of each in isolation; a verdict that also penalises new vulnerabilities; egress-level leakage controls; further ecosystems such as Go, .NET and pnpm; more models, to separate the effect of the model from that of the agent; and retained transcripts, for failure-cause adjudication by several annotators with measured agreement.

Data availability

The VulnBump harness (task builder, sandbox, verifier and analysis scripts) and the 43 evaluation cases (repository, pinned commit, image and commands) are open source at github.com/snyk-labs/snyk-vulnbump. Tasks are derived from a Snyk scan at run time, which requires the Snyk CLI and a token. The repositories are pinned, but the vulnerability database is not, so a later run on the same commits will usually find more than the 2,066 fixable instances reported here (scanned 4–5 October 2026) and may recommend different targets. For example, CVE-2026-76183 (high severity), published between the last agent scan (5 October) and the package manager bump run (8 October), is fixed only from tomcat-embed-websocket 10.1.60, so two upgrades in tokuhirom/java-handbook target 10.1.60 for the package manager bump but 10.1.59 for the agents. Runs scanned on different days should therefore be compared only on tasks with the same recommended upgrade, and against their own baseline.

The aggregated results are on the data page. Questions about Snyk VulnBump can be raised on GitHub.

Source: VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green. Gabija Pasiunaite, Ryan McMorrow, Brian Clark and Liran Tal (Snyk) and Luke Tu (Brown University and Snyk), 12 October 2026.