VulnBump1.0 draft
Menu

Results

Five findings from 921 attempts.

One run per agent on 307 upgrades. All three use Claude Sonnet 5.5. Every claim ships with the caveat that qualifies it.

  1. 01

    Only the vulnerability check separates the agents

    On dependency resolution, build and tests alone, the three agents perform similarly (92.8–93.8%). With the rescan, the Evo Remediation Agent passes 90.2% of upgrades, Claude Code + Snyk scan 77.5% and Claude Code unaided 59.6%, a ranking that holds per repository.

    See the evidence →

    Caveat

    Each agent ran once per upgrade, so the significance tests capture variation across repositories, not reruns.

  2. 02

    An SCA scanner is a key tool for agents fixing vulnerabilities

    Snyk's recommended upgrade applied by the package manager bump passes 46.9% of upgrades and Claude Code without Snyk 59.6%; together, as Claude Code + Snyk scan, they pass 77.5%. Without an SCA scan, Claude Code regresses on 20 of the 144 upgrades the bump solves, 15 of them by choosing a version Snyk still flags: web search alone does not reliably find a safe version.

    See the evidence →

    Caveat

    The rescan uses Snyk's vulnerability database, which Claude Code unaided cannot consult. Databases can disagree on which version fixes a vulnerability, so part of the gap may reflect such disagreement.

  3. 03

    Knowing the target is not enough

    Given the same Snyk findings and model, the Evo Remediation Agent's purpose-built harness passes 12.7 percentage points more than Claude Code + Snyk scan, mostly on the 163 upgrades the package manager bump fails, where the Evo Remediation Agent passes 82.2%, Claude Code + Snyk scan 61.3% and Claude Code unaided 36.2%.

    See the evidence →

    Caveat

    The Evo Remediation Agent combines Snyk intelligence with a loop that builds, tests and repairs, and the setup cannot isolate which component makes the difference.

  4. 04

    The agents fail in opposite ways

    Claude Code mostly stops short and fails silently: the vulnerability remains in 107 of Claude Code unaided's 124 failures, 70 of which it claimed it fixed, and in 48 of Claude Code + Snyk scan's 69. The Evo Remediation Agent usually lands the upgrade, so 21 of its 30 failures break the build or tests, where the project's own checks catch them.

    See the evidence →

    Caveat

    Single repositories dominate some failure categories: one accounts for 28 of the 30 Claude Code dependency-gate failures, so pooled failure counts lean on a few projects.

  5. 05

    Major framework migrations remain hard

    PASS rates fall from 96%, 88% and 70% on non-major upgrades to 81%, 61% and 42% on major ones for the Evo Remediation Agent, Claude Code + Snyk scan and Claude Code unaided. Half of the Evo Remediation Agent's major PASSes edit application source, and 23 upgrades, mostly Spring, Spring Boot and Dropwizard migrations, defeat all three agents.

    See the evidence →

    Caveat

    The jump is that of the recommended upgrade. Smaller jumps are not always safe: 10 of the 189 patch and minor upgrades broke the build or tests for at least one agent.

01Explorer

Where upgrades pass or break

Share of fixable vulnerability instances fixed by an attempt that passed every gate. Transitive vulnerabilities make up 70% of fixable instances, yet for every harness, passing upgrades fix a smaller share of them than of direct ones. Snyk's guidance matters most here: it more than doubles the share of transitive instances Claude Code fixes, from 28% to 59%, against 73% for the Evo Remediation Agent; on direct instances, Claude Code fixes 62% without it. The package manager bump is shown for fix difficulty and dependency type, where the paper reports it.

Smallest semantic-versioning jump from the installed version to a fixed release.

  • Evo Remediation Agent
  • Claude Code + Snyk scan
  • Claude Code unaided
  • Package manager bump
  • did not pass

Patch

713 vulnerabilities

Minor

782 vulnerabilities

Major

520 vulnerabilities

Unclassifiable

51 vulnerabilities

View exact values
Fix difficultyEvo Remediation Agent: fix / pass (n)Claude Code + Snyk scan: fix / pass (n)Claude Code unaided: fix / pass (n)Package manager bump: fix / pass (n)
Patch96% / 79% (713)96% / 67% (713)89% / 37% (713)— / 28% (713)
Minor98% / 82% (782)81% / 65% (782)74% / 45% (782)— / 39% (782)
Major93% / 70% (520)68% / 55% (520)46% / 30% (520)— / 17% (520)
Unclassifiable96% / 88% (51)92% / 33% (51)90% / 24% (51)—
Dataset 1.0.0-draftBars show the share of vulnerability instances fixed by a PASS, within each bucket.Per targeted vulnerability, not per attempt · rounded to whole percentDashed: the package manager bump, pass rate only, as the paper reports it

02Failures

The first gate each failure hit

Claude Code mostly stops short of the fix; the Evo Remediation Agent lands the upgrade and sometimes breaks the code. The rescan is the first failed gate in 107 of Claude Code unaided's 124 failures and 48 of Claude Code + Snyk scan's 69, but in only 9 of the Evo Remediation Agent's 30. 21 of its failures break the build or tests, and on 11 of these upgrades Claude Code + Snyk scan left the vulnerability in place. The package manager bump fails at every gate: of its 163 failures, 78 first fail at the rescan, 54 at dependency resolution and 31 at the build or tests.

Evo Remediation Agent

30 failures

Claude Code + Snyk scan

69 failures

Claude Code unaided

124 failures

Package manager bump

163 failures · non-agentic baseline

  • Failed at dependencies
  • Failed at rescan
  • Failed at build
  • Failed at tests
Root cause of every failed attemptOne root cause per failed attempt, grouped by cause rather than by first failed gate, so counts can differ from the first failed gates above.
CausePackage manager bumpClaude Code unaidedClaude Code + Snyk scanEvo Remediation Agent
Vulnerability remediationThe targeted vulnerability remains.7792408
Upgraded to still-vulnerable version264131
Still reachable via another path7513153
No upgrade made, or reverted015124
Version compatibilityThe new versions violate a version the project pins or requires: a dependency's declared range, the interpreter or the compiler.5430257
Resolver rejects dependency versions2514160
Fix requires newer Python261581
Dependencies need a newer compiler3116
Code compatibilityA changed API or behaviour breaks the project.282415
Changed API breaks build13223
Class or method missing at runtime12016
Changed behaviour fails tests3016
OtherA package removed from PyPI, a source build failure and a failed rescan.4000
Failed attempts1631246930
Dataset 1.0.0-draftThe verifier runs every gate; each failure is counted once, at the first gate it failedRoot causes are grouped by cause, not by gate: runtime version constraints surface at the rescan, compiler ones at the build

Claude Code fails silently. It often avoids the risky upgrade: instead of the major version that fixes every targeted vulnerability, it settles for a patch or minor upgrade that fixes only some. Claude Code unaided stayed within the current major version in 30 attempts, and even when told the target version, Claude Code + Snyk scan did so in 10 (for example spring-boot-starter-test 2.7.18 instead of 3.5.15). Claude Code + Snyk scan also pinned or excluded transitive packages instead (15) or made no change (12). In 70 of its 124 failures, Claude Code unaided claimed to have fixed vulnerabilities that the rescan still found.

Some fixes only work inside the agent's session. In bulletproof-nodejs, Claude Code + Snyk scan left package.json and package-lock.json out of sync, and in express-typescript-starter, Claude Code installed with --legacy-peer-deps without saving the flag. Of the 30 Claude Code attempts that failed the strict dependency gate, 19 passed every other gate; the Evo Remediation Agent never failed it. In 15 Claude Code unaided and 8 Claude Code + Snyk scan failures, the fixed release requires a newer Python than the project targets (filelock 3.20.3 requires Python 3.10); the agent locked the fix only for newer interpreters, leaving the project's own on the vulnerable version.

The Evo Remediation Agent fails visibly. It attempts the upgrades Claude Code often avoids, so half its failures (15 of 30) are code incompatibilities, 13 of them on major upgrades, such as Spring 6's move from javax.* to jakarta.*. It also sometimes upgrades more than needed: in two TypeScript repositories, a broad lockfile regeneration moved transitive type definitions beyond what the project's older compiler can parse (6 failures), and its only regression came from a caret range (agenda@^4.0.0) that floated to a new major version of the MongoDB driver.

The package manager bump never repairs what the upgrade breaks. It always targets the recommended version, but in 75 of its failures the vulnerability stays reachable through another path, mostly in managed builds, where pinning only the named package leaves the rest of its BOM-managed family at old versions.

03Head to head

Four gates, one verdict

The Evo Remediation Agent passes every gate on 90.2% of upgrades, 12.7 percentage points more than Claude Code + Snyk scan and 30.6 more than Claude Code unaided, and fixes 95.9% of vulnerability instances; the Snyk scan and recommended upgrade raise Claude Code's PASS rate by 17.9 points. The individual gates show a trade-off: the Evo Remediation Agent never leaves dependencies unresolved and fixes every targeted vulnerability in 97.1% of upgrades, but its build and test rates trail Claude Code + Snyk scan's by 1.3 and 3.6 points. On dependency resolution, build and tests combined, the agents perform similarly; Claude Code's high build and test rates partly reflect avoidance, since an untouched dependency rarely breaks anything. The package manager bump trails every agent on every gate: it fixes the targeted vulnerabilities about as often as Claude Code unaided, but keeps the project working in only 64.2% of upgrades, because it never repairs what the upgrade breaks. Per-repository (macro) PASS rates give the same ranking.

  • Package manager bump (non-agentic baseline)
  • Claude Code unaided
  • Claude Code + Snyk scan
  • Evo Remediation Agent

Dependencies resolve

The edited manifests and lockfiles install with the package manager's strict command, as CI would run it.

% of upgrades
  • Package manager bump82.4%
  • Claude Code unaided95.4%
  • Claude Code + Snyk scan94.8%
  • Evo Remediation Agent100.0%Best

Targeted vulnerabilities fixed

A post-fix Snyk rescan of every target manifest no longer finds the targeted vulnerabilities through the upgraded package or any newly introduced dependency path.

% of upgrades
  • Package manager bump62.2%
  • Claude Code unaided63.5%
  • Claude Code + Snyk scan83.7%
  • Evo Remediation Agent97.1%Best

Build

The project still compiles, type-checks, or imports its main package.

% of upgrades
  • Package manager bump87.0%
  • Claude Code unaided98.4%Best
  • Claude Code + Snyk scan98.0%
  • Evo Remediation Agent96.7%

Tests pass

Every test that passed in the baseline still passes; a skipped or missing test is a regression.

% of upgrades
  • Package manager bump80.5%
  • Claude Code unaided98.7%
  • Claude Code + Snyk scan99.0%Best
  • Evo Remediation Agent95.4%

Outcomes

Project still works

Dependencies, build and tests all pass, whether or not the vulnerability was fixed.

% of upgrades
  • Package manager bump64.2%
  • Claude Code unaided93.8%Best
  • Claude Code + Snyk scan92.8%
  • Evo Remediation Agent92.8%

All gates pass (PASS)

All four gates pass: the vulnerability is fixed and the project still works.

% of upgrades
  • Package manager bump46.9%
  • Claude Code unaided59.6%
  • Claude Code + Snyk scan77.5%
  • Evo Remediation Agent90.2%Best

Per repository (macro):Package manager bump 51.3%Claude Code unaided 68.0%Claude Code + Snyk scan 80.1%Evo Remediation Agent 93.1%

Vulnerability instances fixed

Share of the fixable vulnerability instances fixed. An instance whose rescan failed counts as not fixed.

% of fixable instances
  • Package manager bump56.5%
  • Claude Code unaided72.7%
  • Claude Code + Snyk scan83.1%
  • Evo Remediation Agent95.9%Best
View exact values
MetricPackage manager bumpClaude Code unaidedClaude Code + Snyk scanEvo Remediation Agent
Dependencies resolve82.4%253 of 30795.4%293 of 30794.8%291 of 307100.0% (best)307 of 307
Targeted vulnerabilities fixed62.2%191 of 30763.5%195 of 30783.7%257 of 30797.1% (best)298 of 307
Build87.0%267 of 30798.4% (best)302 of 30798.0%301 of 30796.7%297 of 307
Tests pass80.5%247 of 30798.7%303 of 30799.0% (best)304 of 30795.4%293 of 307
Project still works64.2%197 of 30793.8% (best)288 of 30792.8%285 of 30792.8%285 of 307
All gates pass (PASS)46.9%144 of 30759.6%183 of 30777.5%238 of 30790.2% (best)277 of 307
Vulnerability instances fixed56.5%1,168 of 2,06672.7%1,503 of 2,06683.1%1,716 of 2,06695.9% (best)1,982 of 2,066
All gates pass, per repository (macro)51.3%68.0%80.1%93.1%
Dataset 1.0.0-draftOne run per harness · Claude Sonnet 5.5 for every agentDashed: the package manager bump, which only applies Snyk's recommended versionGate and PASS rates are out of 307 upgrades; instances out of 2,066

04Shipping gates

Where attempts fall out

Each line follows one harness's upgrades through the gates in order. Claude Code loses most at the rescan, where the vulnerability is still in place. Most of the Evo Remediation Agent's fewer failures come later, at build and test. The package manager bump loses upgrades at every gate.

307 upgrades
→ 4 gates
StartAttempted
Evo Remediation AgentSnyk agentic remediation
90.2% PASS
Claude Code + Snyk scanGiven Snyk target + CLI
77.5% PASS
Claude Code unaidedNo target, Snyk blocked
59.6% PASS
Package manager bumpSnyk's version, no agent
46.9% PASS
Selected gate
Gate 2 · Vulnerabilities fixed

A post-fix Snyk rescan of every target manifest no longer finds the targeted vulnerabilities through the upgraded package or any newly introduced dependency path.

Evo Remediation Agent
−9
vulnerability not fixed · 298 continue
Claude Code + Snyk scan
−48
vulnerability not fixed · 243 continue
Claude Code unaided
−107
vulnerability not fixed · 186 continue
Package manager bump
−78
vulnerability not fixed · 175 continue
View exact values
HarnessAttemptedDependencies resolveVulnerabilities fixedBuildTests pass
Evo Remediation Agent307307 (100.0%) −0298 (97.1%) −9289 (94.1%) −9277 (90.2%) −12
Claude Code + Snyk scan307291 (94.8%) −16243 (79.2%) −48240 (78.2%) −3238 (77.5%) −2
Claude Code unaided307293 (95.4%) −14186 (60.6%) −107183 (59.6%) −3183 (59.6%) −0
Package manager bump307253 (82.4%) −54175 (57.0%) −78159 (51.8%) −16144 (46.9%) −15
Dataset 1.0.0-draftPurple: upgrades still passing after each gate, in order · orange: dropped at that gateOutlined: the non-agentic package manager bump

05Paired comparisons

The same upgrades, the same ranking

Matched on the same upgrades, the Evo Remediation Agent passes 46 that Claude Code + Snyk scan fails and fails 7 that it passes, 39 more PASSes in all. The Snyk scan and recommended upgrade help Claude Code on 64 upgrades and hurt it on 9. Every pairwise difference in macro PASS rate between the agents is statistically significant under a cluster bootstrap over repositories.

ComparisonOnly first passesOnly second passesMacro PASS difference
Evo Remediation Agent vs Claude Code + Snyk scan467+13.0 pp
Evo Remediation Agent vs Claude Code unaided962+25.1 pp
Claude Code + Snyk scan vs Claude Code unaided649+12.1 pp

Who passes each upgrade

Upgrades
All three pass172
Claude Code + Snyk scan and Evo Remediation Agent59
Only Evo Remediation Agent37
None passes23
Claude Code unaided and Evo Remediation Agent9
Only Claude Code + Snyk scan5
Claude Code unaided and Claude Code + Snyk scan2
Only Claude Code unaided0
Dataset 1.0.0-draftAgents matched on the same 307 upgradesEvery macro difference is significant under a cluster bootstrap over repositories (10,000 resamples)Pink bars: upgrades Evo Remediation Agent passed

06Snyk's guidance

Snyk's guidance secures the easy half; the agents separate on the rest

Neither Snyk's guidance nor an agent is enough on its own. Applied by the package manager bump, without an agent, Snyk's recommended upgrade passes 144 of the 307 upgrades (46.9%); Claude Code without Snyk passes 59.6%; given the Snyk scan and recommended upgrade, Claude Code passes 77.5%. On the 144 Snyk-advice-solvable upgrades, what matters is knowing which version is safe: with Snyk's guidance the agents rarely regress (Claude Code + Snyk scan on 6 upgrades, the Evo Remediation Agent on 1), while Claude Code unaided regresses on 20, in 15 of them by upgrading to a version Snyk still flags. On the other 163, which need more than a version bump, the agent decides the outcome, and these upgrades supply 34 of the Evo Remediation Agent's 39 extra PASSes over Claude Code + Snyk scan.

  • Claude Code unaided
  • Claude Code + Snyk scan
  • Evo Remediation Agent

Snyk-advice-solvable144 upgrades

The 144 upgrades the package manager bump passes. Here an agent can only keep or lose an easy fix.

  • Claude Code unaided86.1%124 of 144
  • Claude Code + Snyk scan95.8%138 of 144
  • Evo Remediation Agent99.3%143 of 144

Other upgrades163 upgrades

The 163 upgrades it fails. Here an agent must add something the advice alone does not.

  • Claude Code unaided36.2%59 of 163
  • Claude Code + Snyk scan61.3%100 of 163
  • Evo Remediation Agent82.2%134 of 163
Against the package manager bumpClaude Code unaidedClaude Code + Snyk scanEvo Remediation Agent
Regressions (the bump passes, the agent fails)2061
Gains (the agent passes, the bump fails)59100134
Dataset 1.0.0-draftThe split is a lower bound on what mechanical upgrading can solve, so the gains overstate what only an agent can doPer-repository (macro) rates give the same ranking on both sides

Web search does not reliably find a safe version. Finding a fixed version by web search, for example on OSV, proved unreliable; an SCA scan names the fixed version and can confirm the upgrade. Across all upgrades, Claude Code unaided chose a still-vulnerable version 64 times, Claude Code + Snyk scan 13 times. Vulnerability databases can disagree on which version fixes a vulnerability, and the rescan uses Snyk Open Source, so part of the gap may reflect such disagreement.

A deliberately simple baseline. The package manager bump never moves a BOM or parent or tries another version, so the split is a lower bound on what mechanical upgrading can solve, and the agents' gains overstate what only an agent can do.

07Size of the version jump

Major upgrades separate the agents most

PASS rates fall from 96%, 88% and 70% on the 189 non-major upgrades to 81%, 61% and 42% on the 118 major ones for the Evo Remediation Agent, Claude Code + Snyk scan and Claude Code unaided. The package manager bump passes 67%, 58% and 25% of patch, minor (including 0.x) and major upgrades. Major upgrades are where code changes are needed: the Evo Remediation Agent edited non-test application source in 52% of its 95 major PASSes, against 5% of its 182 non-major ones; Claude Code did so less often (Claude Code + Snyk scan 32% and 3%, Claude Code unaided 24% and 2%). They are also where headroom remains: 23 upgrades in seven Maven and npm repositories defeated all three agents, mostly major framework migrations (Spring 5 to 6 or 7, Spring Boot 2 to 3 or 4, Dropwizard 2 to 5). Smaller jumps are not always safe: 10 of the 189 patch and minor upgrades broke the build or tests for at least one agent, for example logback-classic 1.2.11 to 1.6.3 and flask 3.0.3 to 3.1.3.

  • Claude Code unaided
  • Claude Code + Snyk scan
  • Evo Remediation Agent
  • Package manager bump (non-agentic baseline)

Patch57 upgrades

  • 77%Claude Code unaided · 44 of 57
  • 98%Claude Code + Snyk scan · 56 of 57
  • 96%Evo Remediation Agent · 55 of 57
  • 67%Package manager bump · 38 of 57

Minor123 upgrades

  • 67%Claude Code unaided · 83 of 123
  • 86%Claude Code + Snyk scan · 106 of 123
  • 98%Evo Remediation Agent · 120 of 123
  • 59%Package manager bump · 73 of 123

Major118 upgrades

  • 42%Claude Code unaided · 50 of 118
  • 61%Claude Code + Snyk scan · 72 of 118
  • 81%Evo Remediation Agent · 95 of 118
  • 25%Package manager bump · 30 of 118

Minor (0.x)9 upgrades

  • 67%Claude Code unaided · 6 of 9
  • 44%Claude Code + Snyk scan · 4 of 9
  • 78%Evo Remediation Agent · 7 of 9
  • 33%Package manager bump · 3 of 9

Semantic versioning gives no compatibility guarantee below 1.0.

View exact values
Recommended jumpUpgradesClaude Code unaidedClaude Code + Snyk scanEvo Remediation AgentPackage manager bump
Patch5777.2% (44)98.2% (56)96.5% (55)66.7% (38)
Minor12367.5% (83)86.2% (106)97.6% (120)59.3% (73)
Major11842.4% (50)61.0% (72)80.5% (95)25.4% (30)
Minor (0.x)966.7% (6)44.4% (4)77.8% (7)33.3% (3)
Dataset 1.0.0-draftShare of upgrades passing every gate, by the jump of the recommended upgradeNot the smallest fixing jump used in the per-vulnerability breakdown

08By package manager

The Evo Remediation Agent is at or above both in every package manager

Per-repository (macro) PASS rate in each package manager, from the harness reports. The Evo Remediation Agent's macro PASS rate is at or above both Claude Code agents in every package manager, although several have only three repositories, so per-package-manager estimates are imprecise.

  • Claude Code unaided
  • Claude Code + Snyk scan
  • Evo Remediation Agent

Maven18 repos · 106 upgrades

  • 54%Claude Code unaided · 56 of 106 passed
  • 75%Claude Code + Snyk scan · 78 of 106 passed
  • 89%Evo Remediation Agent · 88 of 106 passed

Gradle3 repos · 66 upgrades

  • 75%Claude Code unaided · 38 of 66 passed
  • 93%Claude Code + Snyk scan · 59 of 66 passed
  • 100%Evo Remediation Agent · 66 of 66 passed

npm11 repos · 57 upgrades

  • 86%Claude Code unaided · 38 of 57 passed
  • 78%Claude Code + Snyk scan · 36 of 57 passed
  • 94%Evo Remediation Agent · 47 of 57 passed

Yarn3 repos · 18 upgrades

  • 87%Claude Code unaided · 16 of 18 passed
  • 100%Claude Code + Snyk scan · 18 of 18 passed
  • 100%Evo Remediation Agent · 18 of 18 passed

pip3 repos · 18 upgrades

  • 93%Claude Code unaided · 17 of 18 passed
  • 97%Claude Code + Snyk scan · 17 of 18 passed
  • 97%Evo Remediation Agent · 17 of 18 passed

Poetry5 repos · 42 upgrades

  • 49%Claude Code unaided · 18 of 42 passed
  • 72%Claude Code + Snyk scan · 30 of 42 passed
  • 98%Evo Remediation Agent · 41 of 42 passed
View all package-manager metrics
Package managerAgentUpgradesDependencies resolveInstances fixedBuildTestsPASS, pooledPASS, macro
MavenClaude Code unaided106100.0%733 of 1,124 (65.2%)100.0%98.1%52.8%0.539
MavenClaude Code + Snyk scan106100.0%874 of 1,124 (77.8%)100.0%98.1%73.6%0.750
MavenEvo Remediation Agent106100.0%1,058 of 1,124 (94.1%)99.1%87.7%83.0%0.885
GradleClaude Code unaided66100.0%160 of 204 (78.4%)100.0%100.0%57.6%0.746
GradleClaude Code + Snyk scan66100.0%179 of 204 (87.7%)100.0%100.0%89.4%0.929
GradleEvo Remediation Agent66100.0%204 of 204 (100.0%)100.0%100.0%100.0%1.000
npmClaude Code unaided5777.2%265 of 318 (83.3%)93.0%98.2%66.7%0.858
npmClaude Code + Snyk scan5771.9%266 of 318 (83.6%)89.5%100.0%63.2%0.783
npmEvo Remediation Agent57100.0%305 of 318 (95.9%)84.2%98.2%82.5%0.937
YarnClaude Code unaided18100.0%124 of 127 (97.6%)100.0%100.0%88.9%0.865
YarnClaude Code + Snyk scan18100.0%127 of 127 (100.0%)100.0%100.0%100.0%1.000
YarnEvo Remediation Agent18100.0%127 of 127 (100.0%)100.0%100.0%100.0%1.000
pipClaude Code unaided1894.4%130 of 131 (99.2%)94.4%94.4%94.4%0.933
pipClaude Code + Snyk scan18100.0%131 of 131 (100.0%)100.0%94.4%94.4%0.967
pipEvo Remediation Agent18100.0%128 of 131 (97.7%)100.0%100.0%94.4%0.967
PoetryClaude Code unaided42100.0%91 of 162 (56.2%)100.0%100.0%42.9%0.490
PoetryClaude Code + Snyk scan42100.0%139 of 162 (85.8%)100.0%100.0%71.4%0.724
PoetryEvo Remediation Agent42100.0%160 of 162 (98.8%)100.0%100.0%97.6%0.978
Dataset 1.0.0-draftMacro = mean of per-repository PASS ratesWith three repositories in several package managers, these rates are indicative only

09Integrity

What else the harness recorded

Signals recorded alongside the gates, for the three agents. Upgrades can pull in packages with their own known vulnerabilities, which PASS does not penalise: post-fix scans show new instances in 62 Evo Remediation Agent attempts (across 16 repositories), 30 Claude Code + Snyk scan and 33 Claude Code unaided attempts. Rows marked in amber deserve a closer look.

SignalClaude Code unaidedClaude Code + Snyk scanEvo Remediation Agent
False fix claimsAttempts where the agent said it fixed a vulnerability that the rescan still found.7021
Attempts with new vulnerabilitiesAttempts whose post-fix scan shows vulnerability instances that were not in the baseline. PASS does not penalise them.333062
New vulnerability instancesSummed over attempts (critical / high / medium / low), which overcounts unique instances because upgrades in the same repository often introduce the same one.7970 / 462 / 328 / 76230 / 336 / 275 / 121,2340 / 703 / 518 / 13
Fixable instances remainingFixable instances not fixed, failed rescans included (critical / high / medium / low).5630 / 282 / 223 / 583500 / 143 / 167 / 40840 / 35 / 35 / 14
Vulnerabilities movedRemaining instances that now arrive through a direct dependency that did not carry them in the baseline.1010
Tests modifiedAttempts that edited test files. Agents may adapt tests to breaking changes, but must not edit them to make them pass.222651
Unrelated bumpsAttempts that bumped a direct dependency outside the fix's lane. Recorded, not failed.342674
Unverified instancesRemaining instances that only a failed rescan reports.12140
Rescan failedAttempts whose verification rescan failed. They count as failures.210
Turn budget exhaustedAttempts that used their whole turn budget.001
Timed outAttempts stopped at the wall-clock budget. They would count as failures.000
.snyk policy editsAttempts that added or edited a .snyk file. Both scans ignore policy, so this cannot hide a vulnerability.000
Infrastructure errorsThe agent never started, or the container ran out of memory. Excluded from every rate.000

10Cost and latency

The Evo Remediation Agent spends more per attempt

The Evo Remediation Agent takes longer and costs more per attempt. Costs were measured differently: its costs are estimated from token counts at list price, and Claude Code's are what the tool reports. Read them as an order of magnitude, not a ratio.

Latency per attempt

  • Claude Code unaided65 s median128 s mean333 s p90542 s p95
  • Claude Code + Snyk scan124 s median180 s mean408 s p90568 s p95
  • Evo Remediation Agent190 s median456 s mean803 s p903,123 s p95

Cost per attempt

  • Claude Code unaided$0.080 median$0.10 mean$0.18 p90$0.21 p95
  • Claude Code + Snyk scan$0.10 median$0.12 mean$0.19 p90$0.25 p95
  • Evo Remediation Agent$0.74 median$2.09 mean$5.02 p90$8.69 p95
Totals and cost method
ConfigurationMean latencyMean costTotal costCost method
Claude Code unaided128 s$0.10$31.13tool-reported, includes prompt caching
Claude Code + Snyk scan180 s$0.12$36.86tool-reported, includes prompt caching
Evo Remediation Agent456 s$2.09$642.76estimated at $2/M input, $10/M output tokens
Dataset 1.0.0-draftDot = median · bar = median to p95 · ticks = mean (dashed) and p90Latency = the agent's run alone, excluding the harness's install, scan, build, and testsCosts use different methods and are not directly comparable; token counts are not comparable either