Skip to content

Stream ZIP member hashes during vendored verification - #587

Merged
Mikola Lysenko (mikolalysenko) merged 2 commits into
mainfrom
refactor/stream-zip-verification-20261002
Oct 2, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 2 commits into
mainfrom
refactor/stream-zip-verification-20261002

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Vendored Maven/NuGet verification inflated each patched ZIP member into a Vec only to hash it. A small compressed archive could therefore cause a large temporary allocation on every verification or re-run. The shared comparator now streams each member through the existing Git SHA-256 reader, using an 8 KiB buffer and validating the declared length against the bytes actually read.

Fixes #569, following the maintainer's direction to stream verification. This is the focused C01 improvement from architecture discussion #560.

The comparator serves Maven, NuGet, service-archive checks and both prestaging paths. Reusing the hash module removes the private buffer-and-hash sequence and keeps Git object framing and stream-length validation in one place. The production change stays in vendor/common.rs; no new size limit is introduced. The compressed archive is still supplied as bytes, while the decompressed member is streamed.

Measured with the same isolated test on macOS (/usr/bin/time -l, debug test binary): a valid deflated member containing 64 MiB + 1 byte of zeros reduced peak process RSS from 78,594,048 to 11,649,024 bytes (about 75 MiB to 11 MiB, an 85% reduction). Fixture construction also streams, so it never allocates the uncompressed body. Both versions completed the test in 0.06 seconds; this measurement establishes the memory improvement, not a runtime speedup.

Validation:

  • 222 core tests passed across vendor::common, vendor::maven_repo, vendor::nuget_feed, vendor::prestage, vendor::service_fetch and hash::git_sha256.
  • The 10 ZIP cases include empty and multi-chunk members, a valid member larger than 64 MiB, declared sizes both shorter and longer than the body, and the existing corruption/path/hash/missing-entry cases. The declared-size regression fails on the base and passes with streaming.
  • 3 CLI ledger/service-artifact tests passed in vendor_ledger_schema_e2e; its fixture regeneration utility remains intentionally ignored.
  • 29 benchmark harness tests passed, plus full-size Yarn Berry hosted and rescan scenarios (3,000 packages, 60 patched; one warmup and three measured runs each).
  • Changed-file rustfmt and git diff --check passed. Strict workspace clippy passed in CI.
  • Full CI passed, including Linux/macOS/Windows and release-mode tests, Docker coverage, and downstream Yarn Berry checks. The full benchmark comparison and all compatibility workflows passed. All 452 checks completed without failure on the final commit, including the post-ready Cursor Bugbot review, which reported no findings.

CI also exposed an outdated Yarn Berry benchmark expectation on both base and head: #465 made hosted redirects update package.json as well as yarn.lock. A separate fixture-only commit includes both expected files, retaining the exact rewritten-file and content-drift checks. No benchmark thresholds or runtime behavior changed.

Implementation and validation are complete. Ready for review.


Generated by Claude Code

@mikolalysenko Mikola Lysenko (mikolalysenko) added the arch-refactor PR opened by the scheduled architecture refactor routine label Oct 2, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[burn-down agent] Labeled Ready for review at 329b14672b418378d933f58a07b6c459d2594e28.

  • CI: 446/452 check runs green on this head (6 skipped by path/matrix filters), 0 failing or pending.
  • Mergeable: 0 commits behind main, no conflicts; only human approval is outstanding.
  • Bugbot: the Cursor Bugbot check passed on 329b146 with no findings; no review threads open.
  • Reviewer focus: the declared-length check in vendor/common.rs. A ZIP member whose declared size does not match the streamed byte count now fails verification instead of being hashed.

Slack announcement not sent (no Slack send tool available this run).


Generated by Claude Code

@mikolalysenko
Mikola Lysenko (mikolalysenko) merged commit b8bf049 into main Oct 2, 2026
453 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the refactor/stream-zip-verification-20261002 branch October 2, 2026 19:16
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Reviewed 329b14672b418378d933f58a07b6c459d2594e28: no actionable findings; ready as-is from this review. The PR merged while review was running, at the same reviewed head.

The streaming reader preserves Git hash framing and ZIP CRC/error checks through EOF, rejects incorrect declared lengths, and removes the inflated-member allocation. Member lookup, path normalization, hash equality and prestaging ownership remain unchanged.

Validation: 43 local tests passed (vendor::common: 23, hash::git_sha256: 15, vendor::prestage: 5), including corruption, declared sizes, empty/multichunk members and the 64 MiB + 1-byte member. Exact-head CI has 446 successful checks, 7 skipped, and no failures or pending checks; Bugbot passed. The reviewed files match their merged versions on main 42e7734d.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arch-refactor PR opened by the scheduled architecture refactor routine Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

zip_bytes_match_after_hashes inflates committed .nupkg/.jar entries with no size cap

2 participants