Stream ZIP member hashes during vendored verification - #587
Mikola Lysenko (mikolalysenko) merged 2 commits into
Conversation
|
[burn-down agent] Labeled Ready for review at
Slack announcement not sent (no Slack send tool available this run). Generated by Claude Code |
|
Reviewed The streaming reader preserves Git hash framing and ZIP CRC/error checks through EOF, rejects incorrect declared lengths, and removes the inflated-member allocation. Member lookup, path normalization, hash equality and prestaging ownership remain unchanged. Validation: 43 local tests passed ( |
Vendored Maven/NuGet verification inflated each patched ZIP member into a
Veconly to hash it. A small compressed archive could therefore cause a large temporary allocation on every verification or re-run. The shared comparator now streams each member through the existing Git SHA-256 reader, using an 8 KiB buffer and validating the declared length against the bytes actually read.Fixes #569, following the maintainer's direction to stream verification. This is the focused C01 improvement from architecture discussion #560.
The comparator serves Maven, NuGet, service-archive checks and both prestaging paths. Reusing the hash module removes the private buffer-and-hash sequence and keeps Git object framing and stream-length validation in one place. The production change stays in
vendor/common.rs; no new size limit is introduced. The compressed archive is still supplied as bytes, while the decompressed member is streamed.Measured with the same isolated test on macOS (
/usr/bin/time -l, debug test binary): a valid deflated member containing 64 MiB + 1 byte of zeros reduced peak process RSS from 78,594,048 to 11,649,024 bytes (about 75 MiB to 11 MiB, an 85% reduction). Fixture construction also streams, so it never allocates the uncompressed body. Both versions completed the test in 0.06 seconds; this measurement establishes the memory improvement, not a runtime speedup.Validation:
vendor::common,vendor::maven_repo,vendor::nuget_feed,vendor::prestage,vendor::service_fetchandhash::git_sha256.vendor_ledger_schema_e2e; its fixture regeneration utility remains intentionally ignored.git diff --checkpassed. Strict workspace clippy passed in CI.CI also exposed an outdated Yarn Berry benchmark expectation on both base and head: #465 made hosted redirects update
package.jsonas well asyarn.lock. A separate fixture-only commit includes both expected files, retaining the exact rewritten-file and content-drift checks. No benchmark thresholds or runtime behavior changed.Implementation and validation are complete. Ready for review.
Generated by Claude Code