Repository navigation
fix: declare aws-chunked content encoding on streaming uploads - #2277
harshavardhana merged 2 commits into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughThe signer now uses shared AWS chunked content-encoding handling for streaming requests. Existing encodings are preserved, duplicate ChangesAWS chunked encoding
Estimated code review effort: 3 (Moderate) | ~20 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Streaming SigV4 and unsigned-trailer uploads framed the body as aws-chunked without declaring it: the req.TransferEncoding writes were silently dropped by net/http. Set the Content-Encoding header at all four call sites, preserving caller-set encodings. Fixes minio#1802
ef2b4ac to
43d9cc1
Compare
RDMA failure fixed in #2276
Description
Streaming SigV4 and unsigned-trailer uploads frame the request body as aws-chunked but never declare it. PR #1803 tried to declare it through
req.TransferEncoding = []string{"aws-chunked"}, which Go'snet/httpsilently drops (it only honors"chunked"), so nothing reached the wire and #1802 was reopened.setAwsChunkedContentEncoding, used by all four production call sites: the shared signed-streaming preparer (StreamingSignV4/Express/Outposts), the unsigned-streaming preparer, and the two trailer paths inrequest-signature-v4.go.Content-Encoding: aws-chunkedbefore the seed signature is computed, so the header is signed. A caller-set encoding survives asaws-chunked,gzip(the two trailer paths previously clobbered it), multi-value headers are merged, and an already-present token (any case, with or without whitespace) is left unchanged.req.TransferEncoding = []string{"aws-chunked"}writes.content-encodingnow joinsSignedHeaders). The AWS-docs-derived trailer golden is unchanged, which pins helper idempotence.SignedHeadersmembership, an httptest wire round-trip, and one test per entry point (StreamingUnsignedV4,SignV4Trailer,UnsignedTrailer, multi-value header).Motivation and Context
Fixes #1802. Servers that do not sniff
x-amz-content-sha256store the chunk framing as object data (see the issue thread and gitea#33638). The AWS SigV4 streaming spec listsContent-Encodingunder the headers a chunked upload must include ("Set the value toaws-chunked... For example:Content-Encoding : aws-chunked,gzip"), and its example request signs it.Compatibility notes:
request-signature-v4.go), so this extends an existing wire shape to plain-HTTP streaming uploads rather than introducing a new one. Conformant verifiers recompute the signature from received headers, so signatures stay valid.trimAwsChunkedContentEncoding. A server that storesContent-Encodingverbatim would now echoaws-chunkedin object metadata for plain-HTTP streaming PUTs, as it already does for trailing-checksum PUTs.How to test this PR?
Side-by-side results, master (
802bd60) versus this PR:802bd60net/httpround trip (httptest), all three streaming pathsContent-Encoding: "",Transfer-Encoding: []— the framing is never declaredContent-Encoding: aws-chunkedon all three pathsContent-Encoding = "", want "aws-chunked"andcontent-encoding missing from SignedHeaders-race, repeated across 8 runsquay.io/minio/minio:latest): PutObject + GetObject + StatObject for plain, user-gzip, and no-trailing shapesx-amz-content-sha256; nothing declares the framingContent-Encodingis""/"gzip"—aws-chunkedstripped, user gzip preserved38cab3af...(content-encoding not signed)00748050...— re-derived independently from first principles; the derivation reproduces the AWS-docs control4f232c43..., and the unchanged trailer golden106e2a8a...still passes byte-identicallyTrimSpaceandHeader.Valuesguards are mutation-verified: reverting either one fails the matching new testTypes of changes
Checklist:
commit-idorPR #here)Summary by CodeRabbit
aws-chunkedencoding values in request headers.