Skip to content

fix: enable TraceOn will cause OOM when putting a large object - #2251

Merged
harshavardhana merged 3 commits into
minio:masterfrom
jiuker:fix-enable-TraceOn-will-cause-OOM-when-putting-a-large-object
Jul 17, 2026
Merged

harshavardhana merged 3 commits into
minio:masterfrom
jiuker:fix-enable-TraceOn-will-cause-OOM-when-putting-a-large-object

Conversation

@jiuker

@jiuker jiuker commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor

fix: enable TraceOn will cause OOM when putting a large object
fix #1771
before pr:
image
we can see max memory is 10GB and CPU is 100%
with this pr:
image
we can see max memory is 26MB and CPU is 34%

Summary by CodeRabbit

Summary

  • Bug Fixes

    • Improved HTTP tracing to avoid handling/suspending large request bodies during trace output, ensuring trace logs focus on request headers without buffering upload-sized content.
  • Tests

    • Added a regression test to confirm tracing with very large payloads keeps memory allocations under control and still includes the request line in the trace output.

fix: enable TraceOn will cause OOM when putting a large object
@coderabbitai

coderabbitai Bot commented Jul 17, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a492efe5-f13e-46f7-9be2-a5df7fa421e4

📥 Commits

Reviewing files that changed from the base of the PR and between 157693e and bb57fe6.

📒 Files selected for processing (1)
  • api_test.go

📝 Walkthrough

Walkthrough

Client.dumpHTTP replaces non-nil request bodies with http.NoBody before HTTP tracing dumps, preventing request payloads from being loaded into trace output. A regression test verifies bounded allocation for a request with a 1 GiB content length.

Changes

HTTP tracing

Layer / File(s) Summary
Suppress request bodies in HTTP dumps
api.go, api_test.go
Client.dumpHTTP replaces present request bodies before generating HTTP trace dumps, while TestDumpHTTPLargeBodyDoesNotAllocate checks memory usage and the traced request line.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: harshavardhana

Poem

A rabbit watched the headers stream,
While giant bodies left the dream.
No payload piled, no memory swoon,
Trace logs stayed light beneath the moon.
Hop-hop—safe dumps arrive soon!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly points to the TraceOn OOM fix for large uploads, which matches the main change.
Linked Issues check ✅ Passed The code stops dumpHTTP from buffering a request body and adds a regression test for large uploads, matching issue #1771.
Out of Scope Changes check ✅ Passed The changes are limited to the TraceOn/OOM fix and its regression test, with no unrelated scope visible.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Comment thread api.go Outdated
Comment thread api.go Outdated
Comment thread api.go
@allanrogerr

Copy link
Copy Markdown
Contributor

The title reads fine as-is; two small wording fixes would make the description accurate:

fix: enable TraceOn will cause OOM when putting a large object
fix #1771
before pr:
<img width="1355" height="384" alt="image" src="https://github.com/user-attachments/assets/8043328c-bbec-43a5-b2e3-e244ceaa0fcd" />
we can see max memory is 10GB and CPU is 100%
with this pr:
<img width="553" height="746" alt="image" src="https://github.com/user-attachments/assets/8e7b6cde-3dc9-47b5-8c92-f2e10411092a" />
we can see max memory is 26MB and CPU is 34%
  • "memery" → "memory" in both measurement lines.
  • The auto-generated release-note bullet says tracing "no longer displays request body contents" — request bodies were never shown in traces: DumpRequestOut is called with body=false and always trimmed the body from the returned dump. What this PR actually changes is that the dump no longer buffers a body-sized dummy in memory while producing the header-only trace. Something like "HTTP tracing no longer buffers a request-body-sized dummy in memory when tracing uploads" would describe it accurately for release notes.

@allanrogerr allanrogerr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jiuker Please add the in-package test

@allanrogerr

Copy link
Copy Markdown
Contributor

Took another pass over this against master and it looks solid — nothing to change.

The mechanism holds up end to end. With body=false, httputil.DumpRequestOut asks outgoingLength(req) for the size, and when the body is non-nil with a set Content-Length it installs a Content-Length-sized neverEnding('x') dummy reader and buffers the whole thing into the dump before slicing it off (net/http/httputil/dump.go). Swapping in http.NoBody drives outgoingLength to 0, so no dummy body is generated and only the header is dumped — which is exactly the goal.

Two things I checked specifically so this can't bite later:

  • The mutation is harmless to the actual transfer. dumpHTTP runs only after httpClient.Do(req) has returned, and executeMethod builds a fresh *http.Request via newRequest() on every retry attempt — so setting req.Body = http.NoBody can't truncate the upload that was sent, nor a retried one.
  • The response side is already safe. DumpResponse(resp, false) uses failureToReadBody{}/emptyBody rather than a Content-Length-sized dummy, so large GETs don't have the symmetric problem. The request-only fix is complete.

And the regression test does what it claims — I ran it both ways:

Command Result
pre-fix (guard removed) go test -run TestDumpHTTPLargeBodyDoesNotAllocate FAIL — dumpHTTP allocated 2,684,481,184 bytes, 6.47s
this PR go test -race -run TestDumpHTTPLargeBodyDoesNotAllocate PASS — 0.01s

go vet, gofumpt -d, and a full-package go test -c are all clean. 👍

@allanrogerr
allanrogerr requested a review from klauspost July 17, 2026 14:03
@harshavardhana
harshavardhana merged commit c30a92d into minio:master Jul 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable TraceOn will cause OOM

4 participants