Decode compressed response bodies incrementally - #1126
Conversation
Previously each raw chunk was fully inflated in a single `decompress()` call before being re-chunked, so a small compressed chunk could inflate to an arbitrarily large buffer and `iter_bytes(chunk_size)` did not actually bound memory. Rework the content decoders to yield bounded pieces as they decode: `gzip`/`deflate` drain a shared `ZlibDecompressor` with `max_length`, `brotli` uses `output_buffer_limit` (now requires `brotli>=1.2.0`), and `zstd` uses `max_length` on the stdlib `compression.zstd` backend. `MultiDecoder` pipes children lazily so the bound holds across stacked encodings. `iter_bytes(chunk_size)` now bounds peak memory like urllib3's `read(amt)`. Also close the underlying stream when decoding raises part-way through, so a decode error releases the connection instead of leaking it.
|
Docs preview: https://c4239a3d-httpx2-docs.pydantic.workers.dev |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 472041bacf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 6 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Merging this PR will not alter performance
Comparing 
Footnotes
|
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 8 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
…ecoding # Conflicts: # src/httpx2/httpx2/_decoders.py
## 2.12.0 (August 18th, 2026) ### Changed * Use `backports.zstd` for Zstandard decoding on Python 3.13 and earlier. ([#1146](pydantic/httpx2#1146)) ### Fixed * Bound peak memory while streaming compressed responses and close response streams when decoding fails. ([#1126](pydantic/httpx2#1126)) ## 2.11.0 (August 18th, 2026) ### Added * Add the public `Origin` value object and `URL.origin` property for normalized, hashable origin comparisons. ([#1134](pydantic/httpx2#1134)) ### Changed * Require Brotli 1.2.0 or later for the `brotli` extra. ([#1141](pydantic/httpx2#1141)) ### Fixed * Restore deprecated status code aliases. ([#1135](pydantic/httpx2#1135)) * Extract HTTP/2 release notes from changelog headings correctly. ([#1136](pydantic/httpx2#1136)) * Respect explicit `Transfer-Encoding` headers and expose buffered request body lengths to WSGI applications. ([#1137](pydantic/httpx2#1137)) * Validate multipart part header names and values before serialization. ([#1142](pydantic/httpx2#1142))
This branch was previously deployed
1 participant
Summary
The content decoders used to inflate each raw chunk fully in a single
decompress()call and re-chunk the result afterwards, so a small compressed chunk could inflate to an arbitrarily large buffer anditer_bytes(chunk_size)did not actually bound peak memory.This reworks the decoders to yield bounded pieces as they decode:
gzipanddeflatedrain a shared zlib decompressor withmax_length.output_buffer_limit, adjusted for the C backend's allocation behavior.max_lengthfromcompression.zstdorbackports.zstd.MultiDecoderpipes its children lazily, so the bound holds across stacked encodings.Each decode step is bounded to 1 MiB (
MAX_DECODE_CHUNK_SIZE).iter_bytes(chunk_size)andaiter_bytes(chunk_size)now bound peak memory rather than materializing a whole decoded chunk before re-slicing it.The change also closes the underlying stream when decoding raises part-way through, so a decode error releases the connection instead of leaking it.
Validation
scripts/checkscripts/test && scripts/coverage- 1,990 passed, 1 skipped, 100% coverageAI Disclaimer
This PR was developed with the assistance of either Claude or Codex. I've reviewed and verified the changes.