zlib: return output inflate still holds after the last input byte by JMLX42 · Pull Request #8909 · RustPython/RustPython

JMLX42 · 2026-09-29T16:12:11Z

decompress_chunks (crates/common/src/compression/zlib.rs) returned before it called inflate when the input was empty, and returned once the input was consumed even when the output buffer was full. Raw deflate has no trailer, so inflate can consume the last input byte and still hold output. Decompress.decompress(b"", n) and Decompress.flush() then returned nothing, and zlib.decompress(raw, -15, bufsize) with a small bufsize raised "incomplete or truncated stream".

This mirrors CPython's zlibmodule.c: inflate runs for empty input, and runs again while it fills the output buffer. A stream with no pending output answers Z_BUF_ERROR, so an empty input or a flush() before any compressed data still leaves eof false (empty_input_does_not_finish_a_stream still passes; empty_flush_does_not_finish_a_stream is new). After a stream error, decompress(b"") now raises again, as CPython does.

Tests

New unit tests empty_input_returns_pending_output, flush_returns_pending_output, small_bufsize_returns_pending_output, empty_flush_does_not_finish_a_stream (the first three fail before the change).
New cases in extra_tests/snippets/stdlib_zlib.py with the reproducer from zlib.decompressobj: decompress(b"", n) and flush() never return output that inflate still holds (raw deflate loses its last bytes under max_length) #8907; they pass on RustPython and on CPython 3.13.5.
rustpython-common: 135 passed; workspace: 1337 passed, 0 failed.
test_zlib + test_gzip: 151 run, 4 skipped, 0 failed, before and after. test_tarfile + test_zipfile + test_shutil: 1360 run, 197 skipped, before and after.
cargo fmt --check and clippy on rustpython-common are clean.

#8908 (#8906) edits the same file and also appends tests at the end of mod tests; whichever merges second needs a trivial rebase.

AI assistance

This change was written with Claude Code (commit trailer Assisted-by: Claude Code:claude-opus-5-5). It is opened as a draft until a human has verified it, per the AI policy.

🤖 Written by Claude Code (2.1.283) using model claude-opus-5-5

Summary by CodeRabbit

Bug Fixes
- Fixed decompression with small output buffers so remaining data can be retrieved by calling decompress again with empty input or by using flush().
- Corrected stream completion reporting: pending output is no longer treated as complete, and empty input does not prematurely finish an incomplete stream.
- Fixed one-shot raw decompression with small buffer sizes.

decompress_chunks returned before it called inflate when the input was empty, and returned once the input was consumed even when the output buffer was full. Raw deflate has no trailer, so inflate can consume the last input byte and still hold output. Decompress.decompress(b"", n) and Decompress.flush() then returned nothing, and zlib.decompress with a small bufsize raised "incomplete or truncated stream". Mirror CPython's zlibmodule.c: call inflate for empty input, and call it again while it fills the output buffer. A stream with no pending output answers Z_BUF_ERROR, so an empty input before any compressed data still leaves eof false. Assisted-by: Claude Code:claude-opus-5-5 Fixes RustPython#8907

coderabbitai · 2026-09-29T16:12:18Z

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: RustPython/RustPython/.coderabbit.yml

Review profile: CHILL

Plan: Advanced

Run ID: 12ebc79e-620d-4349-862d-e9d862ad3f4a

📥 Commits

Reviewing files that changed from the base of the PR and between a70643b and b39f875.

📒 Files selected for processing (1)

crates/common/src/compression/zlib.rs

💤 Files with no reviewable changes (1)

crates/common/src/compression/zlib.rs

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

📝 Walkthrough

Walkthrough

The decompression loop now processes empty input to retrieve pending output. It checks output-buffer fullness before treating exhausted input as complete. Rust and Python regression tests cover pending output, small output limits, and incomplete streams.

Changes

Pending zlib output

Layer / File(s)	Summary
Drain pending output and verify stream state `crates/common/src/compression/zlib.rs`	`decompress_chunks` calls inflate for empty input and continues when the output buffer is full, unless inflate reports stream end.
Test pending output and stream state `crates/common/src/compression/zlib.rs`, `extra_tests/snippets/stdlib_zlib.py`	Tests cover retrieving pending raw-deflate output with empty-input decompression or `flush()`, small output limits, and keeping an incomplete zlib-wrapped stream open for later input.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: youknowone

Merge Risk: ⚪ Minimal · up to b39f8

The reviewed change has no identified issue that needs resolution before merge. Normal checks remain appropriate.

Architecture Summary

Architecture risk: 🟡 Medium · up to a7064

The change affects 2 systems.

Changed systems: crates, extra_tests

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

observed — crates (service) was modified; 1 changed file maps to changed impact.
observed — extra_tests (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

observed — Modified behavior in crates/common/src/compression/zlib.rs: Removed the early return that made empty input produce no output without calling inflate. Empty input now proceeds through the decompression loop, allowing pending output to be returned; the stream remains unfinished when inflate reports a buffer error.
observed — Modified behavior in crates/common/src/compression/zlib.rs: Completion no longer follows merely from exhausted input. The loop also continues when the output buffer was filled, since inflate may have pending output; it returns as unfinished when the buffer is only partially filled and the stream has not ended.
observed — Modified behavior in crates/common/src/compression/zlib.rs: Added tests and a helper that drains raw deflate input in 100-byte output limits until 1,100 of 1,168 bytes are produced. The tests check that empty-input decompression and flush retrieve the remaining output and reach EOF, that one-shot decompression succeeds with buffer sizes 1 and 100, and that flushing an empty stream does not set EOF before later input completes it.
observed — Modified behavior in extra_tests/snippets/stdlib_zlib.py: Added drained_raw_deflate(), which creates a raw-deflate stream for 1168 repeated bytes, consumes output in 100-byte chunks, and asserts that 1100 bytes are returned while the decompressor is not yet at EOF.

Reliability and maintainability

inferred — Risk-relevant change factors for crates: blast_radius_2; direct_dependents_2

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: returning output still held by inflate after the final input byte.
Linked Issues check	✅ Passed	Issue [`#8907`] requires bounded raw-deflate decompression to return output that remains in inflate after input is consumed. `decompress_chunks` now runs inflate for empty input and continues when a ful…
Out of Scope Changes check	✅ Passed	The production change is limited to the shared zlib decompression loop. The added tests cover the pending-output, empty-input, flush, and bounded-output cases required by [`#8907`]. The reviewed changes…
Docstring Coverage	✅ Passed	Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 2 files.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Return when the stream ended, or when the input is used up and inflate left part of the output buffer empty. A full buffer falls through to the existing next iteration, as CPython's loop runs while avail_out is zero, instead of a separate branch ahead of the exit check. Assisted-by: Claude Code:claude-opus-5-5

JMLX42 · 2026-09-29T16:22:06Z

Human review by me.

First impl was using a continue:

                    if produced == additional && !stream_end {
                        // A full output buffer can leave output pending in
                        // inflate, even with no input left.
                        continue 'outer;
                    }

IMHO it was correct but continue makes the code too hard to follow. So I folded it in if stream_end || (data.is_empty() && produced < additional) {.

chatgpt-codex-connector · 2026-09-29T16:23:03Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

youknowone · 2026-09-29T17:18:52Z

+    // Empty input still reaches inflate, like CPython: inflate can hold output
+    // after it consumed the last input byte. A stream with no pending output
+    // answers `BufError`, not `StreamEnd`, so `eof` stays false.


Suggested change

// Empty input still reaches inflate, like CPython: inflate can hold output

// after it consumed the last input byte. A stream with no pending output

// answers `BufError`, not `StreamEnd`, so `eof` stays false.

these comments will look weird after patch

Done in b39f875: the comment is gone. zlib unit tests: 19 passed.

🤖 Written by Claude Code (2.1.283) using model claude-opus-5-5

Assisted-by: Claude Code:claude-opus-5-5

youknowone

looks good now. please rebase it to upstream.
please also check the changes in details yourself.
thanks!

JMLX42 marked this pull request as ready for review September 29, 2026 16:22

youknowone reviewed Sep 29, 2026

View reviewed changes

zlib: drop the comment that stood in for the removed early return

b39f875

Assisted-by: Claude Code:claude-opus-5-5

youknowone approved these changes Oct 1, 2026

View reviewed changes

Navigation Menu

zlib: return output inflate still holds after the last input byte - #8909

zlib: return output inflate still holds after the last input byte#8909
JMLX42 wants to merge 3 commits into
RustPython:mainfrom
JMLX42:fix-zlib-empty-input-drain

JMLX42 commented Sep 29, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Sep 29, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

JMLX42 commented Sep 29, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 29, 2026

Uh oh!

youknowone Sep 29, 2026

Uh oh!

JMLX42 Sep 29, 2026

Uh oh!

youknowone left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	// Empty input still reaches inflate, like CPython: inflate can hold output
	// after it consumed the last input byte. A stream with no pending output
	// answers `BufError`, not `StreamEnd`, so `eof` stays false.

Conversation

JMLX42 commented Sep 29, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests

AI assistance

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Architecture Summary

Uh oh!

JMLX42 commented Sep 29, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 29, 2026

Uh oh!

youknowone Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

JMLX42 Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

youknowone left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JMLX42 commented Sep 29, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Sep 29, 2026 •

edited

Loading