zlib: clear unconsumed_tail when a bounded decompress reaches stream end by JMLX42 · Pull Request #8908 · RustPython/RustPython

JMLX42 · 2026-09-29T16:10:05Z

When a max_length-bounded Decompress.decompress() call reaches the end of the stream, unconsumed_tail kept the previous call's tail, including input the call had already consumed, and a later flush() moved those stale bytes into unused_data.

Decompressor::decompress_inner (crates/common/src/compression/zlib.rs) now empties unconsumed_tail at end of stream and puts the bytes after the stream in unused_data, as CPython's save_unconsumed_input() does since gh-158169 (python/cpython commit 8122ff4818). CPython releases before that change report the bytes after the stream in unconsumed_tail, so the new snippet case accepts either value when it runs under CPython.

Tests

New unit test stream_end_clears_unconsumed_tail (fails before the change: the tail held 29 stale bytes ending in NEXT).
New case in extra_tests/snippets/stdlib_zlib.py with the reproducer from zlib.decompressobj: unconsumed_tail keeps stale input when a max_length-bounded call reaches end of stream #8906.
cargo test -p rustpython-common --features zlib: 16 passed. test_zlib under RustPython: 76 run, 4 skipped, 0 failed, before and after; no expectedFailure marker changed.
cargo fmt --check, clippy on rustpython-common with -Dwarnings, ruff and cspell: clean.

AI assistance

This change was written with Claude Code (commit trailer Assisted-by: Claude Code:claude-opus-5-5). It is opened as a draft until a human has verified it, per the AI policy.

🤖 Written by Claude Code (2.1.283) using model claude-opus-5-5

Summary by CodeRabbit

Bug Fixes
- Fixed zlib decompression when a stream ends with trailing data. Trailing bytes are now reported as unused data, and the unconsumed input is cleared at stream end. Calling flush() preserves the reported trailing data and input state, keeping decompression results consistent after bounded reads.

When a max_length-bounded Decompress.decompress() call reached the end of the stream, the bytes after the stream went to unused_data, but unconsumed_tail kept the previous call's tail, including input the call had already consumed. A later flush() then moved those stale bytes into unused_data. Empty unconsumed_tail at end of stream, as CPython's save_unconsumed_input() does since gh-158169. CPython releases before that change report the bytes after the stream in unconsumed_tail, so the snippet accepts either value when it runs under CPython. Fixes RustPython#8906 Assisted-by: Claude Code:claude-opus-5-5

coderabbitai · 2026-09-29T16:10:13Z

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: RustPython/RustPython/.coderabbit.yml

Review profile: CHILL

Plan: Advanced

Run ID: 1b357bae-0f74-40a4-9818-d7b2be9d73ed

📥 Commits

Reviewing files that changed from the base of the PR and between 99c0071 and 6408380.

📒 Files selected for processing (1)

extra_tests/snippets/stdlib_zlib.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📝 Walkthrough

Walkthrough

When the stream ends, the decompressor now records bytes after the stream in unused_data and clears unconsumed_tail. Regression tests check bounded decompression, trailing data, and state after flush().

Changes

Zlib decompression tail handling

Layer / File(s)	Summary
Tail state and regression coverage `crates/common/src/compression/zlib.rs`, `extra_tests/snippets/stdlib_zlib.py`	When the stream ends, remaining input is added to `unused_data` and `unconsumed_tail` is cleared. Tests check the output, trailing data, and tail state before and after `flush()`.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: youknowone

Merge Risk: ⚪ Minimal · up to 64083

The change addresses the reported decompression-state issue, and the supplied regression coverage checks the affected bounded-decompression and flush behavior. No material merge-blocking risk is established.

Architecture Summary

Architecture risk: 🔵 Low · up to 64083

The change affects 2 systems.

Changed systems: crates, extra_tests

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

observed — crates (service) was modified; 1 changed file maps to changed impact.
observed — extra_tests (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

observed — Modified behavior in crates/common/src/compression/zlib.rs: decompress_inner now stores bytes after the consumed input in unused_data and clears unconsumed_tail when the stream ends; otherwise it replaces the tail with the remaining input, including clearing it when that input is empty. Previously, it updated either buffer only when unconsumed input existed, and could leave stale tail contents when the stream ended with no remaining bytes.
observed — Modified behavior in crates/common/src/compression/zlib.rs: Added stream_end_clears_unconsumed_tail, which checks that after bounded decompression and subsequent stream completion, the output matches the source, trailing NEXT bytes are in unused_data, and unconsumed_tail is empty both before and after flush.
observed — Modified behavior in extra_tests/snippets/stdlib_zlib.py: Adds a regression test for bounded decompression with bytes after the compressed stream. It checks the two-call decompression flow, EOF and output, permits the tail to be empty or the trailing NEXT bytes, and verifies that flushing preserves the tail and updates unused_data to NEXT + tail.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: clearing unconsumed_tail when a bounded decompression reaches stream end.
Linked Issues check	✅ Passed	The change satisfies issue `#8906`. `Decompressor::decompress_inner` appends bytes after stream end to `unused_data`, clears `unconsumed_tail`, and replaces the tail with only the remaining input for no…
Out of Scope Changes check	✅ Passed	The pull request changes zlib decompression state handling and adds focused regression tests. These changes directly support issue `#8906`. The reviewed diff contains no unrelated implementation or test…
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 2 files.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Choose between the empty slice and the rest of the input in one expression instead of reassigning a mutable binding. Assisted-by: Claude Code:claude-opus-5-5

JMLX42 · 2026-09-29T16:16:47Z

Human review by me:

Changed a let mut into let unconsumed = if to avoid keeping a mut in scope.
Approved the if !unconsumed_tail.is_empty() { removal because AFAIK clear() is a no-op if the tail is empty anyway.

chatgpt-codex-connector · 2026-09-29T16:17:04Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

codspeed · 2026-09-29T16:33:56Z

Merging this PR will not alter performance

✅ 62 untouched benchmarks
⏩ 4 skipped benchmarks¹

_{Comparing JMLX42:fix-zlib-stale-unconsumed-tail (99c0071) with main (a7d75d2)}

4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

The snippet no longer branches on sys.implementation. RustPython and CPython after gh-158169 give an empty tail; CPython before it gives the bytes after the stream. The stale tail of RustPythongh-8906 fails either way. Assisted-by: Claude Code:claude-opus-5-5

JMLX42 mentioned this pull request Sep 29, 2026

zlib: return output inflate still holds after the last input byte #8909

Open

zlib: bind the unconsumed input with an if expression

99c0071

Choose between the empty slice and the rest of the input in one expression instead of reassigning a mutable binding. Assisted-by: Claude Code:claude-opus-5-5

JMLX42 marked this pull request as ready for review September 29, 2026 16:17

youknowone reviewed Sep 29, 2026

View reviewed changes

Comment thread extra_tests/snippets/stdlib_zlib.py Outdated

youknowone approved these changes Sep 30, 2026

View reviewed changes

youknowone merged commit 3805038 into RustPython:main Sep 30, 2026
29 checks passed

Navigation Menu

zlib: clear unconsumed_tail when a bounded decompress reaches stream end - #8908

zlib: clear unconsumed_tail when a bounded decompress reaches stream end#8908
youknowone merged 3 commits into
RustPython:mainfrom
JMLX42:fix-zlib-stale-unconsumed-tail

JMLX42 commented Sep 29, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Sep 29, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

JMLX42 commented Sep 29, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 29, 2026

Uh oh!

codspeed Bot commented Sep 29, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

JMLX42 commented Sep 29, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests

AI assistance

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Architecture Summary

Uh oh!

JMLX42 commented Sep 29, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 29, 2026

Uh oh!

codspeed Bot commented Sep 29, 2026

Merging this PR will not alter performance

Footnotes

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JMLX42 commented Sep 29, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Sep 29, 2026 •

edited

Loading