lzma: treat empty input as a no-op, not end of stream - #8752
Conversation
decompress_chunks reports whether the codec signalled LZMA_STREAM_END, and it short-circuits when the chunker has nothing left to feed. That early return answered "the stream ended" for the case where no input was supplied at all, so LZMADecompressor.decompress(b"") claimed a finish the decoder had never signalled. The caller latches that answer with `self.eof |= stream_end`, which only ever flips eof on. A single empty call therefore finished the object for good: needs_input went false and every later call was rejected by the `if self.eof` guard with EOFError, before a single compressed byte had been seen. CPython leaves eof false and needs_input true, and accepts compressed data handed over by a later call. d = lzma.LZMADecompressor() d.decompress(b"") # d.eof was True, now False d.decompress(blob) # was EOFError, now b'data' The zlib engine returns false from the same short-circuit; lzma had drifted away from it. bz2 was fixed incidentally by RustPython#8638 when its engine moved into crates/common, so only lzma needed a code change. Verified on macOS arm64 against CPython 3.14.7: the script from the issue now prints the documented "False True" and b'data' for both decompressors. test_decompressor_chunks_empty in Lib/test/test_lzma.py covers exactly this and no longer needs its expectedFailure marker. The added engine test is named after the zlib one that already existed and is mirrored into bz2, so all three engines stay pinned to the same behaviour. test_lzma, test_bz2, test_zlib, test_gzip, test_tarfile, cargo test --workspace, cargo fmt --check and cargo clippy -p rustpython-common --all-features --all-targets -D warnings are clean. Fixes RustPython#8637 Assisted-by: Claude Code:claude-opus-5
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository: RustPython/RustPython/.coderabbit.yml Review profile: CHILL Plan: Advanced Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (2)
Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review. 📝 WalkthroughWalkthroughThe change keeps empty decompression calls from marking streams as finished. LZMA updates its EOF result, and new tests verify that LZMA and BZ2 accept compressed data in a later call. ChangesCompression empty-input handling
Priority: ➖ Normal Estimated code review effort: 2 (Simple) | ~10 minutes Change: Bug fix · Severity of issue fixed: Medium Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
📦 Library DependenciesThe following Lib/ modules were modified. Here are their dependencies: [x] lib: cpython/Lib/lzma.py dependencies:
dependent tests: (102 tests)
Legend:
|
Merging this PR will not alter performance
Comparing Footnotes
|
One of checkbox below must be checked.
Summary
LZMADecompressor.decompress(b"")finished the decompressor instead of doing nothing.decompress_chunksreturns whether the codec signalledLZMA_STREAM_END, and itshort-circuits when the chunker has nothing left to feed. That early return answered
"the stream ended" for the case where no input was supplied at all — an empty hand,
not an empty stream.
The caller latches that answer with
self.eof |= stream_end, which only ever flipseofon. A single empty call therefore finished the object for good:needs_inputwent false and every later call was rejected by the
if self.eofguard withEOFError, before a single compressed byte had been seen.The zlib engine returns
falsefrom the identical short-circuit incompression/zlib.rs; lzma had drifted away from it, and this puts the two back inagreement.
On the bz2 half of the issue:
BZ2Decompressorwas fixed incidentally by #8638,which moved the bz2 engine into
crates/common. I confirmed currentmainalreadymatches CPython there, so this PR adds only a regression test for bz2 and no
behaviour change.
Testing
Run on macOS arm64 (aarch64-apple-darwin) against a local CPython 3.14.7.
The script from the issue now prints the documented
False True/b'data'for bothdecompressors, matching CPython exactly.
Lib/test/test_lzma.py::test_decompressor_chunks_empty— this is precisely thereported scenario, and it was marked
expectedFailurewith this veryEOFError.Marker removed. I re-checked by reverting only the one-line engine change and
rebuilding: the test fails with
EOFError: End of stream already reachedattest_lzma.py:152, and passes with the change, so it genuinely pins the regressionrather than riding along.
empty_input_does_not_finish_a_stream— new engine unit test, named after the zlibtest that already existed and mirrored into bz2, so all three engines stay pinned to
the same behaviour instead of drifting apart again.
test_lzma— 121 run, 1 skipped, SUCCESStest_bz2,test_zlib,test_gzip,test_tarfile— 1009 run, 113 skipped, SUCCESScargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher --exclude rustpython-capi— cleancargo fmt --check— cleancargo clippy -p rustpython-common --all-features --all-targets -- -D warnings— cleanOne note on the capi suite, which AGENTS.md also asks for:
cargo testincrates/capidies with SIGSEGV onabstract_::iter::tests::next_item. I verified itcrashes identically on unmodified
main(2006c63) with this change stashed, so it ispre-existing and unrelated.
Not verified on Linux or Windows; the change is platform-independent control flow with
no
cfg-gated paths.AI disclosure
Per the AI policy, this patch was written with AI assistance (Claude Code, model
claude-opus-5); the commit carries anAssisted-by:trailer. I reviewed the change,and the CPython comparison, the fails-without-the-fix check, and the test and lint runs
above were all executed locally on real hardware.
Summary by CodeRabbit