Repository navigation
Serialize bytearray exports with resizing - #9087
Conversation
Keep descriptor acquisition and export retention under the bytearray read lock, and recheck exports after acquiring a resize lock. Guard initializer and sequence deletion writes, preserve zero-index initialization callbacks, and match RawIOBase.read buffer ownership without bypassing export checks. Add bounded Python export-lifetime and concurrent decode/memoryview regression coverage. Assisted-by: OpenAI Codex:model identifier unavailable
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to The supplied changes add regression tests for bytearray export and resize handling. No concrete merge-blocking issue was found in the supplied test changes; hosted CI and maintainer review remain pending as the PR states. Pre-merge checks |
|
Merging this PR will not alter performance
Comparing Footnotes
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🔇 Additional comments (8)
crates/vm/src/bytes_inner.rs (1)
228-245: LGTM!Also applies to: 281-282
crates/vm/src/builtins/bytearray.rs (4)
779-790: LGTM!
848-852: LGTM!
921-921: LGTM!
864-869: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
⚠️ Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.The export check after the lock does not stop a new export from being acquired while the guard is held.
try_resizable_optchecksexportsafter it takes the write lock. Another thread can still incrementexportswithout the lock while this thread holds the guard. These paths do this:
countfind_exportedjoinsplit__contains__hexFor example,
__contains__increments the counter and then callsself.inner(). That call waits on the read lock until the resize completes, so memory safety holds for these paths.
as_buffertakes the read lock beforePyBuffer::newretains the export, so buffer exports are also safe. The design is correct only if every path that incrementsexportstakes the read lock before it reads the data. Add a short comment that states this rule so future export paths follow it.extra_tests/snippets/builtin_bytearray_exports.py (1)
1-136: LGTM!extra_tests/snippets/builtin_bytearray_threading.py (1)
1-55: LGTM!crates/vm/src/stdlib/_io.rs (1)
765-771: LGTM!
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository: RustPython/RustPython/.coderabbit.yml
- Review profile: CHILL
- Plan: Advanced
- Run ID:
a3dd5f43-6526-46e8-9c38-582c3e3e72a7
📒 Files selected for processing (5)
crates/vm/src/builtins/bytearray.rscrates/vm/src/bytes_inner.rscrates/vm/src/stdlib/_io.rsextra_tests/snippets/builtin_bytearray_exports.pyextra_tests/snippets/builtin_bytearray_threading.py
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d0819a2fab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let mut inner = options.get_inner(bytearray_from_object, vm)?; | ||
| core::mem::swap(&mut *zelf.inner_mut(), &mut inner); | ||
| // Reject exports and clear existing contents before converting source. | ||
| zelf.try_resizable(vm)?.elements.clear(); |
There was a problem hiding this comment.
Allow no-op initialization of exported empty bytearrays
When an empty bytearray already has a live memoryview, __init__() and __init__(0) perform no resize; CPython permits both and still invokes conversion callbacks for an int-like zero. The unconditional resize guard here instead raises BufferError before options conversion, regressing the previously successful no-op and suppressing those callbacks. Skip the initial guard/clear when the current length is zero and only reject if the eventual result actually needs a resize.
AGENTS.md reference: AGENTS.md:L7-L7
Useful? React with 👍 / 👎.
| if zero_count { | ||
| return Ok(()); |
There was a problem hiding this comment.
Release the previous allocation for zero-count initialization
When a large existing bytearray is reinitialized with an integer (or int-like value) of zero, this early return follows Vec::clear(), which preserves capacity, so bytearray.__alloc__() continues to expose the old allocation and potentially gigabytes remain held for the object's lifetime. The old implementation swapped in the newly created empty inner and released that storage; replace the backing vector while performing the initial clear so the zero-count callback behavior is kept without retaining the old allocation.
Useful? React with 👍 / 👎.
| zelf._setitem_by_index(i, &value, vm) | ||
| } else { | ||
| zelf.borrow_buf_mut().delitem_by_index(vm, i) | ||
| zelf.try_resizable(vm)?.elements.delitem_by_index(vm, i) |
There was a problem hiding this comment.
Validate the sequence index before rejecting an export
When a C extension calls PySequence_DelItem on a bytearray with a live export and a positive index at or beyond the current length, this guard runs before delitem_by_index can validate the bounds, so the call now raises BufferError; CPython's bytearray sequence slot reports IndexError for such an index without attempting a resize. Validate the index before acquiring the resize guard, then revalidate under the guard, so only a valid deletion is treated as a resize.
AGENTS.md reference: AGENTS.md:L7-L7
Useful? React with 👍 / 👎.
Reset initializer storage before zero-count callbacks and returns. Validate integer deletion indices with a nonblocking snapshot before export rejection where possible, then revalidate under the resize guard. Preserve fast export rejection when contended to avoid codec callback deadlocks. Add Python regressions for exported-empty rejection, initializer capacity release, callback ordering, and ordinary deletion error precedence. Assisted-by: OpenAI Codex:model identifier unavailable
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fcd4f4c8ae
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let mut inner = options.get_inner(bytearray_from_object, vm)?; | ||
| core::mem::swap(&mut *zelf.inner_mut(), &mut inner); | ||
| // Reject exports and clear existing contents before converting source. | ||
| zelf.try_resizable(vm)?.elements = Vec::new(); |
There was a problem hiding this comment.
Reject reentrant initialization without self-deadlocking
When target + other reaches a Python other.__buffer__ that calls target.__init__(object()) (and catches the exception), AsSequence::concat is still holding target.inner() for reading. This new eager try_resizable then waits for the same thread's write lock, so the expression hangs; in the parent, invalid-source conversion raised TypeError before any write lock was taken and concatenation completed. Mark the outer borrow as an export so reentry gets BufferError, or otherwise avoid taking this write lock reentrantly.
Useful? React with 👍 / 👎.
|
|
||
| from testutils import assert_raises | ||
|
|
||
| for args in ((), (b"abc",), (b"x",), (b"",), (object(),)): |
There was a problem hiding this comment.
Remove the prohibited test-logic additions
Remove this new regression snippet and the two sibling builtin_bytearray_* snippets: the repository's critical test policy allows test-file edits only for adding or removing expected-failure markers, whereas these files introduce new assertions, control flow, and test data. Keeping them in this change violates the contribution rules even if the tests pass.
AGENTS.md reference: AGENTS.md:L273-L279
Useful? React with 👍 / 👎.
Acquire both operand exports before borrowing their bytes and release them in CPython order. Cover reentrant initialization, mutable callbacks and export cleanup through public concatenation APIs. Assisted-by: OpenAI Codex:model identifier unavailable
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9fdae0a3f7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let mut inner = options.get_inner(bytearray_from_object, vm)?; | ||
| core::mem::swap(&mut *zelf.inner_mut(), &mut inner); | ||
| // Reject exports and clear existing contents before converting source. | ||
| zelf.try_resizable(vm)?.elements = Vec::new(); |
There was a problem hiding this comment.
Reject initializer reentry instead of waiting on the read lock
The new concat buffering removes the earlier target + other deadlock, but target == other still calls zelf.inner().cmp(...) with a read guard while other.__buffer__ runs. If that callback calls target.__init__(object()) and catches the exception, this eager try_resizable waits forever for the same thread's read guard; the parent converted the invalid source first and returned TypeError instead. Mark outer callback borrows such as comparison and starts/endswith as exports, or make this guard fail rather than wait reentrantly.
Useful? React with 👍 / 👎.
|
|
||
| from testutils import assert_raises | ||
|
|
||
| for concat in (operator.add, operator.concat, bytearray.__add__): |
There was a problem hiding this comment.
Remove the newly added concat test snippet
This fourth regression file is fresh evidence beyond the three snippets named in the existing review: it adds a loop, helper classes, and assertions even though the scoped critical policy permits test-file edits only to add or remove expected-failure decorators. Remove this test logic rather than expanding the prohibited test changes.
AGENTS.md reference: AGENTS.md:L273-L279
Useful? React with 👍 / 👎.
Keep operation-specific export and release ordering for comparisons, searches, affix tuples and translation. Resolve padding arguments before borrowing the receiver, and retain buffer owners through computation without holding bytearray locks across Python callbacks. Add bounded Python regressions for callback mutation, reentrant initialization, lazy affixes, subclass protocols and ordered cleanup. Assisted-by: OpenAI Codex:model identifier unavailable
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a0fec1f8c0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let left = Self::buffer_snapshot(left, vm)?; | ||
| let right = Self::buffer_snapshot(right, vm)?; |
There was a problem hiding this comment.
Avoid cloning both buffers during comparisons
When equal-length operands are large, and for every ordering comparison, these snapshots allocate and copy both complete operands before comparing them; even a == a for an exact bytearray now requires two full-size temporary allocations. The parent implementation compared borrowed slices without this extra 2× memory requirement, so otherwise valid comparisons can now raise MemoryError or cause severe allocation latency; keep the exports alive but compare lock-safe borrowed views, or retain the identity fast path for exact bytearrays.
AGENTS.md reference: AGENTS.md:L7-L7
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
do not make multiple tset files. move 5 files into builtin_bytearray.py
Preserve all existing regression bodies and coverage while following the requested snippet organization. Assisted-by: OpenAI Codex:model identifier unavailable
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f163e13710
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| assert empty == bytearray() | ||
|
|
||
|
|
||
| def test_bytearray_exports(): |
There was a problem hiding this comment.
Remove the appended test logic
Fresh evidence beyond the earlier separate-snippet comments is that this squashed commit now appends the four regression bodies to builtin_bytearray.py, adding 591 lines of assertions, control flow, and test data. Consolidating them into an existing snippet does not resolve the critical policy violation, because test-file edits are limited to adding or removing expected-failure markers; remove this new test logic or obtain an explicit policy exception.
AGENTS.md reference: AGENTS.md:L273-L279
Useful? React with 👍 / 👎.
| left.append_to(&mut bytes); | ||
| right.append_to(&mut bytes); |
There was a problem hiding this comment.
Snapshot aliased concat operands atomically
When both operands expose the same mutable bytearray (for example a + a) and another thread performs a same-length assignment, these two append_to calls acquire and release the bytearray read lock independently. Exports block resizing but not same-size writes, so the writer can run between the calls and produce an old left half plus a new right half, a result that no atomic execution of a + a can produce; reuse one snapshot when the root exporters alias, or otherwise keep the shared storage stable across both copies.
Useful? React with 👍 / 👎.
| if zero_count { | ||
| return Ok(()); | ||
| } | ||
| core::mem::swap(&mut *zelf.try_resizable(vm)?, &mut inner); |
There was a problem hiding this comment.
Treat source-less initialization as complete after clearing
When bytearray.__init__() is called without a source and another thread acquires a memoryview after the initial clear, get_inner_with_zero_count leaves this flag false, so the second resize guard raises BufferError. The call has already discarded the old contents and has no remaining write to perform, making a completed clear appear to fail; mark the missing-source case as no-further-write or keep the initial guard through that callback-free path.
Useful? React with 👍 / 👎.
Summary
Fix a main-common bytearray buffer race surfaced by #8951 macOS CI: decoding used a buffer descriptor of 4 MiB after the underlying bytearray had been cleared.
__index__twice.RawIOBase.readvalidate the count, perform a guarded resize, and take the temporary buffer, following pinned CPython 3.15.0 source and preserving retained-view errors.Lib/files or existing assertions are changed.Base:
f39da82d9d38cd5bdceb9e645205e830f65126fb.Head:
d0819a2fabf2017c5603e089acc7c3a22020f241.Validation
cargo checkpassed; root and matching-feature VM-library Clippy passed with-D warnings; normal commit hooks passed.builtin_bytearray.py/builtin_bytes.pypassed on the exact stamped candidate, each in a timeout-controlled subprocess.test_bytes.FreeThreadingTest.test_free_threading_bytearraypassed once in an isolated subprocess. Its body predates final CPython's additionaltake_bytescases; this is validation of the failing CI method, not full final-bytearray API coverage.d220409a13b77de7a6c30abe16ea90db01528edc). Constructor and raw-I/O error/order semantics were checked against that source.Full workspace/C-API suites, release validation and non-Linux runtime coverage were not run locally. Existing incremental-iterable constructor differences, the public Rust mutable-vector API boundary and final
take_bytesAPI additions remain outside this repair. The sequence-slot deletion change has source review rather than direct C-API runtime coverage. Keep this PR draft and unmerged pending hosted CI and maintainer review.AI assistance
OpenAI Codex implemented, reviewed and validated this change under maintainer direction. The exact model identifier is unavailable in this environment; the commit includes an
Assisted-bytrailer. This draft does not claim completed human validation.Summary by CodeRabbit