Skip to content

gh-104223: Fix issues with inheriting from buffer classes #104227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JelleZijlstra merged 14 commits into from
May 8, 2023
+334 −13
Merged

gh-104223: Fix issues with inheriting from buffer classes #104227

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
b7fb7a4
Fix buffer classes using super()
JelleZijlstra May 6, 2023
c717784
Fix bf_releasebuffer in child classes
JelleZijlstra May 6, 2023
9fcb214
Merge branch 'main' into pep688fix
JelleZijlstra May 6, 2023
88ce24d
Add a test
JelleZijlstra May 6, 2023
80a4b0b
Deal with __release_buffer__ saving references to the buffer
JelleZijlstra May 7, 2023
6e31664
Add another test
JelleZijlstra May 7, 2023
b73e29c
More paranoid bufferwrapper_releasebuf
JelleZijlstra May 7, 2023
15ad02d
that did not work
JelleZijlstra May 7, 2023
9ded873
Update Objects/typeobject.c
JelleZijlstra May 7, 2023
3b50138
Update Objects/typeobject.c
JelleZijlstra May 7, 2023
056efee
Merge remote-tracking branch 'upstream/main' into pep688fix
JelleZijlstra May 7, 2023
9ee8eb5
Better message for new flag
JelleZijlstra May 7, 2023
5536853
More tests
JelleZijlstra May 7, 2023
02d7b9c
Merge branch 'main' into pep688fix
JelleZijlstra May 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
More tests
  • Loading branch information
@JelleZijlstra
JelleZijlstra committed May 7, 2023
commit 5536853ed0f11c7e5bb38163af53e16e51432dc9
59 changes: 59 additions & 0 deletions Lib/test/test_buffer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4690,6 +4690,65 @@ def __release_buffer__(self, buffer):
c.buffer.release()
ba.clear()

@kumaraditya303 kumaraditya303 May 7, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test in which the class has multiple inheritance? Also tests for when there are two or classes in mro which support buffer protocol?

@JelleZijlstra JelleZijlstra May 7, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding some tests. It's hard to come up with a case where multiple bases support the C buffer protocol, because that will almost inevitably lead to:

Traceback (most recent call last):
  File "/Users/jelle/py/cpython/Lib/test/test_buffer.py", line 4707, in test_two_buffer_bases
    class A(bytearray, bytes):
TypeError: multiple bases have instance lay-out conflict

@kumaraditya303 kumaraditya303 May 7, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, maybe try one var length and one pure python type which implements buffer protocol.

@JelleZijlstra JelleZijlstra May 7, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, just pushed. Thanks for the review!

@kumaraditya303 kumaraditya303 May 7, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding some tests. It's hard to come up with a case where multiple bases support the C buffer protocol, because that will almost inevitably lead to:

I see, you would have to write a custom type in C whose layout doesn't conflicts for that. Probably an object which has just object header and supports buffer protocol.

@kumaraditya303 kumaraditya303 May 7, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH, I agree this is an edge case but I try to be extra careful when touching typeobject, hope you don't mind the extra work.

@JelleZijlstra JelleZijlstra May 7, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to catch this now then right before 3.12 final!

@JelleZijlstra JelleZijlstra May 7, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed the bug I outlined above, I'll have to step out for a few hours and fix it after that.

@JelleZijlstra JelleZijlstra May 7, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about it more, that crash isn't actually due to PEP 688: it reproduces without any Python __buffer__ method. So I don't think there's any more interesting cases to cover for PEP 688.

@JelleZijlstra JelleZijlstra May 8, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed #104297 for that case.

def test_multiple_inheritance_buffer_last(self):
class A:
def __buffer__(self, flags):
return memoryview(b"hello A")

class B(A, bytearray):
def __buffer__(self, flags):
return super().__buffer__(flags)

b = B(b"hello")
with memoryview(b) as mv:
self.assertEqual(mv.tobytes(), b"hello A")

class Releaser:
def __release_buffer__(self, buffer):
self.buffer = buffer

class C(Releaser, bytearray):
def __buffer__(self, flags):
return super().__buffer__(flags)

c = C(b"hello C")
with memoryview(c) as mv:
self.assertEqual(mv.tobytes(), b"hello C")
c.clear()
with self.assertRaises(ValueError):
c.buffer.tobytes()

def test_multiple_inheritance_buffer_last(self):
class A:
def __buffer__(self, flags):
raise RuntimeError("should not be called")

def __release_buffer__(self, buffer):
raise RuntimeError("should not be called")

class B(bytearray, A):
def __buffer__(self, flags):
return super().__buffer__(flags)

b = B(b"hello")
with memoryview(b) as mv:
self.assertEqual(mv.tobytes(), b"hello")

class Releaser:
buffer = None
def __release_buffer__(self, buffer):
self.buffer = buffer

class C(bytearray, Releaser):
def __buffer__(self, flags):
return super().__buffer__(flags)

c = C(b"hello")
with memoryview(c) as mv:
self.assertEqual(mv.tobytes(), b"hello")
c.clear()
self.assertIs(c.buffer, None)


if __name__ == "__main__":
unittest.main()