Repository navigation
memoryview.cast("?") has undefined behavior on some platforms/compilers #148390
Description
Bug report
Bug description:
I am opening an issue to follow-up #83870 and https://discuss.python.org/t/behavior-of-struct-format-native-bool/3774 while I was investigating #148286. Here is the problem:
>>> import struct
>>> struct.pack("????", 0, 1, 2, 3)
b'\x00\x01\x01\x01'
>>> struct.unpack("????", b"\0\1\2\3")
(False, True, True, True)For struct.[un]pack, we see that there is a conversion from bool(object) to \x00 or \x01 as well as from \x00 to False and anything else that is 1 byte to True. The docs say:
For the '?' format character, the return value is either True or False. When packing, the truth value of the argument object is used. Either 0 or 1 in the native or standard bool representation will be packed, and any non-zero value will be True when unpacking.
but as the DPO thread says, it may be confusing. I think it makes more sense to say that packing to ? (for Python) is more or less equivalent to cast it to bool and then to the C _Bool (in Python, there is a well-defined conversion from any object to our own bool type).
However, for memoryview, we have the following:
>>> memoryview(b'\0\1\2\3').cast("?").tolist()
[False, True, False, True]In particular, there is a reduction modulo tolist() performs the following on each memoryview's item:
_Bool c;
memcpy((char *)&c, ptr, sizeof _Bool);The above is an undefined behavior. I think memoryview.cast(fmt).tolist() should behave as if we were doing struct.unpack(fmt * len(mv), mv.obj) instead. If this is not desired, we should at least document this discrepancy. Unfortunately, PEP-3118 does not address this (AFAIK).
cc @encukou @vstinner @serhiy-storchaka @ngoldbaum
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Activity
vstinner commentedon Apr 11, 2026
Previously,
struct.unpack('?', data)had an undefined behavior: see issue gh-125118. It has been fixed by the commit 87d7315.I agree that we should fix
memoryview()behavior to fix the undefined behavior.picnixz commentedon Apr 11, 2026
Should we consider it a real feature or a bug (i.e how to backport it?)? I don't know how annoying it can be for downstream users. It might hide subtle bugs, but I don't know how it's used in general.
skirpichev commentedon Apr 12, 2026
I think this should be backported on same ground as for referenced issue. I don't think that people may rely on undefined behavior.
pythongh-148390: fix undefined behavior of `memoryview(...).cast("?")`
gh-148390: fix undefined behavior of `memoryview(...).cast("?")` (#14…
vstinner commentedon Apr 15, 2026
Hum, I fail to reproduce the
memoryviewbehavior on Fedora Stable with GCC 15.2.1. On a release build (gcc -O3), I get:I also get
[False, True, True, True]on the Python executables provided by Fedora (I tested Python 3.11 to Python 3.15): they are also built with GCC.Ah! I'm able to reproduce the issue when building Python with Clang 21.1.8 on Fedora 43.
picnixz commentedon Apr 15, 2026
Yeah, @encukou told me it wasn't present on all compilers/platforms (which is why I reworded the NEWS entry).
[-]`memoryview.cast("?")` has undocumented undefined behavior[/-][+]`memoryview.cast("?")` has undefined behavior on some platforms/compilers[/+]vstinner commentedon Apr 15, 2026
I'm also able to reproduce the issue on Python 3.3 built with Clang 21. (
memoryview.cast()method was added to Python 3.3.)@picnixz:
This question is tricky. Getting a different answer depending on the compiler is quite unusual and sounds more like a bug to me. I would suggest backporting the fix.
If nobody reported the issue already, it's maybe because no one tried to use this feature before.
If there are affected users, they should upgrade Python, or use a workaround like:
There is at least FreeBSD which builds Python with clang.
picnixz commentedon Apr 15, 2026
I also decided to backport this. This may hide nasty issues, and it's a real bug. I prefer breaking broken things rather than leaving them unnoticed.
picnixz commentedon Apr 15, 2026
Note: I will do the backport next week as the automatic ones failed and I do not k oe if I will be available before.
[3.14] pythongh-148390: fix undefined behavior of `memoryview(...).ca…
[3.14] pythongh-148390: fix undefined behavior of `memoryview(...).ca…
pythongh-148390: fix undefined behavior of `memoryview(...).cast("?")` (
[3.14] gh-148390: fix undefined behavior of `memoryview(...).cast("?"…
[3.13] gh-148390: fix undefined behavior of `memoryview(...).cast("?"…