Repository navigation
ctypes: bit field data does not survive round trip #97588
Description
Bit-fields in structures don't seem to give you back the data you put in?
from ctypes import Structure, c_uint, c_ulonglong, c_ushort
class Foo(Structure):
_fields_ = [("A", c_uint, 1), ("B", c_ushort, 16)]
class Bar(Structure):
_fields_ = [("A", c_ulonglong, 1), ("B", c_uint, 32)]
if __name__ == "__main__":
for a in [Foo(), Bar()]:
a.A = 0
a.B = 1
print(a.A, a.B)The above should print
0 1
0 1
But it actually prints
$ python3.10 mini.py
0 0
0 0For comparison and to test my understanding, I expect the following C code to be equivalent to the Python code above:
#include<stdio.h>
struct Foo {
unsigned int A: 1;
unsigned short B: 16;
};
struct Bar {
unsigned long long int A: 1;
unsigned int B: 32;
};
int main(int argc, char** argv) {
struct Foo foo;
foo.A = 0;
foo.B = 1;
printf("%d %d\n", foo.A, foo.B);
struct Bar bar;
bar.A = 0;
bar.B = 1;
printf("%d %d\n", bar.A, bar.B);
return 0;
}The C version prints what we expect:
$ gcc -fsanitize=undefined test.c && ./a.out
0 1
0 1
Your environment
I am on ArchLinux with Python 3.10.7. Python 3.11 and main are also affected. I also randomly tried Python 3.6 with the same result. (Python 3.6 is the oldest one that was easy to install.)
More comprehensive test case
Here's how I actually found the problem reported above. Using Hypothesis:
import ctypes
import string
from hypothesis import assume, example, given, note
from hypothesis import strategies as st
unsigned = [(ctypes.c_ushort, 16), (ctypes.c_uint, 32), (ctypes.c_ulonglong, 64)]
signed = [(ctypes.c_short, 16), (ctypes.c_int, 32), (ctypes.c_longlong, 64)]
types = unsigned + signed
unsigned_types = list(zip(*unsigned))[0]
signed_types = list(zip(*signed))[0]
names = st.lists(st.text(alphabet=string.ascii_letters, min_size=1), unique=True)
@st.composite
def fields_and_set(draw):
names_ = draw(names)
ops = []
results = []
for name in names_:
t, l = draw(st.sampled_from(types))
res = (name, t, draw(st.integers(min_value=1, max_value=l)))
results.append(res)
values = draw(st.lists(st.integers()))
for value in values:
ops.append((res, value))
ops = draw(st.permutations(ops))
return results, ops
def fit_in_bits(value, type_, size):
expect = value % (2**size)
if type_ not in unsigned_types:
if expect >= 2 ** (size - 1):
expect -= 2**size
return expect
@given(fops=fields_and_set())
def test(fops):
(fields, ops) = fops
class BITS(ctypes.Structure):
_fields_ = fields
b = BITS()
for (name, type_, size), value in ops:
expect = fit_in_bits(value, type_, size)
setattr(b, name, value)
j = getattr(b, name)
assert expect == j, f"{expect} != {j}"
if __name__ == "__main__":
test()Thanks to @mdickinson for pointing me in this direction.
Activity
pythongh-97588: Some tests to demonstrate the issue
matthiasgoergens commentedon Sep 27, 2022
Since I'm looking into bit-fields and ctypes anyway at the moment, I'll plan to work on a fix. But if someone else wants to give it a go, I'm happy to let them.
matthiasgoergens commentedon Sep 27, 2022
As far as I can tell the problem might be with
PyCField_FromDescincfield.c. The code looks both a bit complicated and is not expected to be a hot inner loop (it's only for setting up aStructurefrom a description.) So perhaps we might want to move most of its logic into Python.Let's me dig some more.
[-]ctypes: data does not survice round trip[/-][+]ctypes: bit field data does not survice round trip[/+]pythongh-97588: Fix ctypes structs
mdickinson commentedon Oct 8, 2022
@matthiasgoergens I think this is essentially the same issue as existing issues like #95496, #84039, #73939, #59324, etc; it would be good to try to consolidate some of these so we don't end up with an explosion of issues.
matthiasgoergens commentedon Oct 8, 2022
@mdickinson Good idea. That's actually what I am already doing with the fix I have in the works, but consolidating the issues makes sense, too.
[-]ctypes: bit field data does not survice round trip[/-][+]ctypes: bit field data does not survive round trip[/+]matthiasgoergens commentedon Oct 9, 2022
@mdickinson I mentioned the issues that my PR fixes on the PR itself. Is there a way (or necessity) to also consolidate the issues themselves here?
Backport 'pythongh-97588: Fix ctypes structs' to 3.11
Squash 'pythongh-97588: Fix ctypes structs' for backport to 3.10
thesamprice commentedon Mar 27, 2023
@matthiasgoergens Could you review pull request !103052
matthiasgoergens commentedon Mar 27, 2023
@thesamprice I'll have a look, but keep in mind that I don't have any authority here.
thesamprice commentedon Mar 27, 2023
More interested if the one line change passes the tests you wrote .
matthiasgoergens commentedon Mar 27, 2023
Oh, ok. Sure, I can run that against my tests.
I guess I should publish my Python Hypothesis tests, too; and not just the tests in my PR. So you can run them yourself.
Give me a bit of time!
Edit: I'll answer over on #103052
matthiasgoergens commentedon Mar 27, 2023
@thesamprice I added your tests from #103052 to this PR, too. I hope you don't mind.
gh-97588: Align ctypes struct layout to GCC/MSVC (GH-97702)
pythongh-97588: Align ctypes struct layout to GCC/MSVC (pythonGH-97702)
pythongh-97588: Align ctypes struct layout to GCC/MSVC (pythonGH-97702)
gh-97588: Move ctypes struct/union layout logic to Python (GH-123352)
encukou commentedon Sep 9, 2024
Thank you for the report, and the initial fix!
The layout calculation is now done in Python, and a test suite is in place, so we're in a better place to solve the other struct/union layout issues. I'll close this one.
pythongh-97588: remove unused functions in _ctypes/cfield.c
gh-97588: remove unused functions in _ctypes/cfield.c (GH-124010)
pythongh-97588: remove unused functions in _ctypes/cfield.c (pythonGH…