test_ctypes fails on AIX with xlc

@ncoghlan

BPO	27643
Nosy	@ncoghlan, @ericvw, @vadmium, @aixtools
PRs	bpo-27643 - fix test_ctypes test_bitfields with XLC compiler. #5164 2.7 bpo-27643 - Test_C test case needs "signed short" bitfields, but the IBM XLC compiler (on AIX) does not support this (GH-5164) #5556
Files	disable-signed-short.patch ctypes_test_sign_bitfields_2.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-12-28.14:08:05.720>
created_at = <Date 2016-07-28.15:01:29.862>
labels = ['3.7', '3.8', 'type-bug', 'tests']
title = 'test_ctypes fails on AIX with xlc'
updated_at = <Date 2018-12-28.14:08:05.719>
user = 'https://github.com/aixtools'

bugs.python.org fields:

activity = <Date 2018-12-28.14:08:05.719>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2018-12-28.14:08:05.720>
closer = 'ncoghlan'
components = ['Tests']
creation = <Date 2016-07-28.15:01:29.862>
creator = 'Michael.Felt'
dependencies = []
files = ['44005', '44190']
hgrepos = []
issue_num = 27643
keywords = ['patch']
message_count = 29.0
messages = ['271555', '271558', '271561', '271592', '271634', '271635', '271659', '271773', '271960', '272451', '272850', '273005', '273309', '273386', '273393', '273397', '273411', '273510', '273511', '273513', '273528', '309336', '309823', '309827', '309843', '318520', '321918', '326834', '332528']
nosy_count = 5.0
nosy_names = ['ncoghlan', 'ericvw', 'martin.panter', 'David.Edelsohn', 'Michael.Felt']
pr_nums = ['5164', '5556']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27643'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

I am preparing a system with gcc to see if it is compiler related, i.e., goes away with gcc. On the one hand, fingers crossed - but on the other, having bitfields working regardless of the compiler should be preferred.

This issue is similar to a Solaris (C compiler) http://bugs.python.org/issue16275

During the build using clc _ get these messages - which make me hope it is an unexpected compiler issue.

"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for M is not valid. Type unsigned assumed.
"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for N is not valid. Type unsigned assumed.
"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for O is not valid. Type unsigned assumed.
"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for P is not valid. Type unsigned assumed.
"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for Q is not valid. Type unsigned assumed.
"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for R is not valid. Type unsigned assumed.
"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for S is not valid. Type unsigned assumed.

Note: I modified test_bitfields to do an additional print. Maybe this gives an idea about what the issue is (when considering the compiler messages)

root@x064:[/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test]../../../python ./runtests.py
A 0x1001f 0
B 0x2001d 0
C 0x3001a 0
D 0x40016 0
E 0x50011 0
F 0x6000b 0
G 0x70004 0
H 0x80018 4
I 0x9000f 4
M 0x1000e 6
N 0x2000c 6
O 0x30009 6
P 0x40005 6
Q 0x50000 6
R 0x6000a 8
S 0x70003 8
.............s....................ssss............................ssssssssssssssssssssssssssss...s.......s.....................sssssssssssssssssssss......s...........s....s.....................FF...s...............................................s.............sssss.ss....s.......ss..sss...................s......s.....................s..................................................s...........ss.sssss................s.s.............s.
======================================================================
FAIL: test_ints (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test/test_bitfields.py", line 41, in test_ints
    self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name)))
AssertionError: Tuples differ: ('A', 1, -1) != ('A', 1, 1)

First differing element 2:
-1
1

('A', 1, -1)
? -

+ ('A', 1, 1)

======================================================================
FAIL: test_shorts (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test/test_bitfields.py", line 48, in test_shorts
    self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)

First differing element 2:
-1
1

('M', 1, -1)
? -

+ ('M', 1, 1)

----------------------------------------------------------------------
Ran 440 tests in 0.883s (0 modules skipped)
Unavailable resources: printing, refcount

FAILED (failures=2)
root@x064:[/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test]

FYI: similar (exact) results when 64-bit mode:
root@x064:[/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test]../../../python ./runtests.py
A 0x1001f 0
B 0x2001d 0
C 0x3001a 0
D 0x40016 0
E 0x50011 0
F 0x6000b 0
G 0x70004 0
H 0x80018 4
I 0x9000f 4
M 0x1000e 6
N 0x2000c 6
O 0x30009 6
P 0x40005 6
Q 0x50000 6
R 0x6000a 8
S 0x70003 8
.............s....................ssss............................ssssssssssssssssssssssssssss...s.......s.....................sssssssssssssssssssss......s...........s....s.....................FF...s...............................................s.............sssss.ss....s.......ss..sss...................s......s.....................s..................................................s...........ss.sssss................s.s.............s.
======================================================================
FAIL: test_ints (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test/test_bitfields.py", line 41, in test_ints
    self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name)))
AssertionError: Tuples differ: ('A', 1, -1) != ('A', 1, 1)

First differing element 2:
-1
1

('A', 1, -1)
? -

+ ('A', 1, 1)

======================================================================
FAIL: test_shorts (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test/test_bitfields.py", line 48, in test_shorts
    self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)

First differing element 2:
-1
1

('M', 1, -1)
? -

+ ('M', 1, 1)

----------------------------------------------------------------------
Ran 440 tests in 0.870s (0 modules skipped)
Unavailable resources: printing, refcount

FAILED (failures=2)

So, it seems to be a compiler issue - when built using gcc (v4.7.4) the tests take a bit longer, but no failures.

root@x065:[/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test]../../../python runtests.py
A 0x1001f 0
B 0x2001d 0
C 0x3001a 0
D 0x40016 0
E 0x50011 0
F 0x6000b 0
G 0x70004 0
H 0x80018 4
I 0x9000f 4
M 0x1000e 6
N 0x2000c 6
O 0x30009 6
P 0x40005 6
Q 0x50000 6
R 0x6000a 8
S 0x70003 8
.............s....................ssss............................ssssssssssssssssssssssssssss...s.......s.....................sssssssssssssssssssss......s...........s....s..........................s...............................................s.............sssss.ss....s.......ss..sss...................s......s.....................s..................................................s...........ss.sssss................s.s.............s.
----------------------------------------------------------------------
Ran 440 tests in 1.028s (0 modules skipped)
Unavailable resources: printing, refcount

OK

The test seems to be comparing ctypes and native C bit-field structures, c_int and c_short versus native int and short. In general in C, if a bit-field has type “int” without a signed or unsigned qualifier, it is up to the implementation which mode is chosen. This is different from normal, where int always means signed int.

Since ctypes documents c_int as both meaning “int” and “signed int”, I would say that a ctypes bit-field using c_int should always behave as “signed int”. So perhaps we should just change the test structure to use “signed int” and “signed short”. That should also help avoid the warnings, and may help the Solaris situation (if that is still relevant).

I presume this also affects Python 3.

I presume this also affects Python 3.

I presume this as well - however, harder to verify...

michael@x071:[/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test]../../../python test_bitfield.py
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Failed to import the site module
Traceback (most recent call last):
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 571, in <module>
    main()
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 557, in main
    known_paths = addusersitepackages(known_paths)
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 281, in addusersitepackages
    user_site = getusersitepackages()
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 257, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 247, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/sysconfig.py", line 587, in get_config_var
    return get_config_vars().get(name)
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/sysconfig.py", line 536, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/sysconfig.py", line 408, in _init_posix
    from _sysconfigdata import build_time_vars
ImportError: No module named '_sysconfigdata'

I guess I will have to install it, rather than only work from a test directory.

The test seems to be comparing ctypes and native C bit-field structures, c_int and c_short versus native int and short. ... int always means signed int.

I believe int means signed on AIX as well - as the comment from the compiler is that type "unsigned" is assumed for a bitfield.

Note: the single line of code is:
+382 short M: 1, N: 2, O: 3, P: 4, Q: 5, R: 6, S: 7;

And from your comments I am wondering if the fix is 'simple' - to:

+382 unsigned short M: 1, N: 2, O: 3, P: 4, Q: 5, R: 6, S: 7;

I can at least try that.

Looking at Python-3.6.0.0a2:

the line concerned is identical - so that is the verification (for now) that it is also in Python3

FYI - after a couple of test compiles, it seems that IBM XLC does not report a message when the size specified is int, unsigned or long (i.e., does not like short).

Are you interested in the results using a size other than short - or would that break something else I cannot forsee?

Looks like your Python 3 build is messed up. Maybe it doesn’t like running from a different directory. I would try from the main build directory, and note the test_bitfields has an S:

./python -m unittest -v ctypes.test.test_bitfields

What I am suggesting as a fix is to change line 381 from plain “int” to “signed int”, and 382 to “signed short”. I can make a patch later if that will help. Hopefully with these changes the C compiler will use signed integer logic, matching what I believe the ctypes library uses for c_int and c_short.

Using a type other than short is not right, because the Python test is explicitly trying to test c_short behaviour. If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test.

On 30-Jul-16 02:51, Martin Panter wrote:

Martin Panter added the comment:

Looks like your Python 3 build is messed up. Maybe it doesn’t like running from a different directory. I would try from the main build directory, and note the test_bitfields has an S:
I would not know why - it is just a copy of the sources and then
configure, make, make DESTDIR=/some/where install (rather than make
install).
Same procedure I use for python2.7.

./python -m unittest -v ctypes.test.test_bitfields
No changes: 3.5.1

root@x064:[/data/prj/aixtools/python/python-3.5.1.2]./python -m unittest
-v ctypes.test.test_bitfields
test_anon_bitfields (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_c_wchar (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_longlong (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_1 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_2 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_3 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_4 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_multi_bitfields_size (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_nonint_types (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_signed (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_single_bitfield_size (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32_swap_big_endian (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32_swap_little_endian (ctypes.test.test_bitfields.BitFieldTest)
... ok
test_uint64 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_ulonglong (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_unsigned (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_ints (ctypes.test.test_bitfields.C_Test) ... FAIL
test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL

======================================================================
FAIL: test_ints (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
   File 
"/data/prj/aixtools/python/python-3.5.1.2/Lib/ctypes/test/test_bitfields.py", 
line 41, in test_ints
     self.assertEqual(getattr(b, name), func(byref(b), 
name.encode('ascii')))
AssertionError: -1 != 1

======================================================================
FAIL: test_shorts (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
   File 
"/data/prj/aixtools/python/python-3.5.1.2/Lib/ctypes/test/test_bitfields.py", 
line 48, in test_shorts
     self.assertEqual(getattr(b, name), func(byref(b), 
name.encode('ascii')))
AssertionError: -1 != 1

Ran 19 tests in 0.013s

FAILED (failures=2)

What I am suggesting as a fix is to change line 381 from plain “int” to “signed int”, and 382 to “signed short”. I can make a patch later if that will help. Hopefully with these changes the C compiler will use signed integer logic, matching what I believe the ctypes library uses for c_int and c_short.
Working from Python-2.7.12 - added your changes, test still fails:

root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python -m
unittest -v ctypes.test.test_bitfields
test_anon_bitfields (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_c_wchar (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_longlong (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_1 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_2 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_3 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_4 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_multi_bitfields_size (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_nonint_types (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_signed (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_single_bitfield_size (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32_swap_big_endian (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32_swap_little_endian (ctypes.test.test_bitfields.BitFieldTest)
... ok
test_uint64 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_ulonglong (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_unsigned (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_ints (ctypes.test.test_bitfields.C_Test) ... ok
test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL

======================================================================
FAIL: test_shorts (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
   File 
"/data/prj/aixtools/python/python-2.7.12.0/Lib/ctypes/test/test_bitfields.py", 
line 48, in test_shorts
     self.assertEqual((name, i, getattr(b, name)), (name, i, 
func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)

First differing element 2:
-1
1

('M', 1, -1)
? -

+ ('M', 1, 1)

----------------------------------------------------------------------
Ran 19 tests in 0.068s

FAILED (failures=1)

Using a type other than short is not right, because the Python test is explicitly trying to test c_short behaviour. If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test.
Looks like this may be the approach: as signed int, signed short give
the same message - it seems that xlc (and maybe Sun C) does not accept
"short" as a bitfield type.

skipping test - only if compiler is not gcc, as when using gcc, test
works fine.

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue27643\>

Okay, so to be clear, I am assuming XLC supports all of the following fields, and uses unsigned bit fields by default:

struct UNSIGNED_BITS {
    unsigned int AU: 1;
    int A: 1; /* Equivalent to unsigned int */
    signed int AS: 1;
    unsigned short MU: 1;
    short M: 1; /* Equivalent to unsigned short; triggers warning */
};

and that it cannot compile the following:

struct SIGNED_BITS {
    signed short MS: 1; /* Not supported */
};

Attached is what I think a patch to resolve this would look like. However it needs a line in Modules/_ctypes/_ctypes_test.c completed to detect the compiler:

#ifndef /* Something to identify XLC */

Can you figure out a way to test for XLC (but not GCC, which the AIX buildbot uses), and then try my patch out? Hopefully you see no more compiler warnings, test_ints() should now pass, and test_shorts() should be skipped.

Had some flooding (leaking pipes, rather values) issues to fix. Will
look at this asap.

On 04-Aug-16 10:58, Martin Panter wrote:

Martin Panter added the comment:

Okay, so to be clear, I am assuming XLC supports all of the following fields, and uses unsigned bit fields by default:

struct UNSIGNED_BITS {
unsigned int AU: 1;
int A: 1; /* Equivalent to unsigned int */
signed int AS: 1;
unsigned short MU: 1;
short M: 1; /* Equivalent to unsigned short; triggers warning */
};

and that it cannot compile the following:

struct SIGNED_BITS {
signed short MS: 1; /* Not supported */
};

Attached is what I think a patch to resolve this would look like. However it needs a line in Modules/_ctypes/_ctypes_test.c completed to detect the compiler:

#ifndef /* Something to identify XLC */

Can you figure out a way to test for XLC (but not GCC, which the AIX buildbot uses), and then try my patch out? Hopefully you see no more compiler warnings, test_ints() should now pass, and test_shorts() should be skipped.

----------
keywords: +patch
Added file: http://bugs.python.org/file44005/disable-signed-short.patch

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue27643\>

On 8/4/2016 10:58 AM, Martin Panter wrote:

Can you figure out a way to test for XLC (but not GCC, which the AIX buildbot uses), and then try my patch out? Hopefully you see no more compiler warnings, test_ints() should now pass, and test_shorts() should be skipped.

Was not as simple as it sounds. I could not find a specific parameter
provided by xlc and as you wanted something that would only skip on AIX

testing for __GNUC__ (in the positive sense) will not work either.

What I am thinking of now is

#if defined(__GNUC__) || !defined(_AIX)

...

#endif

As this will pass on AIX when GCC is used, but only be skipped on AIX
when GCC is not used.

+++++++++++++++++++++++++++++++++++++++++++++++++++++ More TESTS ++++

Summary - maybe the problem is with the "byref()" routine...

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

My initial test of the patch confused me, so I went back to the original
code - and started again.

I have three partitions I am testing on - x064 (AIX 5.3 TL7, vac v.11);
x065 (AIX 5.3 TL7, gcc 4.7.4),
and x067 ( Debian 3.16.7-ckt9-3 (2015-04-23) ppc64, gcc (Debian
5.3.1-21) 5.3.1 20160528)

My working directory is /data/prj/aixtools/python/python-2.7.12.0 which
is a copy of the extracted Python-2.7.12.
This directory is on NFS, so each system is using the same files as source:

The first builds - no changes - expect to test_bitfields.py:

##for n in "ABCDEFGHIMNOPQRS":
##    print n, hex(getattr(BITS, n).size), getattr(BITS, n).offset
for n in "ABCDEFGHIMNOPQRS":
     b = BITS()
     print n, hex(getattr(BITS, n).size), getattr(BITS, n).offset, 
func(byref(b), n)

x065:
root@x065:[/data/prj/aixtools/python/python-2.7.12.0]./python
Lib/ctypes/test/test_bitfields.py
A 0x1001f 0 0
B 0x2001d 0 0
C 0x3001a 0 0
D 0x40016 0 0
E 0x50011 0 0
F 0x6000b 0 0
G 0x70004 0 0
H 0x80018 4 0
I 0x9000f 4 0
M 0x1000e 6 0
N 0x2000c 6 0
O 0x30009 6 0
P 0x40005 6 0
Q 0x50000 6 0
R 0x6000a 8 0
S 0x70003 8 0

root@x067:/data/prj/aixtools/python/python-2.7.12.0# ./python
Lib/ctypes/test/test_bitfields.py
A 0x1001f 0 0
B 0x2001d 0 0
C 0x3001a 0 0
D 0x40016 0 0
E 0x50011 0 0
F 0x6000b 0 0
G 0x70004 0 0
H 0x80018 4 0
I 0x9000f 4 0
M 0x1000e 6 0
N 0x2000c 6 0
O 0x30009 6 0
P 0x40005 6 0
Q 0x50000 6 0
R 0x6000a 8 0
S 0x70003 8 0
...................
----------------------------------------------------------------------
Ran 19 tests in 0.087s

OK

root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python
Lib/ctypes/test/test_bitfields.py
A 0x1001f 0 0
B 0x2001d 0 0
C 0x3001a 0 0
D 0x40016 0 0
E 0x50011 0 0
F 0x6000b 0 0
G 0x70004 0 0
H 0x80018 4 0
I 0x9000f 4 0
M 0x1000e 6 0
N 0x2000c 6 0
O 0x30009 6 0
P 0x40005 6 0
Q 0x50000 6 0
R 0x6000a 8 0
S 0x70003 8 0
.................FF
======================================================================
FAIL: test_ints (main.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
   File "Lib/ctypes/test/test_bitfields.py", line 45, in test_ints
     self.assertEqual((name, i, getattr(b, name)), (name, i, 
func(byref(b), name)))
AssertionError: Tuples differ: ('A', 1, -1) != ('A', 1, 1)

First differing element 2:
-1
1

('A', 1, -1)
? -

+ ('A', 1, 1)

======================================================================
FAIL: test_shorts (main.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
   File "Lib/ctypes/test/test_bitfields.py", line 52, in test_shorts
     self.assertEqual((name, i, getattr(b, name)), (name, i, 
func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)

First differing element 2:
-1
1

('M', 1, -1)
? -

+ ('M', 1, 1)

----------------------------------------------------------------------
Ran 19 tests in 0.010s

FAILED (failures=2)
root@x064:[/data/prj/aixtools/python/python-2.7.12.0]

Changed the test again to include:
##for n in "ABCDEFGHIMNOPQRS":
## print n, hex(getattr(BITS, n).size), getattr(BITS, n).offset
for n in "ABCDEFGHIMNOPQRS":
b = BITS()
print n, hex(getattr(BITS, n).size), getattr(BITS, n).offset,
getattr(b,n), func(byref(b), n)

root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python
Lib/ctypes/test/test_bitfields.py
A 0x1001f 0 0 0
B 0x2001d 0 0 0
C 0x3001a 0 0 0
D 0x40016 0 0 0
E 0x50011 0 0 0
F 0x6000b 0 0 0
G 0x70004 0 0 0
H 0x80018 4 0 0
I 0x9000f 4 0 0
M 0x1000e 6 0 0
N 0x2000c 6 0 0
O 0x30009 6 0 0
P 0x40005 6 0 0
Q 0x50000 6 0 0
R 0x6000a 8 0 0
S 0x70003 8 0 0
.................FF
======================================================================
FAIL: test_ints (main.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
   File "Lib/ctypes/test/test_bitfields.py", line 45, in test_ints
     self.assertEqual((name, i, getattr(b, name)), (name, i, 
func(byref(b), name)))
AssertionError: Tuples differ: ('A', 1, -1) != ('A', 1, 1)

First differing element 2:
-1
1

('A', 1, -1)
? -

+ ('A', 1, 1)

======================================================================
FAIL: test_shorts (main.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
   File "Lib/ctypes/test/test_bitfields.py", line 52, in test_shorts
     self.assertEqual((name, i, getattr(b, name)), (name, i, 
func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)

First differing element 2:
-1
1

('M', 1, -1)
? -

+ ('M', 1, 1)

----------------------------------------------------------------------
Ran 19 tests in 0.010s

FAILED (failures=2)

So, I am wondering if the problem could be better repaired in
setattr(b, name, i)

And changed the test again:

comment out the initial changes
the following changes (note range limited to 5 for both, rather 512
for first and 256 for second)

class C_Test(unittest.TestCase):

     def test_ints(self):
         for i in range(5):
             for name in "ABCDEFGHI":
                 b = BITS()
                 setattr(b, name, i)
                 print "name:", name, "range:", i, "getattr():", 
getattr(b,name), "byref():", func(byref(b), name)
                 # self.assertEqual((name, i, getattr(b, name)), (name, 
i, func(byref(b), name)))

     def test_shorts(self):
         for i in range(5):
             for name in "MNOPQRS":
                 b = BITS()
                 setattr(b, name, i)
                 print "name:", name, "range:", i, "getattr():", 
getattr(b,name), "byref():", func(byref(b), name)
                 # self.assertEqual((name, i, getattr(b, name)), (name, 
i, func(byref(b), name)))

And for most tests (remember the range is 5 now) - all is fine

root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python
Lib/ctypes/test/test_bitfields.py
.................name: A range: 0 getattr(): 0 byref(): 0
name: B range: 0 getattr(): 0 byref(): 0
name: C range: 0 getattr(): 0 byref(): 0
name: D range: 0 getattr(): 0 byref(): 0
name: E range: 0 getattr(): 0 byref(): 0
name: F range: 0 getattr(): 0 byref(): 0
name: G range: 0 getattr(): 0 byref(): 0
name: H range: 0 getattr(): 0 byref(): 0
name: I range: 0 getattr(): 0 byref(): 0
name: A range: 1 getattr(): -1 byref(): 1
name: B range: 1 getattr(): 1 byref(): 1
name: C range: 1 getattr(): 1 byref(): 1
name: D range: 1 getattr(): 1 byref(): 1
name: E range: 1 getattr(): 1 byref(): 1
name: F range: 1 getattr(): 1 byref(): 1
name: G range: 1 getattr(): 1 byref(): 1
name: H range: 1 getattr(): 1 byref(): 1
name: I range: 1 getattr(): 1 byref(): 1
name: A range: 2 getattr(): 0 byref(): 0
name: B range: 2 getattr(): -2 byref(): 2
name: C range: 2 getattr(): 2 byref(): 2
name: D range: 2 getattr(): 2 byref(): 2
name: E range: 2 getattr(): 2 byref(): 2
name: F range: 2 getattr(): 2 byref(): 2
name: G range: 2 getattr(): 2 byref(): 2
name: H range: 2 getattr(): 2 byref(): 2
name: I range: 2 getattr(): 2 byref(): 2
name: A range: 3 getattr(): -1 byref(): 1
name: B range: 3 getattr(): -1 byref(): 3
name: C range: 3 getattr(): 3 byref(): 3
name: D range: 3 getattr(): 3 byref(): 3
name: E range: 3 getattr(): 3 byref(): 3
name: F range: 3 getattr(): 3 byref(): 3
name: G range: 3 getattr(): 3 byref(): 3
name: H range: 3 getattr(): 3 byref(): 3
name: I range: 3 getattr(): 3 byref(): 3
name: A range: 4 getattr(): 0 byref(): 0
name: B range: 4 getattr(): 0 byref(): 0
name: C range: 4 getattr(): -4 byref(): 4
name: D range: 4 getattr(): 4 byref(): 4
name: E range: 4 getattr(): 4 byref(): 4
name: F range: 4 getattr(): 4 byref(): 4
name: G range: 4 getattr(): 4 byref(): 4

Back to x065...

root@x065:[/data/prj/aixtools/python/python-2.7.12.0]./python Lib
.................name: A range: 0 getattr(): 0 byref(): 0
name: B range: 0 getattr(): 0 byref(): 0
name: C range: 0 getattr(): 0 byref(): 0
name: D range: 0 getattr(): 0 byref(): 0
name: E range: 0 getattr(): 0 byref(): 0
name: F range: 0 getattr(): 0 byref(): 0
name: G range: 0 getattr(): 0 byref(): 0
name: H range: 0 getattr(): 0 byref(): 0
name: I range: 0 getattr(): 0 byref(): 0
name: A range: 1 getattr(): -1 byref(): -1
name: B range: 1 getattr(): 1 byref(): 1
name: C range: 1 getattr(): 1 byref(): 1
name: D range: 1 getattr(): 1 byref(): 1
name: E range: 1 getattr(): 1 byref(): 1
name: F range: 1 getattr(): 1 byref(): 1
name: G range: 1 getattr(): 1 byref(): 1
name: H range: 1 getattr(): 1 byref(): 1
name: I range: 1 getattr(): 1 byref(): 1
name: A range: 2 getattr(): 0 byref(): 0
name: B range: 2 getattr(): -2 byref(): -2
name: C range: 2 getattr(): 2 byref(): 2
name: D range: 2 getattr(): 2 byref(): 2
name: E range: 2 getattr(): 2 byref(): 2
name: F range: 2 getattr(): 2 byref(): 2
name: G range: 2 getattr(): 2 byref(): 2
name: H range: 2 getattr(): 2 byref(): 2
name: I range: 2 getattr(): 2 byref(): 2
name: A range: 3 getattr(): -1 byref(): -1
name: B range: 3 getattr(): -1 byref(): -1
name: C range: 3 getattr(): 3 byref(): 3
name: D range: 3 getattr(): 3 byref(): 3
name: E range: 3 getattr(): 3 byref(): 3
name: F range: 3 getattr(): 3 byref(): 3
name: G range: 3 getattr(): 3 byref(): 3
name: H range: 3 getattr(): 3 byref(): 3
name: I range: 3 getattr(): 3 byref(): 3
name: A range: 4 getattr(): 0 byref(): 0
name: B range: 4 getattr(): 0 byref(): 0
name: C range: 4 getattr(): -4 byref(): -4
name: D range: 4 getattr(): 4 byref(): 4
name: E range: 4 getattr(): 4 byref(): 4
name: F range: 4 getattr(): 4 byref(): 4
name: G range: 4 getattr(): 4 byref(): 4
name: H range: 4 getattr(): 4 byref(): 4
name: I range: 4 getattr(): 4 byref(): 4
.name: M range: 0 getattr(): 0 byref(): 0
name: N range: 0 getattr(): 0 byref(): 0
name: O range: 0 getattr(): 0 byref(): 0
name: P range: 0 getattr(): 0 byref(): 0
name: Q range: 0 getattr(): 0 byref(): 0
name: R range: 0 getattr(): 0 byref(): 0
name: S range: 0 getattr(): 0 byref(): 0
name: M range: 1 getattr(): -1 byref(): -1
name: N range: 1 getattr(): 1 byref(): 1
name: O range: 1 getattr(): 1 byref(): 1
name: P range: 1 getattr(): 1 byref(): 1
name: Q range: 1 getattr(): 1 byref(): 1
name: R range: 1 getattr(): 1 byref(): 1
name: S range: 1 getattr(): 1 byref(): 1
name: M range: 2 getattr(): 0 byref(): 0
name: N range: 2 getattr(): -2 byref(): -2
name: O range: 2 getattr(): 2 byref(): 2
name: P range: 2 getattr(): 2 byref(): 2
name: Q range: 2 getattr(): 2 byref(): 2
name: R range: 2 getattr(): 2 byref(): 2
name: S range: 2 getattr(): 2 byref(): 2
name: M range: 3 getattr(): -1 byref(): -1
name: N range: 3 getattr(): -1 byref(): -1
name: O range: 3 getattr(): 3 byref(): 3
name: P range: 3 getattr(): 3 byref(): 3
name: Q range: 3 getattr(): 3 byref(): 3
name: R range: 3 getattr(): 3 byref(): 3
name: S range: 3 getattr(): 3 byref(): 3
name: M range: 4 getattr(): 0 byref(): 0
name: N range: 4 getattr(): 0 byref(): 0
name: O range: 4 getattr(): -4 byref(): -4
name: P range: 4 getattr(): 4 byref(): 4
name: Q range: 4 getattr(): 4 byref(): 4
name: R range: 4 getattr(): 4 byref(): 4
name: S range: 4 getattr(): 4 byref(): 4
.
-----------------------------------------------------------------
Ran 19 tests in 0.056s

OK

Now looking at the results - it seems the byref() function is where the
difference lies - as getattr() returns the same value for both x064/xlc
and x065/gcc

Hope this helps!

Michael, byref() is just a helper for passing an object’s address to a C function. Calling func(byref(b), ...) in Python is equivalent to the C code

unpack_bitfields(&b, ...)

I still think the root problem is in unpack_bitfields(). When compiled with XLC, your experiments confirm that it always returns unsigned values, and with GCC, it returns signed values.

It looks like you can detect XLC with the __IBMC__ and __xlC__ macros (not sure if there is a practical difference). So perhaps we can fix this with

#ifndef __xlC__

I'll get to this asap. Have to install a new "clean" environment aka a
new virtual machine.

On 18-Aug-16 02:43, Martin Panter wrote:

Martin Panter added the comment:

Michael, byref() is just a helper for passing an object’s address to a C function. Calling func(byref(b), ...) in Python is equivalent to the C code

unpack_bitfields(&b, ...)

I still think the root problem is in unpack_bitfields(). When compiled with XLC, your experiments confirm that it always returns unsigned values, and with GCC, it returns signed values.

It looks like you can detect XLC with the __IBMC__ and __xlC__ macros (not sure if there is a practical difference). So perhaps we can fix this with

#ifndef __xlC__

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue27643\>

Michael: When posting to the bug tracker, please trim irrelevant parts of old messages. It makes it hard to see if you actually added anything new.

[Me] If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test.

Proper support for “signed short” according to standard C I guess would mean if you define

struct BITS {
    signed short M: 1;
} b;
b.M = -1;

then reading back b.M gives -1. However I realized test_bitfields tests overflowing values rather than negative values. In any case, I think we have established that neither of these cases work with XLC.

I haven’t changed the signed int A–I fields yet. That was part of my patch. I was waiting for confirmation about the __xlC__ check, before committing the whole thing.

Eric: I proposed to conditionally skip the test; see disable-signed-short.patch. Since many other compilers apparently pass the test and support signed short, we should probably keep the test.

As I was not responding properly (too verbose) - I'll reread the thread for the initial patch suggestion - and hope it still fits the current 'master'.

My apologies for the long silence on this.

On 30/07/2016 02:51, Martin Panter wrote:

Martin Panter added the comment:

./python -m unittest -v ctypes.test.test_bitfields

What I am suggesting as a fix is to change line 381 from plain “int” to “signed int”, and 382 to “signed short”.

As some time has passed, I assume you mean look at this change:

struct BITS {
     signed int A: 1, B:2, C:3, D:4, E: 5, F: 6, G: 7, H: 8, I: 9;
     signed short M: 1, N: 2, O: 3, P: 4, Q: 5, R: 6, S: 7;
};

I can make a patch later if that will help. Hopefully with these changes the C compiler will use signed integer logic, matching what I believe the ctypes library uses for c_int and c_short.

Not helping the compilation in any case - same messages as before.

Default make:

michael@x071:[/data/prj/python/git/xlc-python3-3.7]xlc_r -DNDEBUG -O
-I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/>
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for M is not valid.
Type unsigned assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for N is not valid.
Type unsigned assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for O is not valid.
Type unsigned assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for P is not valid.
Type unsigned assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for Q is not valid.
Type unsigned assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for R is not valid.
Type unsigned assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for S is not valid.
Type unsigned assumed.

Adding (manually) -qbitfields=signed - almost the same.

michael@x071:[/data/prj/python/git/xlc-python3-3.7]xlc_r
-qbitfields=signed -DNDEBUG -O -I/opt/include -O2 -qmaxmem=-1 >
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for M is not valid.
Type signed assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for N is not valid.
Type signed assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for O is not valid.
Type signed assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for P is not valid.
Type signed assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for Q is not valid.
Type signed assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for R is not valid.
Type signed assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c",
line 418.5: 1506-159 (E) Bit field type specified for S is not valid.
Type signed assumed.

Using a type other than short is not right, because the Python test is explicitly trying to test c_short behaviour. If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test.
Shall reply, asap, to your suggestion for a test that knows the
difference in compilers.

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue27643\>

On 04/08/2016 10:58, Martin Panter wrote:

Martin Panter added the comment:

Okay, so to be clear, I am assuming XLC supports all of the following fields, and uses unsigned bit fields by default:

struct UNSIGNED_BITS {
unsigned int AU: 1;
int A: 1; /* Equivalent to unsigned int */
signed int AS: 1;
unsigned short MU: 1;
short M: 1; /* Equivalent to unsigned short; triggers warning */
};

and that it cannot compile the following:

struct SIGNED_BITS {
signed short MS: 1; /* Not supported */
};

Attached is what I think a patch to resolve this would look like. However it needs a line in Modules/_ctypes/_ctypes_test.c completed to detect the compiler:

#ifndef /* Something to identify XLC */

Can you figure out a way to test for XLC (but not GCC, which the AIX buildbot uses),

I'll figure this out. HOWEVER, Just want to mention that both parts of
C_Test are failing when using xlc (i.e., on "A" and "M"). Note line 40
and 51.

(Have not done the compiler detect part yet, so it still returns
something != 999)

   +33 class C_Test(unittest.TestCase):
   +34
   +35      def test_ints(self):
   +36          for i in range(512):
   +37              for name in "ABCDEFGHI":
   +38                  b = BITS()
   +39                  setattr(b, name, i)
   +40                  self.assertEqual(getattr(b, name),
func(byref(b), name.encode('ascii')))
   +41
   +42      def test_shorts(self):
   +43          b = BITS()
   +44          name = "M"
   +45          if func(byref(b), name.encode('ascii')) == 999:
   +46              self.skipTest("Compiler does not support signed
short bitfields")
   +47          for i in range(256):
   +48              for name in "MNOPQRS":
   +49                  b = BITS()
   +50                  setattr(b, name, i)
   +51                  self.assertEqual(getattr(b, name),
func(byref(b), name.encode('ascii')))
   +52

======================================================================
FAIL: test_ints (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
   File 
"/data/prj/python/git/xlc-python3-3.7/Lib/ctypes/test/test_bitfields.py", 
line 40, in test_ints
     self.assertEqual(getattr(b, name), func(byref(b), 
name.encode('ascii')))
AssertionError: -1 != 1

======================================================================
FAIL: test_shorts (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------

Traceback (most recent call last):
   File 
"/data/prj/python/git/xlc-python3-3.7/Lib/ctypes/test/test_bitfields.py", 
line 51, in test_shorts
     self.assertEqual(getattr(b, name), func(byref(b), 
name.encode('ascii')))
AssertionError: -1 != 1

and then try my patch out? Hopefully you see no more compiler warnings, test_ints() should now pass,
again - not done the changes to _ctypes_test.c yet - so maybe the
test_ints() will pass after that.
and test_shorts() should be skipped.

----------
keywords: +patch
Added file: http://bugs.python.org/file44005/disable-signed-short.patch

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue27643\>

@panter - your patch seems to be working well. Thanks.

PR 5164 submitted for review

I know it is not earth shattering - but it will permit one more test to pass. Please review the PR. Thx.

Again, the PR worked months ago. I expect it to still work.

So, I guess the question is: is there any interest. After several weeks of waiting after the last ttt - still waiting :)

well, update: the bpo-34603 merged 16 days ago has broken this PR - that has been waiting for nearly 10 months.

Unhappy camper.

And, just as a short reminder - there were earlier ¨patches (that I just copied) going back more than 2 years.

Please, some attention.

New changeset 22462da by Nick Coghlan (Michael Felt) in branch 'master':
bpo-27643 - skip test_ctypes test case with XLC compiler. (GH-5164)
22462da

Navigation Menu

Repository navigation

test_ctypes fails on AIX with xlc #71830

7 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test_ctypes fails on AIX with xlc #71830

Description

Activity

aixtools commented on Jul 28, 2016

aixtools commented on Jul 28, 2016

aixtools commented on Jul 28, 2016

vadmium commented on Jul 29, 2016

aixtools commented on Jul 29, 2016

aixtools commented on Jul 29, 2016

vadmium commented on Jul 30, 2016

aixtoolsgmailcom commented on Aug 1, 2016

vadmium commented on Aug 4, 2016

aixtoolsgmailcom commented on Aug 11, 2016

aixtoolsgmailcom commented on Aug 16, 2016

vadmium commented on Aug 18, 2016

aixtoolsgmailcom commented on Aug 21, 2016

7 remaining items

vadmium commented on Aug 24, 2016

aixtools commented on Jan 1, 2018

aixtools commented on Jan 11, 2018

aixtools commented on Jan 11, 2018

aixtools commented on Jan 12, 2018

aixtools commented on Jun 3, 2018

aixtools commented on Jul 18, 2018

aixtools commented on Oct 1, 2018

ncoghlan commented on Dec 26, 2018

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions