C extension naming doesn't take bitness into account

@malemburg

BPO	22980
Nosy	@malemburg, @warsaw, @doko42, @ncoghlan, @pitrou, @larryhastings, @tjguk, @ned-deily, @ericsnowcurrently, @berkerpeksag, @zware, @1st1, @zooba, @dstufft
Files	22980_windows.patch 22980_2.patch: Patch with version in tag abi_bitness.patch ma.diff: alternate patch ma2.diff larry.whatsnew35.ext.module.suffix.diff.1.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-09-12.16:13:19.053>
created_at = <Date 2014-12-02.14:35:25.510>
labels = ['deferred-blocker', 'type-feature']
title = "C extension naming doesn't take bitness into account"
updated_at = <Date 2017-01-14.07:25:16.112>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2017-01-14.07:25:16.112>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2015-09-12.16:13:19.053>
closer = 'larry'
components = []
creation = <Date 2014-12-02.14:35:25.510>
creator = 'pitrou'
dependencies = []
files = ['37379', '37384', '38155', '38964', '39027', '40420']
hgrepos = []
issue_num = 22980
keywords = ['patch']
message_count = 96.0
messages = ['232000', '232001', '232002', '232003', '232008', '232014', '232016', '232026', '232030', '232031', '232033', '232034', '232036', '232037', '232038', '232039', '232040', '232041', '232042', '232043', '232044', '232045', '232047', '232052', '232063', '232065', '232251', '232256', '232266', '232268', '232270', '232272', '232277', '232297', '232710', '232711', '232729', '232740', '232759', '232760', '232764', '232765', '232807', '233822', '236109', '236111', '237558', '237559', '240516', '240548', '240746', '240748', '240750', '240761', '240762', '240769', '240775', '241065', '241127', '241139', '241141', '241159', '241167', '241183', '241211', '241223', '241228', '241230', '241231', '241236', '241242', '241248', '241249', '241250', '241251', '241252', '241253', '241254', '241283', '241285', '241287', '241291', '241512', '241513', '246247', '249725', '250045', '250299', '250302', '250308', '250341', '250424', '250425', '250431', '250524', '285464']
nosy_count = 16.0
nosy_names = ['lemburg', 'barry', 'doko', 'ncoghlan', 'pitrou', 'larry', 'tim.golden', 'ned.deily', 'Arfrever', 'python-dev', 'eric.snow', 'berker.peksag', 'zach.ware', 'yselivanov', 'steve.dower', 'dstufft']
pr_nums = []
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22980'
versions = ['Python 3.5']

Currently, C extensions are named something like "_helperlib.cpython-34dm.so". This doesn't take into account the bitness of the interpreter (32- vs. 64-bit), which makes it awkward to use the same working copy with two different interpreters (you have to rebuild everything each time you switch bitnesses).

Worse, under Windows it seems ABI tags aren't even used, giving generic names such as "_helperlib.pyd". Why is that?

See also the PEP-3149.

PEP-3149 says """It is not currently clear that the facilities in this PEP are even useful for Windows""". Well, it seems I have found a use for it :-)

Ideally, we would use distutils.util.get_platform(). However, there are two cases where it relies on other modules:

the re module under CygWin
the sysconfig and _osx_support under OS X

Of course, ideally we should be able to hardcode this into the compiled CPython executable...

As a side-note, it is interesting to note that Python currently wrongly identifies 32-bit builds under 64-bit Linux:

Python 3.5.0a0 (default:64a54f0c87d7, Nov  2 2014, 17:18:13) 
[GCC 4.9.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, os, sysconfig
>>> sys.maxsize
2147483647
>>> os.uname()
posix.uname_result(sysname='Linux', nodename='fsol', release='3.16.0-25-generic', version='#33-Ubuntu SMP Tue Nov 4 12:06:54 UTC 2014', machine='x86_64')
>>> sysconfig.get_platform()
'linux-x86_64'

AFAIU, sysconfig.get_platform() (or its sibling distutils.util.get_platform()) is used for the naming of binary distributions...

The MULTIARCH variable can help at least under Linux:

>>> import sysconfig
>>> sysconfig.get_platform()
'linux-x86_64'
>>> sysconfig.get_config_var('MULTIARCH')
'i386-linux-gnu'

There is also platform.architecture(). I don't like its implementation, it relies on the external file program :-(

I'm very much in favor of adding this for .pyds on Windows.

I assume the hard part will be getting the details for Linux (doesn't bitness have to be compiled in there? For Windows it can be determined at compile-time...), but preferring an extension with the ABI tag and falling back on one without seems easy enough.

(Would/could this also work for .py files? So a 2.7/3.x or Jython/CPython/IronPython package could include tags in pure-Python code files?)

Note that there's a difference between the platform's architecture (which is what get_platform() returns) and the pointer size of the currently running Python executable.

On 64-bit Linux, it's rather rare to have an application built as 32-bit executable. On 64-bit Windows, it's rather common to have 32-bit applications running.

The best way to determine 32-bit vs. 64-bit is by using the struct module:

    # Determine bitsize used by Python (not necessarily the same as
    # the one used by the platform)
    import struct
    bits = struct.calcsize('P') * 8

This should be portable across all platforms and will always refer to the pointer size of the currently running Python executable.

Note that there's a difference between the platform's architecture

Yes, that's pointed out above.

Nothing new should be necessary for pyc files under Windows:

Python 3.4.2 |Continuum Analytics, Inc.| (default, Oct 22 2014, 11:51:45) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.implementation.cache_tag
'cpython-34'

The problem is with C extensions:

>>> import _imp
>>> _imp.extension_suffixes()
['.pyd']

Compare with Linux:

>>> import _imp
>>> _imp.extension_suffixes()
['.cpython-35dm.so', '.abi3.so', '.so']

I assume the hard part will be getting the details for Linux (doesn't bitness
have to be compiled in there? For Windows it can be determined at compile-
time...), but preferring an extension with the ABI tag and falling back on
one without seems easy enough.

Sticking to bitness should be easy (although I wonder if it would be desirable for platforms with fat binaries - Ned?). If we can go the extra mile and include platform identification all the better, of course.

I was more interested in source file resolution than bytecode caching. If Python 3.5 would prefer "spam.cpython-35.py" or "spam.cpython-3.py" over "spam.py" and Python 2 preferred "spam.py", then I can more easily separate the code that won't parse in the alternative.

Happy to be told it's unrelated and I should raise it separately, but from my POV resolving .pyd filenames looks very similar to resolving .py files.

On 02.12.2014 19:02, Antoine Pitrou wrote:

Sticking to bitness should be easy (although I wonder if it would be desirable for platforms with fat binaries - Ned?). If we can go the extra mile and include platform identification all the better, of course.

I hear the "can of worms" alarm ringing :-)

Seriously, I think that putting platform infos into the file name
is bound to cause more trouble than it tries to solve. Fat builds
leave the decision to the linker, which is a good method and avoids
the file name clashes.

I think we should only focus on platforms where fat builds are
uncommon, while at the same time you do have to support multiple
architectures, like e.g. Windows:

http://en.wikipedia.org/wiki/Fat_binary

Note that on Linux, 32-bit and 64-bit versions are typically placed
into different directory trees:

http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

so I'm not sure whether it's a real problem on Linux.

I think we should add something to the 3.5 "What's New" document about these changes and which platforms are affected. Otherwise is there anything left to do before closing?

I sure hope not.

I'm leaving this open just because we're apparently waiting on some "What's New" docs.

Here's an attempt at a What's New section for this change. I expect it's wrong! Maybe someone can fix it. Maybe it's actually better than not having one at all.

Can we maybe get a round or two of edits on this and get something in for 3.5 final?

Adding Yury since he and Elvis are working on Doc/whatsnew/3.5.rst and they might want to take a look at the latest patch.

Only thing I'd add is that the extra tag is optional (on Windows at least), and Python will happily import extensions without it. But extensions with a mismatched tag won't be loaded.

thanks for the draft!

I'm not sure how to describe this properly. The extension names are derived from https://wiki.debian.org/Multiarch/Tuples

and this again is derived from the GNU triplets/quadruplets.

there is no "cpu" and "os" part, depending on the architecture some ABI parts are either encoded in the "cpu" part or the "os" part.

So what about just enumerating the most common cases (i386-linux-gnu, x86_64-linux-gnu, arm-linux-gnueabi (still used for the old Raspberry Pi), arm-linux-gnueabihf), and then point to the "spec"? The above examples have some irregular cases, most other cases just follow the triplets.

I wouldn't mention x86_64-linux-gnux32 explicitly. Until now there are only unreleased or experimental distros.

Not sure if it is worth mentioning that this would allow distributing "fat" wheels.

New changeset 1744d65705d0 by Yury Selivanov in branch '3.5':
whatsnew/3.5: Describe changes in issue bpo-22980
https://hg.python.org/cpython/rev/1744d65705d0

New changeset cfbcb3a6a848 by Yury Selivanov in branch 'default':
Merge 3.5 (issue bpo-22980, whatsnew/3.5)
https://hg.python.org/cpython/rev/cfbcb3a6a848

Larry, Matthias, Steve, Berker - I've mentioned this issue in the whatsnew (applied Larry's patch with some modifications to address comments from Steve and Matthias). Please review.

There's no dot before the debug marker on Windows. Otherwise, looks good to me. Thanks for writing this up.

It's fixed! So it's finally closed.

New changeset 80fc40a9ae47 by Martin Panter in branch '3.5':
Issue bpo-22980: Skip a sysconfig test if _ctypes is not available.
https://hg.python.org/cpython/rev/80fc40a9ae47

Navigation Menu

Repository navigation

C extension naming doesn't take bitness into account #67169

72 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

C extension naming doesn't take bitness into account #67169

Description

Activity

pitrou commented on Dec 2, 2014

vstinner commented on Dec 2, 2014

pitrou commented on Dec 2, 2014

pitrou commented on Dec 2, 2014

pitrou commented on Dec 2, 2014

pitrou commented on Dec 2, 2014

vstinner commented on Dec 2, 2014

zooba commented on Dec 2, 2014

malemburg commented on Dec 2, 2014

pitrou commented on Dec 2, 2014

pitrou commented on Dec 2, 2014

pitrou commented on Dec 2, 2014

zooba commented on Dec 2, 2014

malemburg commented on Dec 2, 2014

72 remaining items

ned-deily commented on Jul 4, 2015

larryhastings commented on Sep 4, 2015

larryhastings commented on Sep 7, 2015

larryhastings commented on Sep 9, 2015

berkerpeksag commented on Sep 9, 2015

zooba commented on Sep 9, 2015

doko42 commented on Sep 9, 2015

python-dev commented on Sep 10, 2015

1st1 commented on Sep 10, 2015

zooba commented on Sep 11, 2015

larryhastings commented on Sep 12, 2015

python-dev commented on Jan 14, 2017

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions