rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET)

@warsaw

BPO	23968
Nosy	@warsaw, @doko42, @ncoghlan, @vstinner, @larryhastings, @ned-deily, @freakboy3742, @xdegaye, @zware, @koobs, @tpetazzoni, @rkuska, @wscullin, @yan12125
Files	platdir.diff: patch platdir2.diff issue23968-android-regression.patch: Fix a regression after changeset 5c77488830bc

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-08-29.18:04:43.978>
created_at = <Date 2015-04-15.19:04:07.449>
labels = ['expert-installation', 'release-blocker']
title = 'rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET)'
updated_at = <Date 2016-08-31.18:22:20.390>
user = 'https://github.com/doko42'

bugs.python.org fields:

activity = <Date 2016-08-31.18:22:20.390>
actor = 'yan12125'
assignee = 'none'
closed = True
closed_date = <Date 2016-08-29.18:04:43.978>
closer = 'doko'
components = ['Installation']
creation = <Date 2015-04-15.19:04:07.449>
creator = 'doko'
dependencies = []
files = ['39049', '43214', '44260']
hgrepos = []
issue_num = 23968
keywords = ['patch', 'needs review']
message_count = 32.0
messages = ['241143', '243874', '243896', '244362', '244411', '244605', '248423', '267342', '267352', '268528', '268530', '268536', '268829', '268833', '269237', '269240', '269360', '271636', '271690', '271691', '271692', '271693', '272790', '273179', '273182', '273185', '273875', '273876', '273903', '273904', '274042', '274056']
nosy_count = 17.0
nosy_names = ['barry', 'doko', 'ncoghlan', 'vstinner', 'larry', 'ned.deily', 'Arfrever', 'freakboy3742', 'xdegaye', 'python-dev', 'zach.ware', 'koobs', 'sYnfo', 'thomas-petazzoni', 'rkuska', 'wscullin', 'yan12125']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue23968'
versions = ['Python 3.6']

Currently there is only one platform directory for all linux architectures, there are two reasons to have a separate directory for each architecture:

the contents of the files in this directory differ
(most prominent case are the RTLD_* constants on various
architectures).
The _sysconfigdata.py module is installed in the platdir.
A cross build of the extensions in the standard library
is done by running the python interpreter for the build
machine with the platdir for the host machine. This
currently works for cross builds targeting a different
os, but not cross building from one linux architecture
to another.

This would then remove Lib/plat-linux and introduce some Lib/plat-$(PLATFORM_TRIPLET) directories.

ping? I would like to get this into 3.5.

+1 from me.

+1 from the Python team from Fedora, the patch looks good from downstream standpoint.

I don't know what problems I might have run into previously, but it's working now and seems fine to me. +1

What Lib/plat-$(PLATFORM_TRIPLET) directories will be introduced?

This patch would make situation worse on rarely used architectures (these without Lib/plat-$(PLATFORM_TRIPLET) distributed in repository of CPython) if patch for bug bpo-12619 is not applied.

This would likely improve life for folks with cross-compile toolchains.

attaching an updated patch. besides renaming of the platform directory, the following changes are made:

it makes the name of the internal _sysconfigdata module dependent
on the abiflags
it moves the _sysconfigdata module into the platform directory.
This address as well bpo-22724, it should address bpo-18083 as well,
provided that some multiarch tuples for solaris are defined.
it fixes regeneration of the Python modules from C headers for
cross builds.
it introduces a sys.implementation._multiarch attribute to construct
the name of the platform directory without having the
_sysconfigdata module available (which now is located in the platform
dir)

One outstanding issue probably is the generation of the platform directory in the source tree, not in the build tree (but this is existing behaviour).

This should address bpo-12619 (this patch probably needs an update),

This may be of interest for iOS and Android cross-compilation, so I've added Xavier de Gaye to the nosy list, while Russell Keith-Magee added himself.

New changeset 78d2cb7f66b6 by doko in branch 'default':

Issue bpo-23968: Rename the platform directory from plat-$(MACHDEP) to
https://hg.python.org/cpython/rev/78d2cb7f66b6

now checked in, after 3.6.0a2, to watch behaviour on different platforms. Still needs doc updates.

New changeset a062c60c9f7d by doko in branch 'default':

Issue bpo-23968: Update distutils/sysconfig.py to look for the renamed
https://hg.python.org/cpython/rev/a062c60c9f7d

New changeset e2b09c00ee24 by Ned Deily in branch 'default':
Issue bpo-23968: Fix installs of the renamed config directory for OS X
https://hg.python.org/cpython/rev/e2b09c00ee24

There is a problem with the current implementation. When running ./configure using compilers other than a current gcc, for example, when using clang or older versions of gcc, the build variable MULTIARCH may not get set because those compilers do not support --print-multiarch:

configure.ac:744
MULTIARCH=$($CC --print-multiarch 2>/dev/null)

But, a bit later in ./configure, PLATFORM_TRIPLET gets set as a result of running conftest.c. The end result is that PLATFORM_TRIPET is non-empty but MULTIARCH is empty. Still later in configure.ac (around 4477:

if test x$PLATFORM_TRIPLET = x; then
LIBPL='$(prefix)'"/lib/python${VERSION}/config-${LDVERSION}"
else
LIBPL='$(prefix)'"/lib/python${VERSION}/config-${LDVERSION}-${PLATFORM_TRIPLET}"
fi

So, the value of LIBPL is extended to include PLATFORM_TRIPLET regardless of whether MULTIARCH has a value.

This causes problems at runtime because get_makefile_filename() in Lib/sysconfig.py now uses the value of sys.implemntation's _multiarch attribute (whose value is compiled in sysmodule.c from MULTIARCH) to form the path to the config directory (the equivalent of LIBPL). In these cases, sys.implementation will have no_multiarch attribute so sysconfig looks for the config directory in the pre-3.6 location (e.g. .../lib/python3.6/config-3.6m) even though it was installed in the new location (.../lib/python3.6/config-3.6m-darwin). Fortunately, both test_sysconfig and test_distutils catch this:

======================================================================
FAIL: test_srcdir (test.test_sysconfig.TestSysConfig)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/py/dev/3x/root/uxd/lib/python3.6/test/test_sysconfig.py", line 346, in test_srcdir
    self.assertTrue(os.path.isdir(srcdir), srcdir)
AssertionError: False is not true : /py/dev/3x/root/uxd/lib/python3.6/config-3.6dm

======================================================================
FAIL: test_get_makefile_filename (test.test_sysconfig.MakefileTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/py/dev/3x/root/uxd/lib/python3.6/test/test_sysconfig.py", line 414, in test_get_makefile_filename
    self.assertTrue(os.path.isfile(makefile), makefile)
AssertionError: False is not true : /py/dev/3x/root/uxd/lib/python3.6/config-3.6dm/Makefile

======================================================================
FAIL: test_srcdir (distutils.tests.test_sysconfig.SysconfigTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/py/dev/3x/root/uxd/lib/python3.6/distutils/tests/test_sysconfig.py", line 61, in test_srcdir
    self.assertTrue(os.path.isdir(srcdir), srcdir)
AssertionError: False is not true : /py/dev/3x/root/uxd/lib/python3.6/config-3.6dm

I'm not sure what the best solution is.

It's not just broken for cross-compiling. As I noted earlier, it's broken when using other than gcc, such as is the case with current OS X systems. This needs to be fixed before release.

It is also broken for gcc, see the buildbot 'compile' output.
It seems broken for all non windows builds whatever the compiler.

New changeset ddc4bdae5e41 by Ned Deily in branch 'default':
Issue bpo-23968: Make OS X installer build script aware of renamed platform
https://hg.python.org/cpython/rev/ddc4bdae5e41

Any update on this issue?

test_srcdir() of test_sysconfig still fails on "x86 Gentoo Installed with X 3.x" buildbot. I opened the issue bpo-27798 which was closed a duplicate of this issue.

First failure of the buildbot:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Installed%20with%20X%203.x/builds/747

This build contains many changes. I don't know which one introduced the issue?

change a062c60c9f7d60db2b63af88e2d676e617398d01: Update distutils/sysconfig.py to look for the renamed _sysconfigdata module too
change 78d2cb7f66b602ec496800fbbe71e590c0f56b00: Rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET)

Ned Deily: "This needs to be fixed before release."

I agree. I use the release blocker priority as a reminder. Well, if the issue cannot be fixed, I suggest to revert changes related to this issue.

"It's not just broken for cross-compiling. As I noted earlier, it's broken when using other than gcc, such as is the case with current OS X systems."

test_sysconfig fails on "x86 Gentoo Installed with X 3.x" buildbot which uses GCC. Extract of the configure output:

checking for gcc... gcc

please could somebody test the attached patch (I'm also trying to setup a gentoo env)? The idea is to keep the multiarch and triplet macros in sync.

diff -r a7f3678f9509 configure.ac
--- a/configure.ac	Sat Aug 20 03:05:13 2016 +0200
+++ b/configure.ac	Sat Aug 20 03:54:39 2016 +0200
@@ -882,6 +882,8 @@
   if test x$PLATFORM_TRIPLET != x$MULTIARCH; then
     AC_MSG_ERROR([internal configure error for the platform triplet, please file a bug report])
   fi
+elif test x$PLATFORM_TRIPLET != x && test x$MULTIARCH = x; then
+  MULTIARCH=$PLATFORM_TRIPLET
 fi
 if test x$PLATFORM_TRIPLET = x; then
   PLATDIR=plat-$MACHDEP

I've just tried doko's patch on Arch Linux. Before this patch there are three test failures in test_distutils and test_sysconfig, just like Gentoo. After the patch, all tests in test_distutils and test_sysconfig pass.

On my PC both gcc -print-multiarch and gcc --print-multiarch print an empty string.

By the way, could you attach this patch file to roundup, so that others can download it?

Reference: Arch Linux's build script of python-hg: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-hg

New changeset 5c77488830bc by doko in branch 'default':

Issue bpo-23968, keep platform_triplet and multiarch macros in sync
https://hg.python.org/cpython/rev/5c77488830bc

now fixed.

Changeset 5c77488830bc brings a regression - Android build fails with the following message:

export H2PY; H2PY="$PYTHON_FOR_BUILD /home/yen/python3-android/src/cpython/build-target/../Tools/scripts/h2py.py"; \
cd ../Lib/plat-aarch64-linux-gnu; ./regen
+ _PYTHON_PROJECT_BASE=/home/yen/python3-android/src/cpython/build-target _PYTHON_HOST_PLATFORM=linux-aarch64 PYTHONPATH=/home/yen/python3-android/src/cpython/build-target/build/lib.linux-aarch64-3.6:../Lib:../Lib/plat-aarch64-linux-gnu python3.6 /home/yen/python3-android/src/cpython/build-target/../Tools/scripts/h2py.py -i '(u_long)' /opt/android-ndk/platforms/android-21/arch-arm64/usr/include/sys/types.h /opt/android-ndk/platforms/android-21/arch-arm64/usr/include/netinet/in.h /opt/android-ndk/platforms/android-21/arch-arm64/usr/include/dlfcn.h
./regen: line 33: _PYTHON_PROJECT_BASE=/home/yen/python3-android/src/cpython/build-target: No such file or directory
make[1]: *** [Makefile:1324: ../Lib/plat-aarch64-linux-gnu] Error 127
make[1]: Leaving directory '/home/yen/python3-android/src/cpython/build-target'
make: *** [Makefile:36: python] Error 2

The previous changeset 1902e1d79e25 builds fine. Prepending PYTHON_FOR_BUILD in configure.ac with env makes the whole build work again.

From the message seems plat-* directories are re-generated with a different name. Is it expected?

Oh didn't see the title of this issue. Here's the patch for Android.

I don't see how the suggested android fix is related, and what it is supposed to fix. It adds the env command before setting the environment variables, which should be a no-op.

However I think we should introduce different platform triplets for android. Please follow-up in http://bugs.python.org/issue27917

Well changes in this issue are not the root cause of my problem - it just exposes a broken implementation done long time ago. I'll examine more and open a new bug.

Navigation Menu

Repository navigation

rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET) #68156

6 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET) #68156

Description

Activity

doko42 commented on Apr 15, 2015

doko42 commented on May 23, 2015

ncoghlan commented on May 23, 2015

sYnfo commented on May 29, 2015

warsaw commented on May 29, 2015

Arfrever commented on Jun 1, 2015

wscullin commented on Aug 11, 2015

doko42 commented on Jun 4, 2016

ncoghlan commented on Jun 5, 2016

python-dev commented on Jun 14, 2016

doko42 commented on Jun 14, 2016

python-dev commented on Jun 14, 2016

python-dev commented on Jun 18, 2016

ned-deily commented on Jun 18, 2016

6 remaining items

ned-deily commented on Jul 30, 2016

xdegaye commented on Jul 30, 2016

python-dev commented on Aug 15, 2016

vstinner commented on Aug 20, 2016

doko42 commented on Aug 20, 2016

yan12125 commented on Aug 20, 2016

python-dev commented on Aug 29, 2016

doko42 commented on Aug 29, 2016

yan12125 commented on Aug 30, 2016

yan12125 commented on Aug 30, 2016

doko42 commented on Aug 31, 2016

yan12125 commented on Aug 31, 2016

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions