Add sys.platlibdir and configure --with-platlibdir to use /usr/lib64 on Fedora and SuSE

@malemburg

BPO	1294959
Nosy	@malemburg, @warsaw, @doko42, @jcea, @abalkin, @vstinner, @matejcik, @tarekziade, @mcepl, @merwok, @akitada, @agbuckley, @encukou, @mgorny, @grimreaper, @stratakis, @hroncok, @miss-islington, @pillarsdotnet
PRs	bpo-1294959 - better support for systems with /usr/lib64 #3698 [WIP] bpo-1294959: Better support for systems with /usr/lib64 #11755 bpo-1294959: Add --with-platlibdir to configure #18381 bpo-21016: pydoc and trace use sysconfig #18476 [3.8] bpo-21016: pydoc and trace use sysconfig (GH-18476) #18482 [3.7] bpo-21016: pydoc and trace use sysconfig (GH-18476) #18483 [WIP] bpo-1294959: setup.py uses sys.platlibdir #18917 bpo-1294959: Fix typo for new attribute platlibdir. #18960 bpo-1294959: Try to clarify the meaning of platlibdir #20332 [3.9] bpo-1294959: Try to clarify the meaning of platlibdir (GH-20332) #20497 [3.9] bpo-40854: Allow overriding sys.platlibdir via PYTHONPLATLIBDIR env-var (GH-20605) #20725
Files	python-2.3.4-lib64-regex.patch: One of the Fedora patches. python-2.4.1-lib64.patch: Another of the Fedora patches. japanese-codecs-lib64.patch: The third obviously lib64 Fedora patch. Python-2.6.2-multilib.patch: SUSE patch for the lib64 issue Python-3.3.0b2-multilib.patch: SUSE patch in python3 python-3.6.0-multilib-new.patch lib64_tests.py lib64_tests-2.py lib64_tests-3.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-03-13.16:04:36.444>
created_at = <Date 2005-09-19.03:05:28.000>
labels = ['type-bug', '3.9', 'build']
title = 'Add sys.platlibdir and configure --with-platlibdir to use /usr/lib64 on Fedora and SuSE'
updated_at = <Date 2021-08-10.16:25:34.973>
user = 'https://bugs.python.org/jafo'

bugs.python.org fields:

activity = <Date 2021-08-10.16:25:34.973>
actor = 'eric.araujo'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-13.16:04:36.444>
closer = 'vstinner'
components = ['Build']
creation = <Date 2005-09-19.03:05:28.000>
creator = 'jafo'
dependencies = []
files = ['1777', '1778', '1779', '14726', '32529', '46301', '48892', '48962', '48966']
hgrepos = ['12']
issue_num = 1294959
keywords = ['patch']
message_count = 96.0
messages = ['26314', '59310', '64592', '64593', '81099', '82929', '91559', '91769', '91781', '92541', '94941', '94982', '94986', '94989', '132424', '132503', '132505', '132506', '132511', '132535', '132537', '132538', '132540', '132542', '132544', '132546', '160482', '160671', '160691', '202341', '202343', '202354', '202359', '202365', '202366', '202367', '202368', '202378', '222643', '242608', '285126', '285222', '285556', '285737', '285738', '307231', '308555', '361844', '361847', '361849', '361850', '361851', '361853', '361854', '361882', '361888', '361890', '361893', '361894', '361897', '361899', '361900', '361901', '361902', '361903', '361907', '361909', '362324', '363288', '363441', '363728', '363811', '363816', '363894', '363896', '363900', '364024', '364104', '364105', '369323', '369326', '369327', '369352', '369354', '369360', '369365', '369368', '369467', '369744', '370238', '370242', '370248', '370704', '371008', '382964', '399339']
nosy_count = 26.0
nosy_names = ['lemburg', 'barry', 'doko', 'jafo', 'jcea', 'belopolsky', 'vstinner', 'matejcik', 'tarek', 'ivazquez', 'mcepl', 'eric.araujo', 'Arfrever', 'akitada', 'andybuckley', 'petr.viktorin', 'catalin.iacob', 'mgorny', 'python-dev', 'piotr.dobrogost', 'eitan.adler', 'cstratak', 'hroncok', 'miss-islington', 'pillarsdotnet', 'carlos.velasco']
pr_nums = ['3698', '11755', '18381', '18476', '18482', '18483', '18917', '18960', '20332', '20497', '20725']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1294959'
versions = ['Python 3.9']

Adding Tarek to the ticket.

Please note another aspect of this problem will bite all Python developers on Ubuntu 11.04. With the introduction of multiarch, not all stdlib Python extension modules can be built out of the box, as seen here:

https://bugs.launchpad.net/ubuntu/+source/db4.8/+bug/738213/comments/13

Ubuntu's source package was hacked to make things work, by calling out to dpkg-architecture and adding the resulting directories to library_dirs and include_dirs search paths. That patch would obviously have to be modified at the very least to be robust on non-Debian/Ubuntu platforms.

I'm not sure what the right solution is for upstream.

Here's a fix that works for me on Ubuntu 11.04.

I should note that I'd love to backport this to Python 3.2, 3.1, 2.7 and 2.6 since none of them can build entirely now on multiarch systems. Since it only affects search order in the build process, one could argue that it's not a new feature :).

Barry: does it allow to install Python into /usr/lib/whateverarch, or is it just a partial fix for something slightly unrelated to this issue?

A proper fix is to introduce sys.libdir, which would be controllable by --libdir=${value} option of configure. If --libdir=${value} is not passed, then sys.libdir would default to sys.prefix + "/lib".
sysconfig, distutils etc. would have to use sys.libdir.

On Mar 29, 2011, at 07:28 PM, Antoine Pitrou wrote:

Antoine Pitrou <pitrou@free.fr> added the comment:

Barry: does it allow to install Python into /usr/lib/whateverarch, or is it
just a partial fix for something slightly unrelated to this issue?

Antoine, you're right that the problem and fix I'm talking about is probably
different than the original bug report. I really should create a new issue,
since mine isn't about building multiarch for Python, but instead just being
able to build Python on a multiarch system. My problem is thankfully much
simpler.

I retract my patch for this bug because the issue described here is actually different than the one I want to fix. See bpo-11715 for the problem of building Python on multiarch Debian and Ubuntu (e.g. Ubuntu 11.04).

On Mar 29, 2011, at 07:38 PM, Arfrever Frehtes Taifersar Arahesis wrote:

A proper fix is to introduce sys.libdir, which would be controllable by
--libdir=${value} option of configure. If --libdir=${value} is not passed,
then sys.libdir would default to sys.prefix + "/lib". sysconfig, distutils
etc. would have to use sys.libdir.

Please note that I'm not interested (right now <wink>) in building Python
multiarch, but building Python *on* multiarch. So I think simply adding the
right search paths to setup.py is the right way to go, and I've opened a
separate issue for it.

heh, that's easy, just add the multiarch id to the extension name ;-)

On 29.03.2011 21:28, Antoine Pitrou wrote:

> Barry: does it allow to install Python into /usr/lib/whateverarch,
no, it looks for headers and libraries in more directories. But really, this
whole testing for paths is wrong. Just use the compiler to search for headers
and libraries, no need to check these on your own.

> or is it just a partial fix for something slightly unrelated to this issue?

IMO, unrelated to the original report.

On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:

no, it looks for headers and libraries in more directories. But really, this
whole testing for paths is wrong. Just use the compiler to search for headers
and libraries, no need to check these on your own.

You're probably right about that, but reimplementing Python's build system is
out of scope for right now. ;) For one thing, doing so wouldn't allow me to
backport to older Pythons, which I really want to do.

On Mar 29, 2011, at 10:11 PM, Matthias Klose wrote:

heh, that's easy, just add the multiarch id to the extension name ;-)

Clever! :)

I currently think that sys.libdir should be only basename of libdir (e.g. "lib" or "lib64") to allow to easily use it with something else than sys.prefix.

On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:

no, it looks for headers and libraries in more directories. But really, this
whole testing for paths is wrong. Just use the compiler to search for headers
and libraries, no need to check these on your own.

Do all compilers provide this info, including Windows ones? If so, that would be a nice feature for distutils2.

Arfrever Frehtes Taifersar Arahesis (Arfrever) on 2012-05-12 19:00

I currently think that sys.libdir should be only basename of libdir (e.g.
"lib" or "lib64") to allow to easily use it with something else than sys.prefix.

With the new sysconfig module I don’t think we need to clutter sys with another attribute.

Éric Araujo wrote:

Éric Araujo <merwok@netwok.org> added the comment:

On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:
> no, it looks for headers and libraries in more directories. But really, this
> whole testing for paths is wrong. Just use the compiler to search for headers
> and libraries, no need to check these on your own.

Do all compilers provide this info, including Windows ones? If so, that would be a nice feature for distutils2.

This only works for a handful of system library paths, not the extra
ones that you may need to search for local installations of
libraries and which you have to inform the compiler about :-)

Many gcc installations, for example, don't include the /usr/local
or /opt/local dir trees in the search. On Windows, you have to
run the correct vc*.bat files to have the paths setup and optional
software rarely adds the correct paths to LIB and INCLUDE.

The compiler also won't help with the problem Sean originally
pointed to: building software on systems that can run both
32-bit and 64-bit and finding the right set of libs to
link at.

Another problem is finding the paths to the right version of a
library (both include files and corresponding libraries).

While it would be great to have a system tool take care of setting
things up correctly, I don't know of any such tool, so searching
paths and inspecting files using REs appears to be the only way
to build a general purpose detection scheme.

mxSetup.py (included in egenix-mx-base) uses such a scheme, distutils
has one too.

Is this issue still relevant?

Yes. We still have a patch for two things:

fix setup.py script to look for libraries in correct (lib64) prefixes, and
fix values returned from sysconfig, to reflect that python resides in lib64

"$prefix/lib" is hardcoded in many places. Lib64 is probably not going away anytime soon, so it would be nice if this was solved once and for all ;)
The good thing is that with sysconfig, we don't have to do much beyond teaching sysconfig about the right values.

To reiterate, our current solution is to introduce "sys.lib" (and "sys.arch", but that is never used anymore) that is either "lib" or "lib64", and use this in place of the string "lib" wherever appropriate. We find the value for sys.lib through configure magic.

the patch in msg202343 is wrong, hardcoding lib64 on Debian/Ubuntu. At least the configure check should check for lib64 as a directory and not a symlink, and only then default to lib64.

two other issues with the patch:

I would like to see any new OS-dependent locations in the sysconfig module, not the sys module.
Please don't depend on uname for the autoconf check, but on the gnu host triplet.
Please don't add another `arch' attribute to sys. We already have enough of these in sysconfig.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dne 7.11.2013 16:56, Matthias Klose napsal(a):

Matthias Klose added the comment:

the patch in msg202343 is wrong, hardcoding lib64 on Debian/Ubuntu.

This patch is provided for reference only - it works for us at SUSE. I'll be
happy to spend some time improving it for general usage, if this has any
chance of being commited.

At least the configure check should check for lib64 as a directory and not
a symlink, and only then default to lib64.

Maybe this should be detected differently altogether. Perhaps by working with
LIBDIR, which is known to configure?

two other issues with the patch:

I would like to see any new OS-dependent locations in the sysconfig
module, not the sys module.

how would you propose to put the value into sysconfig in the first place? It
seems to rely heavily on existing attributes from sys.

Please don't depend on uname for the autoconf check, but on the gnu host
triplet.

Please don't add another `arch' attribute to sys. We already have enough
of these in sysconfig.

We don't use this one anymore, and i'm not entirely sure that we ever did. I
am happy to drop it.

----------

_______________________________________ Python tracker
<report@bugs.python.org> <http://bugs.python.org/issue1294959\>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSe7pdAAoJEIskb84DCy7LvwcP/2n74K2XDsRu7K6OV9S4SzDa
v7vpDVhAgTBQlHglY+wavUQU2WLBlGyVEk2xHDV8WdI4zU7rAbn7XAW5URxznctq
t/Ptvt0IsDAqONrF8ezg8/eTUkcP3nV2Hk90RNe0gliDH6uc0wekKUZzVaTObO1L
3vM8XfEtTQstmK1VxQVpYolUPZm8n7Fe8NEPA6A8bu8CU736cg+wWdbDrr6Mjowo
OuO4b56J1P3BIQkBcOLe3mH20Bv8O03P9iNADwYHUOayvgthFWCmoDzh0Y1dQa9/
ynT+G9BuYyXOli6Yr15W0L8OFU+nwxByK81lEClz6UonCvoStnEWnXIN3JYW15Yb
rNwb5HKNmKB16yx/RuV3WCvlKbg6ziMlfWGW6qTA1g0P0ivU+sRVQXv5gI8NHcQ9
/4jmaUh7Dr1T4KHujI57Z99kLQHvSlHEM3v4aT96IZNaPghkA+e7TjhMdHmvYtQz
YoY75FIy0xVStdXzw8zbM2LVlKp8vxncrjYbuzJYgG1jiYwmF6gDoztXRVP/zwei
PypIiui4QaQc32V5dCwpQYpAvpgCVHm6sGSO0HbLWSUK71M8f1YU2BPwvglHb6jh
N9tmYfmahvQSyIFOZdb4C6HLlzEezENdYYIf7oMW07z8SOOLU+8eKP13dp6NWINP
HCSU34LLYTfwnQ+27aMk
=raBO
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Nov 07, 2013, at 04:05 PM, jan matejek wrote:

> - I would like to see any new OS-dependent locations in the sysconfig
> module, not the sys module.

how would you propose to put the value into sysconfig in the first place? It
seems to rely heavily on existing attributes from sys.

Actually, I think it mostly parses these values from the installed Makefile.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.15 (GNU/Linux)

iQIcBAEBCAAGBQJSe8T6AAoJEBJutWOnSwa/6oAQAJM3IksGydXk9CEp7rbSrHea
DvYUccmyon1xrmu5RjWnz6ZbJdWlFwx8ouFpbzmZvfAF8E2m0HliNzW+/w28sik5
F37p5/7hScQ2x/AcmdnVrzDotMkvcvMILsCDIhSy/nPIWkI4hAuRGLdPJmgiE3HE
b3hjQDCgsVDrp8arJioepx6xWSMpE1dQ/KsL6DDuNl3rUNVR1E1jTYEqY12+fOjc
Mcgz6lVndyZreY958iQE/UR7MKoW5lcLnin44PtMYrizwWv82kgwhkqU5tubnTBD
m9tD16uCz7xvATM5XI8nmXeLcLMSMfUdaG+4ny//cIjDKYIC6XXoZvCgE7iSEws1
pEIQCmrs6mpk6d83Yz/XDXc4OqjqC+tUPY1TGNqAI/nm416uoKUuO/f1eU397EV+
2RqJqev8Ho8Sgk7skFJGwcCfTO4yDR40+0wm3u2BiM9bTcnGiJaC7z2TAp9eb4Qs
jo/cmYi3BbqPu9Xx3P4oX11NHmjTPBUcZjqsJa8w8q3lf9r5haE5EqlLaNgDnGtL
efu7OMom2yQHXdwIJ2efmefjoby812uNFSbTiMvDxZTVCCUyBczBT/Q7gu/4S9Ks
Mv3oY1bkq6qAXKKOwzKoblzHJ6VW+A3Rn15Lh6Tb2kj1pTbdS9fFJASWr6CprjWi
XbCdez4dMhd+PGwgxHs6
=/r/D
-----END PGP SIGNATURE-----

On Nov 07, 2013, at 03:40 PM, jan matejek wrote:

To reiterate, our current solution is to introduce "sys.lib" (and "sys.arch",
but that is never used anymore) that is either "lib" or "lib64", and use this
in place of the string "lib" wherever appropriate. We find the value for
sys.lib through configure magic.

PEP-421 added sys.implementation, which contains provisions for
implementation-specific additions. So a better place to put these
non-standard values is sys.implementation._lib and sys.implementation._arch,
either instead of or in addition to sysconfig variables.

I disagree about sys.implementation. It's useless and wrong for cross builds. Please use sysconfig instead. What sysconfig is maybe missing is a set of variables which you can rely on.

configure.ac should accept an option, which allows to set any custom libdir.

Examples (architecture: libdir) in Gentoo:
x32: libx32
mips o32: lib
mips n32: lib32
mips n64: lib64

On Nov 07, 2013, at 04:56 PM, Matthias Klose wrote:

I disagree about sys.implementation. It's useless and wrong for cross builds.
Please use sysconfig instead. What sysconfig is maybe missing is a set of
variables which you can rely on.

Agreed that sysconfig is a better place for more general values. My point was
that if OpenSUSE wants to carry deltas that are specific only to its platform,
then sys.implementation._<whatever> is the standard place to put it.

Is there agreement on what needs doing here? I'd like to see this into 3.5 before it reaches its 10th birthday :)

Also see bpo-15631

at this again, when porting SUSE patches to 3.6.0 :) ( :( )

Last time there was a discussion, Barry suggested using sysconfig variables to find the proper libdir. Trouble is, to fill out the variables, sysconfig itself uses two sources:
a) compiled-in information from the binary, and
b) Makefile items
Makefile seems an obvious location for storing the libdir info, except, well, to FIND the makefile in the first place, we are using sysconfig variables.... you see where this is heading.

So, given that sysconfig is the correct Primary Source for libdir info, we have three options to get it there:

compile it into the binary. sys.implementation.platlibdir seems as good as any -- i'm not completely convinced that this is a property of the _implementation_, but, well.
modify sysconfig.py at build time
guess it from the environment

And options (2) and (3) seem wrong, so i'm going with (1) for now

The downstream patch we currently use in Fedora [0].

[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/plain/00102-lib64.patch

Attached is a patch that I'd like to propose for inclusion.

It introduces a new configure option "--with-custom-platlibdir=", which defaults to basename $libdir. This is converted to makefile variable "platlibdir", which is used in getpath.c to generate value of the lib_python variable.

sysconfig and distutils.sysconfig retrieve the variable from makefile (i learned about existence of _sysconfigdata so that is where the variable now lives) to correctly substitute in posix_prefix (and unix_prefix for distutils install) scheme.

Sysconfig is then used in pydoc and trace module, instead of locally calculating paths. Perhaps distutils.sysconfig should also use more of sysconfig instead of duplicating the functionality?

As it stands, the python stdlib is installed into /usr/$platlibdir, because I changed SCRIPTDIR to $libdir instead of $prefix/lib. Maybe this should be also separately configurable?
In any case, third-party modules get installed either into /usr/lib or /usr/$platlibdir depending on whether they are arch-independent or not. Hence the final modification in site.py that adds /usr/lib to search path on systems where $platlibdir != "lib".

This caused test_sysconfig to fail because it checks that posix_user and posix_prefix schemes generate "similar" paths. I have left the posix_user scheme alone, so all its paths are based on "lib", where the posix_prefix scheme uses $platlibdir. Maybe posix_user scheme should be modified as well?
Anyway, for now, I have accounted for the changes in the test.

What now?
(please don't make me write a PEP please don't make me write a PEP please don't make......)

I don't think we need a PEP for this, however I would like to name the new macro libsubdir, or sublibdir, to make it clear it's just the libdir component. Or make the value absolute, like it already is for libdir ...

On Jan 18, 2017, at 03:57 PM, Matthias Klose wrote:

I don't think we need a PEP for this

Correct. Generally we don't need PEPs for build system changes.

Pinging here. Is there some way to push the issue forward?

the patch looks ok. was it tested on a Debian or Ubuntu system to produce the same layout with this patch and without the new configure option?

I closed bpo-858809 as a duplicate of this issue.

I marked bpo-15631 as a duplicate of this issue.

I marked bpo-34032 as a duplicate of this issue. It has an open PR written by Matěj Cepl (OpenSUSE): PR 8068. It looks similar to PR 3698.

See also bpo-34058 "Default Python 3.7 install broken on openSUSE Leap 42.3: $PYTHONHOME/lib64/python3.7/lib-dynload/ not linked to $PYTHONHOME/lib/python3.7/lib-dynload/": OpenSUSE and Centos are mentioned as being impacted.

See also bpo-14791: "setup.py only adds /prefix/lib, not /prefix/lib64".

I marked bpo-18092 ("Python 2.7.5 installation broken on OpenSuse 12.2") as a duplicate of this issue.

See also bpo-21016: "trace: $prefix and $exec_prefix improperly replaced on Fedora".

New changeset 4fac7ed by Victor Stinner in branch 'master':
bpo-21016: pydoc and trace use sysconfig (GH-18476)
4fac7ed

New changeset ca133e5 by Miss Islington (bot) in branch '3.7':
bpo-21016: pydoc and trace use sysconfig (GH-18476)
ca133e5

New changeset ac6f4d2 by Miss Islington (bot) in branch '3.8':
bpo-21016: pydoc and trace use sysconfig (GH-18476)
ac6f4d2

Summary:

This issue is mostly about pushing a 16-years old patch from Fedora and SuSE to Python upstream. Other Linux distributions use different solutions to support multiple architectures in the same filesystem. For example, Debian and Ubuntu uses "Multiarch" which continues to use /usr/lib/ for 32-bit and 64-bit x86 architecture (x86 and x86-64).
I pushed the non controversial part of PR 3698: fix pydoc and trace modules to use sysconfig to get the stdlib path
CI tests pass on PR 18381.
Matthias Klose asked to not add another sys attribute, but Jan Matějek explained that the option value is needed to import the sysconfig module. Moreover, the option value is needed in the site module and we are trying to avoid "import sysconfig" in site since it would slowdown Python startup time.
I chose "./configure --with-python-libdir" and "sys.python_libdir" names. PR 3698 uses "./configure --with-custom-platsubdir" and sysconfig.get_config_var('platsubdir') names. Other proposed names: sys.lib, os.lib_dir. It was proposed to drop "custom" from "./configure --with-custom-platsubdir" option name.

Python use many paths. The sysconfig module is a good starting point to discover them all in the _INSTALL_SCHEMES directories. PR 18381 changes the following paths:

posix_prefix: stdlib, platstdlib, platlib
posix_user: stdlib, platstdlib, platlib

Examples on Fedora:

pure Python: /usr/lib64/python3.7/os.py
C extension: /usr/lib64/python3.7/lib-dynload/_multiprocessing.cpython-37m-x86_64-linux-gnu.so

Examples on SuSE:

pure Python: /usr/lib64/python3.7/os.py

Ubuntu 16.04:

pure Python: /usr/lib/python3.5/os.py
C extension: /usr/lib/python3.5/lib-dynload/_multiprocessing.cpython-35m-x86_64-linux-gnu.so

Default with prefix=/usr:

posix_prefix:
- stdlib: '/usr/lib/python3.9'
- platstdlib: '/usr/lib/python3.9'
- platlib: '/usr/lib/python3.9/site-packages'
posix_user:
- stdlib: '~/.local/lib/python3.9'
- platstdlib: '~/.local/lib/python3.9'
- platlib: '~/.local/lib/python3.9/site-packages'

Customized with prefix=/usr and python_libdir=lib64, PR 18381:

posix_prefix:
- stdlib: '/usr/lib64/python3.9'
- platstdlib: '/usr/lib64/python3.9'
- platlib: '/usr/lib64/python3.9/site-packages'
posix_user:
- stdlib: '~/.local/lib64/python3.9'
- platstdlib: '~/.local/lib64/python3.9'
- platlib: '~/.local/lib64/python3.9/site-packages'

posix_user:

stdlib: '~/.local/lib64/python3.9'

platstdlib: '~/.local/lib64/python3.9'

platlib: '~/.local/lib64/python3.9/site-packages'

I am still not sure about the this one. I know that I have mentioned a possibility of multiplatform $HOME on NFS, but it seems too far fetched idea to break the current not-completely-standard.

Also note that the extension modules have archful suffix, so they will not collide in $HOME, unless they are installed as name.so.

posix_user:

stdlib: '~/.local/lib64/python3.9'

platstdlib: '~/.local/lib64/python3.9'

platlib: '~/.local/lib64/python3.9/site-packages'

Honestly, I have no idea how posix_user is used.

~/.local/lib64/ directory doesn't exist on my work machine, whereas I'm installed various Python modules and applications.

In practice, ~/.local/lib/ is used on Fedora, even if posix_user scheme uses ~/.local/lib64/. Examples:

pure Python: ~/.local/lib/python3.7/site-packages/pip/main.py
C extension: ~/.local/lib/python3.7/site-packages/typed_ast/_ast3.cpython-37m-x86_64-linux-gnu.so

So changing posix_user should have no impact on end users.

I also tested a legacy "python3 setup.py install" to install a C extension, it also lands into ~/.local/lib:

~/.local/lib/python3.7/site-packages/test_cext-0.0.0-py3.7-linux-x86_64.egg/test_cext.cpython-37m-x86_64-linux-gnu.so

The setup.py uses setuptools.

Miro:

Also note that the extension modules have archful suffix, so they will not collide in $HOME, unless they are installed as name.so.

Yeah, it seems like pip install adopted Debian multiarch naming: "x86_64-linux-gnu" triplet in "_ast3.cpython-37m-x86_64-linux-gnu.so" filename.

> Also note that the extension modules have archful suffix, so they will not collide in $HOME, unless they are installed as name.so.

Yeah, it seems like pip install adopted Debian multiarch naming: "x86_64-linux-gnu" triplet in "_ast3.cpython-37m-x86_64-linux-gnu.so" filename.

For reference: bpo-22980

SuSE currently uses this patch for lib64 in the Python 3 package:
https://build.opensuse.org/package/view_file/devel:languages:python:Factory/python3/F00102-lib64.patch?expand=1

Ooops, SuSE has 2 patches.

SuSE currently uses these patches for lib64 in the Python 3 package:

(1) https://build.opensuse.org/package/view_file/devel:languages:python:Factory/python3/F00102-lib64.patch?expand=1

(2) https://build.opensuse.org/package/view_file/devel:languages:python:Factory/python3/SUSE-FEDORA-multilib.patch?expand=1

lib64_tests.py: test suite hardcoded for Fedora, SuSE and PR 18381 with --with-python-libdir=lib64. I wrote it to help me to understand differences between Fedora, SuSE and PR 18381.

On Fedora, the test fails because we have an additional patch on distutils to install packages in /usr/local instead of /usr when using pip:

======================================================================
FAIL: test_distutils_install (main.Lib64Tests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "lib64_tests.py", line 81, in test_distutils_install
    self.assertEqual(cmd.install_platlib, PLATLIB)
AssertionError: '/usr/local/lib64/python3.7/site-packages' != '/usr/lib64/python3.7/site-packages'
- /usr/local/lib64/python3.7/site-packages
?

+ /usr/lib64/python3.7/site-packages

Links on Debian Multiarch:

Why "sys.python_libdir"? Isn't that too public? A lot of similar path-related info is in sysconfig; why shouldn't it be there?
How does "sys.python_libdir" work on other platforms?

Why "sys.python_libdir"? Isn't that too public? A lot of similar path-related info is in sysconfig; why shouldn't it be there?

I replied in a previous comment:

"Matthias Klose asked to not add another sys attribute, but Jan Matějek explained that the option value is needed to import the sysconfig module. Moreover, the option value is needed in the site module and we are trying to avoid "import sysconfig" in site since it would slowdown Python startup time."

How does "sys.python_libdir" work on other platforms?

On Windows, the variable is equal to "lib". It's equal to "lib" on all platforms, except if you use configure to override its value.

Does it reply to your question?

See also bpo-24871: "freeze.py doesn't work on x86_64 Linux out of the box".

On 64-bit Linux freeze.py uses lib instead of lib64 when constructing
path to Makefile etc. Using sysconfig fixes this issue.

lib64_tests-2.py: Updated test for PR 18381. It now uses sys.platlibdir and it tests also Makefile LIBPL variable.

lib64_tests-3.py: Updated script to handle different ABI flags (release vs debug mode).

New changeset 8510f43 by Victor Stinner in branch 'master':
bpo-1294959: Add sys.platlibdir attribute (GH-18381)
8510f43

I chose to exclude setup.py changes from my commit 8510f43, whereas the Fedora downstream patch has a few changes:
https://src.fedoraproject.org/rpms/python39/blob/master/f/00102-lib64.patch

See also bpo-14791: "setup.py only adds /prefix/lib, not /prefix/lib64".

diff --git a/setup.py b/setup.py
index 51e67fe4a5..bafa0bf99a 100644
--- a/setup.py
+++ b/setup.py
@@ -649,7 +649,7 @@ class PyBuildExt(build_ext):
         # directories (i.e. '.' and 'Include') must be first.  See issue
         # 10520.
         if not CROSS_COMPILING:
-            add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib')
+            add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib64')
             add_dir_to_list(self.compiler.include_dirs, '/usr/local/include')
         # only change this for cross builds for 3.3, issues on Mageia
         if CROSS_COMPILING:
@@ -955,11 +955,11 @@ class PyBuildExt(build_ext):
             elif curses_library:
                 readline_libs.append(curses_library)
             elif self.compiler.find_library_file(self.lib_dirs +
-                                                     ['/usr/lib/termcap'],
+                                                     ['/usr/lib64/termcap'],
                                                      'termcap'):
                 readline_libs.append('termcap')
             self.add(Extension('readline', ['readline.c'],
-                               library_dirs=['/usr/lib/termcap'],
+                               library_dirs=['/usr/lib64/termcap'],
                                extra_link_args=readline_extra_link_args,
                                libraries=readline_libs))
         else:

Another related issue is bpo-34058: "Default Python 3.7 install broken on openSUSE Leap 42.3: $PYTHONHOME/lib64/python3.7/lib-dynload/ not linked to $PYTHONHOME/lib/python3.7/lib-dynload/".

Another related issue is bpo-34058: "Default Python 3.7 install broken on openSUSE Leap 42.3: $PYTHONHOME/lib64/python3.7/lib-dynload/ not linked to $PYTHONHOME/lib/python3.7/lib-dynload/".

I marked bpo-34058 as a duplicate of this issue.

New changeset 3c29675 by Raúl Cumplido in branch 'master':
bpo-1294959: Fix typo for new attribute platlibdir. (GH-18960)
3c29675

Miro Hrončok ran rpmdiff to compare the Fedora with the old downstream patches and with the new upstream commit 8510f43: compare the Python package with and without setup.py changes.

https://src.fedoraproject.org/rpms/python39/pull-request/31#comment-39392

In short, there is no difference. Moreover, Miro Hrončok, Matej Cepl and me don't know the rationale of these setup.py changes.

I propose to drop them and wait until someone complains. If someone complains, we would know the rationale and so have a good reason to have these setup.py changes.

For readline: Fedora libreadline is already linked to libtinfo and so setup.py doesn't go up to self.compiler.find_library_file(self.lib_dirs + ['/usr/lib/termcap'], 'termcap') code path. This code is deadcode on Fedora.

For /usr/local/lib vs /usr/local/lib64: I'm not sure why /usr/local/lib is used in the first place. Fedora installs libraries in /usr, not in /usr/local. Maybe it's there for users who install dependencies without Fedora: by installed them manually in /usr/local. Again, since it's unclear how /usr/local is supposed to be used, I prefer to leave it unchanged.

I closed the PR 18917.

Morevoer, if it becomes an issue in Fedora, we can start with a downstream patch again, to see how it goes, before going directly upstream.

I close this very old issue (15 years old!). Thank you *very much* to everybody who was involved in helping to get it fixed!

If I forgot someone, comment this issue or open a new one.

There is one last minor issue, but there is already a dedicated issue: bpo-24871 "freeze.py doesn't work on x86_64 Linux out of the box".

Does this mean that platforms using /usr/lib64 for shared libraries are now forced to install Python into /usr/lib64/python*?

Does this mean that platforms using /usr/lib64 for shared libraries are now forced to install Python into /usr/lib64/python*?

Not at all. This means that it is possible to do so. It remains optional.

Not at all. This means that it is possible to do so. It remains optional.

...but then sys.platlibdir is going to incorrectly list 'lib', isn't it? According to https://docs.python.org/3.9/library/sys.html#sys.platlibdir it's used 'to build the path of platform-specific dynamic libraries and the path of the standard library'. This means that people relying on it for the former will get 32-bit libraries instead.

If you don't use the new --with-platlibdir configuration option, Python 3.9 behaves exactly as Python 3.8: there is zero change.

Python 3.9 only builds a different sys.path if sys.platlibdir is different than "lib".

Can we clarify the wording to clearly indicate it's to be used only for Python modules/extensions and not system dynamic libs?

Can we clarify the wording to clearly indicate it's to be used only for Python modules/extensions and not system dynamic libs?

Sure. Can you please propose a different wording? English is not my first language.

I created bpo-40684: make install doesn't respect configure --with-platlibdir=lib64.

Can you please propose a different wording? English is not my first language.

Mine neither but I'll try. How about:

'Name of the platform-specific library directory. It is used to build the path of the standard library and C extension modules of the standard library.'

Can you propose a PR with your doc enhancement? I suggest to start with updating https://docs.python.org/dev/library/sys.html#sys.platlibdir documentation.

Ok, it seems that I misunderstood it at first. Judging by the code, it also affects extensions installed into site-packages, so I've tried to make that clear and add one more example to the bullet list.

New changeset 242d956 by Michał Górny in branch 'master':
bpo-1294959: Try to clarify the meaning of platlibdir (GH-20332)
242d956

New changeset a5936ad by Miss Islington (bot) in branch '3.9':
bpo-1294959: Try to clarify the meaning of platlibdir (GH-20332)
a5936ad

Thank you!

Note: https://docs.python.org/3.9/install/index.html probably needs some update wrt this change.

Oh, it seems like one of the changes merged in this issue broke the following use case if platlibdir is different than "lib":

export PYTHONHOME=<Prefix>
export PYTHONPATH=<Prefix>/lib/python3.9
python3.9

See bpo-40854 "Allow overriding sys.platlibdir: add PYTHONPLATLIBDIR env var".

I marked bpo-40505 as a duplicate of this issue.

Follow-up: bpo-44860 is removing platlibdir from posix_user scheme

Navigation Menu

Repository navigation

Add sys.platlibdir and configure --with-platlibdir to use /usr/lib64 on Fedora and SuSE #42382

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add sys.platlibdir and configure --with-platlibdir to use /usr/lib64 on Fedora and SuSE #42382

Description

Activity

malemburg commented on Nov 6, 2009

warsaw commented on Mar 28, 2011

warsaw commented on Mar 29, 2011

warsaw commented on Mar 29, 2011

pitrou commented on Mar 29, 2011

Arfrever commented on Mar 29, 2011

warsaw commented on Mar 29, 2011

warsaw commented on Mar 29, 2011

warsaw commented on Mar 29, 2011

doko42 commented on Mar 29, 2011

doko42 commented on Mar 29, 2011

warsaw commented on Mar 29, 2011

warsaw commented on Mar 29, 2011

Arfrever commented on May 12, 2012

merwok commented on May 14, 2012

malemburg commented on May 15, 2012

warsaw commented on Nov 7, 2013

matejcik commented on Nov 7, 2013

doko42 commented on Nov 7, 2013

matejcik commented on Nov 7, 2013

warsaw commented on Nov 7, 2013

warsaw commented on Nov 7, 2013

doko42 commented on Nov 7, 2013

Arfrever commented on Nov 7, 2013

warsaw commented on Nov 7, 2013

BreamoreBoy commented on Jul 9, 2014

BreamoreBoy commented on May 5, 2015

matejcik commented on Jan 10, 2017

stratakis commented on Jan 11, 2017

matejcik commented on Jan 16, 2017

doko42 commented on Jan 18, 2017

warsaw commented on Jan 18, 2017

stratakis commented on Nov 29, 2017

doko42 commented on Dec 18, 2017

vstinner commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

miss-islington commented on Feb 12, 2020

miss-islington commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

mcepl commented on Feb 12, 2020

hroncok commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

hroncok commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

vstinner commented on Feb 12, 2020

encukou commented on Feb 20, 2020

vstinner commented on Mar 3, 2020

vstinner commented on Mar 5, 2020

vstinner commented on Mar 9, 2020

vstinner commented on Mar 10, 2020

vstinner commented on Mar 10, 2020

vstinner commented on Mar 11, 2020

vstinner commented on Mar 11, 2020

vstinner commented on Mar 11, 2020

miss-islington commented on Mar 12, 2020

vstinner commented on Mar 13, 2020

vstinner commented on Mar 13, 2020

mgorny commented on May 19, 2020

hroncok commented on May 19, 2020

mgorny commented on May 19, 2020

vstinner commented on May 19, 2020

mgorny commented on May 19, 2020

vstinner commented on May 19, 2020

vstinner commented on May 19, 2020

mgorny commented on May 19, 2020

vstinner commented on May 20, 2020

mgorny commented on May 23, 2020

vstinner commented on May 28, 2020