Repository navigation
sysconfig's posix_user scheme has different platlib value to distutils's unix_user #89023
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-09-07.11:46:12.337>
created_at = <Date 2021-08-07.14:29:18.323>
labels = ['release-blocker', 'library', '3.9', '3.10', '3.11']
title = "sysconfig's posix_user scheme has different platlib value to distutils's unix_user"
updated_at = <Date 2021-10-04.19:18:43.703>
user = 'https://github.com/uranusjr'bugs.python.org fields:
activity = <Date 2021-10-04.19:18:43.703>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-07.11:46:12.337>
closer = 'pablogsal'
components = ['Distutils']
creation = <Date 2021-08-07.14:29:18.323>
creator = 'uranusjr'
dependencies = []
files = []
hgrepos = []
issue_num = 44860
keywords = ['patch']
message_count = 18.0
messages = ['399186', '399315', '399316', '399318', '399326', '401217', '401218', '401221', '401241', '401242', '401250', '401264', '401266', '401373', '401448', '401452', '401455', '403172']
nosy_count = 10.0
nosy_names = ['vstinner', 'eric.araujo', 'petr.viktorin', 'lukasz.langa', 'dstufft', 'uranusjr', 'hroncok', 'frenzy', 'pablogsal', 'miss-islington']
pr_nums = ['27655', '28197', '28235', '28251']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44860'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
Activity
uranusjr commentedon Aug 7, 2021
On POSIX, the user scheme has a different 'platlib' location between distutils and sysconfig, dispite the comment claiming they should be the same.
This can be reproduced on Fedora 34's stock Python 3.9:
$ docker run -it --rm -h=p fedora:34 bash ... [root@p /]# yum install python3 -y ... [root@p /]# type python3 python3 is hashed (/usr/bin/python3) [root@p /]# python3 -V Python 3.9.6 [root@p /]# python3.9 -q >>> from distutils.command.install import install >>> from distutils.dist import Distribution >>> c = install(Distribution()) >>> c.user = True >>> c.finalize_options() >>> c.install_platlib '/root/.local/lib/python3.9/site-packages' >>> import sysconfig >>> sysconfig.get_path('platlib', 'posix_user') '/root/.local/lib64/python3.9/site-packages'This issue was introduced by the sys.platlibdir value, and its usage in distutils and sysconfig. sysconfig sets posix_user's lib paths like this:
cpython/Lib/sysconfig.py
Lines 100 to 108 in a40675c
But distutils naively sets both to the same value that does not account for platlibdir:
cpython/Lib/distutils/command/install.py
Lines 68 to 87 in a40675c
causing the mismatch, dispite the comment above clearly indicating the values are supposed to be the same.
This was introduced in bpo-1294959 which changed the platlib template to depend on sys.platlibdir, so a mismatch happens when the value of sys.platlibdir is not 'lib'.
(Adding frenzy and vstinner to the nosy list since you introduced the comment in distutils and the sys.platlibdir change, respectively.)
vstinner commentedon Aug 10, 2021
sys.platlibdir was introduced to install libraries in /usr/lib64 rather than /usr/lib. I'm not sure if it should be used to install libraries in $HOME/.local/lib64 rather than $HOME/.local/lib. Previously, Fedora already used $HOME/.local/lib and $HOME/.local/lib64 is not in the sys.path.
Does the site module add $HOME/.local/lib64 to sys.path if it exists?
uranusjr commentedon Aug 10, 2021
This was also briefly discussed in bpo-1294959, but did not go through since “changing posix_user should have no impact on end users”.
It does not, only lib is checked right now.
cpython/Lib/site.py
Lines 288 to 298 in c7ea1e3
----
There are two possible solutions from what I can tell. We could just make posix_user match posix_prefix and always respect sys.platlibdir. This could be confusing to existing Python 3.9 users however since many of them already pip-installed things into ~/.local/lib and this would make their user-site packages split in two locations. The other would be to restore the pre-3.9 behaviour in sysconfig to use lib instead of depending on sys.platlibdir. I don’t know who uses sysconfig right now and can’t say what would break, but for pip this would be less disruptive since it currently installs things into ~/.local/ib (provided by distutils).
hroncok commentedon Aug 10, 2021
Installing to ~/.local/lib works, installing to ~/.local/lib64 breaks things, as it is not on sys.path. I agree that restoring the pre-3.9 behavior in sysconfig to use lib instead of depending on sys.platlibdir is a better fix, at least for 3.9 and 3.10. We can redesign things in 3.11 (but I wouldn't).
uranusjr commentedon Aug 10, 2021
I’ve updated the linked PR to change sysconfig instead to not use sys.platlibdir when generating the posix_user scheme. This means posix_user would behave the same in 3.9+ as 3.8 and prior.
encukou commentedon Sep 7, 2021
Marking as *potential* release blocker for 3.10.
Pablo, without this change the newest pip (with [PR 10358]) will not work on Python 3.10 built --with-platlibdir.
This configure option was added in 3.9 for distros that separate
/usr/liband/usr/lib64, which used downstream patches before.Things will be broken if:
pipswitches to _USE_SYSCONFIG in 3.10 (with merged [PR 10358])So, if this isn't merged in 3.10, either pip or downstreams will need to implement a workaround. (If pip doesn't hold off, downstreams that build --with-platlibdir will likely carry this exact patch.)
On the other hand, we're very late in the release cycle.
Note that there's a similar bug in bpo-45035
[PR 10358]: pypa/pip#10358
pablogsal commentedon Sep 7, 2021
IIUC we need to backport PR27655 to 3.10 no? Or do we need something else?
encukou commentedon Sep 7, 2021
Possibly together with PR28011.
pablogsal commentedon Sep 7, 2021
New changeset 608a629 by Miss Islington (bot) in branch '3.10':
bpo-44860: Make sysconfig posix_user not depend on platlibdir (GH-27655) (GH-28197)
608a629
pablogsal commentedon Sep 7, 2021
Petr, is something left to do for this release blocker? I am planning to start the release if everything is OK
encukou commentedon Sep 7, 2021
I believe everything is in order now.
vstinner commentedon Sep 7, 2021
See also the email on python-dev:
https://mail.python.org/archives/list/python-dev@python.org/thread/5UU6V2B3KBS4Z7OG5T7D6YQZASFNSBJM/
vstinner commentedon Sep 7, 2021
Let's continue the discussion for the unix_home scheme in bpo-45035.
hroncok commentedon Sep 8, 2021
There seem to be a regression in test_user_similar: https://bugs.python.org/issue45136
vstinner commentedon Sep 9, 2021
New changeset 49acac0 by Victor Stinner in branch 'main':
bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235)
49acac0
vstinner commentedon Sep 9, 2021
New changeset 11103eb by Miss Islington (bot) in branch '3.10':
bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235) (GH-28251)
11103eb
vstinner commentedon Sep 9, 2021
I marked bpo-45136 as a duplicate of this issue.
pablogsal commentedon Oct 4, 2021
New changeset 01be518 by Pablo Galindo (Miss Islington (bot)) in branch '3.10':
bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235) (GH-28251)
01be518