Repository navigation
3.11.0a3: under tox, sys._base_executable is wrong #90186
Description
| BPO | 46028 |
|---|---|
| Nosy | @vsajip, @vstinner, @tiran, @nedbat, @zooba, @pablogsal, @saaketp |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2021-12-10.00:24:21.754>
labels = ['3.11']
title = '3.11.0a3: under tox, sys._base_executable is wrong'
updated_at = <Date 2022-01-18.15:47:49.048>
user = 'https://github.com/nedbat'bugs.python.org fields:
activity = <Date 2022-01-18.15:47:49.048>
actor = 'steve.dower'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-12-10.00:24:21.754>
creator = 'nedbat'
dependencies = []
files = []
hgrepos = []
issue_num = 46028
keywords = ['patch', '3.11regression']
message_count = 30.0
messages = ['408166', '408167', '408168', '408180', '408195', '408206', '408278', '408311', '408312', '408313', '408318', '408319', '408483', '408484', '408489', '408490', '408491', '408492', '408493', '408494', '408525', '408526', '408528', '408569', '408692', '408710', '408744', '410850', '410872', '410873']
nosy_count = 7.0
nosy_names = ['vinay.sajip', 'vstinner', 'christian.heimes', 'nedbat', 'steve.dower', 'pablogsal', 'saaketp']
pr_nums = ['30144']
priority = 'normal'
resolution = None
stage = 'commit review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue46028'
versions = ['Python 3.11']
Activity
nedbat commentedon Dec 10, 2021
Under tox, sys._base_executable is not an actual file for 3.11.0a3. It was fine in 3.11.0a2.
To reproduce:
--- 8< --------------------
# tox.ini
[tox]
envlist = py{310,311a2,311}
skipsdist = True
[testenv]
commands =
python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
[testenv:py311a2]
# This is the path to 3.11.0a2 if you have it.
basepython = /usr/local/pyenv/pyenv/versions/3.11.0a2/bin/python3
----------------------------
Create a new Python 3.8 virtualenv, and install latest tox (3.24.4 for me).
Then run "tox". I see:
--------------------------------------------------------------------------------
py310 create: /Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py310
py310 run-test-pre: PYTHONHASHSEED='534434199'
py310 run-test: commands[0] | python -c 'import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))'
/Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py310/bin/python
True
py311a2 create: /Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py311a2
py311a2 run-test-pre: PYTHONHASHSEED='534434199'
py311a2 run-test: commands[0] | python -c 'import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))'
/Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py311a2/bin/python
True
py311 create: /Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py311
py311 run-test-pre: PYTHONHASHSEED='534434199'
py311 run-test: commands[0] | python -c 'import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))'
/usr/local/pyenv/pyenv/versions/3.11.0a3/python
False
_________________________________________________________ summary _________________________________________________________
py310: commands succeeded
py311a2: commands succeeded
py311: commands succeeded
congratulations :)
--------------------------------------------------------------------------------
This came to my attention because the coverage.py test suite uses "python -m venv" to create environments. They worked under 3.11.0a2, but failed under a3. I tracked it down to a difference in sys._base_executable.
I couldn't see a difference in those values without tox, but I'm not sure why it changes the results.
pablogsal commentedon Dec 10, 2021
Steve, could this be related to the changes in getpath?
pablogsal commentedon Dec 10, 2021
Ned, are you able to bisect this or provide a simpler reproducer that doesn't involve tox?
tiran commentedon Dec 10, 2021
Commit 9f2f7e4 has correct _base_executable
_base_executable in commit 99fcf15 is wrong
nedbat commentedon Dec 10, 2021
git bisect also identifies that commit as the first bad:
99fcf15 is the first bad commit
commit 99fcf15
Author: Steve Dower <steve.dower@python.org>
Date: Fri Dec 3 00:08:42 2021 +0000
pablogsal commentedon Dec 10, 2021
Indeed, seems my original hunch is correct. Steve, could you take a look when you have some time?
zooba commentedon Dec 11, 2021
I'm going to need a decent amount of time to learn all of these components, because I never use this OS, Tox, nor virtualenv :) I'll try and get to it, but don't hold your breath. Luckily, Modules/getpath.py is much easier to follow and modify than the old systems.
If there's a pyvenv.cfg involved, base_executable should be calculated based on the "home" key in it. Previously, I don't think we calculated it at all on Linux - it was just sys.executable before site.py changes anything.
On Windows, it was always intended to be "the executable that new venvs should be created with" so that venvs created from venvs would be based off the same install, rather than trying to chain. I have no idea what the correct path for that is in this context, so could do with some help.
nedbat commentedon Dec 11, 2021
Tox isn't needed, just venv from the stdlib:
zooba commentedon Dec 11, 2021
What's the contents of the pyvenv.cfg in these cases?
It looks like the first case is definitely wrong, because the base
executable should not be in "venv_a2" (that's sys.executable), but I
don't know where it should be on your system.
nedbat commentedon Dec 11, 2021
The two venvs seem analogous:
saaketp commentedon Dec 11, 2021
I tried the same stuff as nedbat on WSL2, and I see similar change in the path of sys._base_executable (though I get a different "base" path on a3, so the path exists even there).
13 remaining items
zooba commentedon Dec 14, 2021
I assume /private/tmp/bpo-46028/311-nested/bin/python3.11 exists though?
Probably that's the issue here - I don't know how else to get the real
executable *name* other than copying from argv[0], and it isn't supposed
to be necessary.
You also have v311/bin/python3.11, right? If you use that one, does it
work? I'm trying to narrow down where the base executable is actually
being launched and why.
[-]3.11.0a3: sys._base_executable is wrong, breaks venv - it wasn't under 3.11.0a2[/-][+]3.11.0a3: under tox, sys._base_executable is wrong[/+][-]3.11.0a3: sys._base_executable is wrong, breaks venv - it wasn't under 3.11.0a2[/-][+]3.11.0a3: under tox, sys._base_executable is wrong[/+]zooba commentedon Dec 14, 2021
Or possibly that error is coming from the attempt to copy it? And since
both executable and base_executable don't have the 3/3.x suffix, the
copy is failing because the "real" binary does have the suffix.
This could be corrected in getpath.py with a platform-specific quirk
that searches for suffixed binaries for base_executable, but for
performance reasons I think we'd prefer to have that check in venv so
that it doesn't impact every single launch of CPython.
The actual binary could also be added to pyvenv.cfg as another value -
parsing that out in getpath.py is now considerably easier for someone to
add than it used to be.
nedbat commentedon Dec 14, 2021
Yes, that file exists, but it's a symlink to a non-existent file:
zooba commentedon Dec 14, 2021
Does the first venv's 'python' link to python3[.11]? If so, maybe _base_executable should be based on real_executable's filename rather than executable (that is, *after* resolving symlinks rather than before).
I don't *think* that will cause any issues, and it shouldn't be any more expensive to compute. Only has to change for when a pyvenv.cfg is detected I think.
nedbat commentedon Dec 16, 2021
Here's the experiment again with 3.10.1 and 3.11.0a3, and more ls's:
zooba commentedon Dec 16, 2021
This PR *might* be a fix, but I think it's only partial. It isn't going to work if you've made a venv and copied "python3"->"python", for example.
It still might be best solved by writing base_executable into pyvenv.cfg, and then simply reading it out.
nedbat commentedon Dec 17, 2021
Thanks, your change looks good. The exact symlinks in the nested venv's are different for 3.10.1 and your 3.11, but they both work:
$
vstinner commentedon Jan 18, 2022
See also: https://discuss.python.org/t/virtual-environments-vs-nix-python-upgrades/12588 "Virtual environments vs. *nix Python upgrades".
zooba commentedon Jan 18, 2022
New changeset 7407fe4 by Steve Dower in branch 'main':
bpo-46028: Calculate base_executable by resolving symlinks in a venv (GH-30144)
7407fe4
zooba commentedon Jan 18, 2022
Merged my PR, but I want to leave this open in commit review for now - I'm not sure it deals with all the issues here, and probably not everything from the Discourse thread linked by Victor (though it might come close).
zooba commentedon Aug 25, 2022
I'm going to assume that since we made it to RC without further comment, the change I made fixes it.
If not, I guess 3.11 is going to be the dividing line where the behaviour changed...