Repository navigation
venv using symlinks and empty pyvenv.cfg isn't recognized as venv / able to find python home #135773
Description
Bug report
Bug description:
Previously, it was possible to have a functional (enough) venv by simply doing two things:
- Creating an empty pyvenv.cfg file
- Creating a bin/python3 which symlinked to the actual interpreter
The net effect was you had a sufficiently relocatable venv (at the interpreter level) to make it into a fully relocatable venv (at the application level). This is because Python would follow the symlink to locate its home, which let it get through initial bootstrapping, enough to pick a venv-relative site-packages directory.
The net effect was Python followed the symlink to find home, which let it bootstrap itself enough to find the venv's site-packages. Once at that point, application code is able to hook in and do some fixups.
This bug seems to be introduced by #126987
I'm still looking at the changes to figure out why, exactly. I'm pretty confident it's that PR, though -- in my reproducer, things work before it, but break after it.
This behavior of automatically find Python home is something Bazel's rules_python has come to rely on. Insofar as venvs are concerned, the two particular constraints Bazel imposes are:
- Absolute paths can't be used. This is because they may be machine specific. In Bazel, the machine that generates pyvenv.cfg may be different than who uses it at runtime (e.g a remote build worker generates it, and a user runs it locally).
- The file system must be assumed to be read-only after building. This prevents creating/modifying a pyvenv.cfg at runtime in the venv's directory to work around (1).
(As an aside, how the env var PYTHONEXECUTABLE gets handled might have also been affected by this; I haven't looked deeply yet, but one of my tricks was to set that to the venv bin/python3 path and invoke the real interpreter, but that seems to no longer work, either)
I'll post a more self contained reproducer in a bit. In the meantime, something like this can demonstrate the issue:
mkdir myvenv
cd myvenv
touch pyvenv.cfg
mkdir bin
ln -s <path to python> bin/python3
mkdir -p lib/python3.14/site-packages
bin/python3 -c 'import sys; print(sys.prefix); print(sys.base_prefix); print(sys.path)'
After the change, it'll print warnings and e.g. /usr/local as the prefixes, which is incorrect. sys.path won't have the venv's site-packages directory, which is incorrect.
Before the change, it'll print the venv directory, underlying interpreter directory, and the venv's site-packages, which is correct.
cc @FFY00 (author of above PR)
CPython versions tested on:
3.14
Operating systems tested on:
Linux
Activity
rickeylev commentedon Jun 20, 2025
After digging in a bit more, I think the fix might be simple: don't null out
venv_prefixif a home key wasn't found. i.e. remove thevenv_prefix=Noneline around L418 in getpath.pyI'm also more convinced this should be considered a regression because the logic that was ported from site.py to getpath.py doesn't result in the same behavior as before.
I think what was happening previously was:
(1) getpath.py would calculate venv_prefix, and ultimately set it to None. This meant nothing, though, because the
venv_prefixvariable was entirely unused (all references to it are on commented out lines).(2) site.py comes around and does its own pyvenv.cfg logic. This does two things: sets sys.prefix to the pyvenv.cfg location and adds the venv's (sys.prefix) site-packages directory to sys.path
The net effect is sys.prefix vs sys.base_prefix looks correct, and the venv's site packages is also correct. Yay, functional venv.
After the change:
getpath.py now computes and sets sys.prefix. However, it doesn't preserve the "treat pyvenv.cfg location as sys.prefix" logic from (2) -- the "else" clause of the "find home key" loop sets venv_prefix=None. So, it ignores the pyvenv.cfg location and acts like there's no venv. Now I understand what the comment about "home key is required" means.
The net effect is boo, the venv is ignored and we don't have a functional venv.
I realize that "else" clause to set venv_prefix=None has been there awhile, however:
venv_prefixvariable in getpath.py existing at all was weird -- it was never actually used? It looks like it has always been commented out, even in the commits that introduced it? I get the sense that the logic in site.py was supposed to have lived in getpath.py all along, but circumstances ended up with it not working at the time, so things were commented out in getpath.py and hacks put into site.py (and then some later hacks to handle site.py changing sys.prefix after the fact).zooba commentedon Jun 21, 2025
You were relying on entirely undocumented and unsupported functionality - can you phrase your ask in the form of a feature request?
You might also be able to use
PYTHONHOMEto achieve what you need (which is an existing documented and supported feature).rickeylev commentedon Jun 21, 2025
Hi Steve,
Thanks for the response.
"unsupported" seems a bit strong. There's clearly a lot of logic in there to handle home missing and reading through the symlink, and it's been there a long time. It very much looks explicitly implemented to behave that way.
PYTHONHOME might work in some cases, but tends to be problematic because it gets inherited by subprocesses and interfere with them. Our users have reported this category of problem; subsequently, we have to avoid setting any PYTHON* environment vars.
To clarify, what is broken by this isn't a single, specific, application. What it breaks is the ability to create venv-based python programs using Bazel. I spent many, many hours trying to find alternatives to an empty pyvenv.cfg file that still acted correctly enough, but it was the only thing that seemed to function.
ncoghlan commentedon Jun 23, 2025
Given the behaviour was undocumented either way, and the offered PR is straightforward, locking in the historical behaviour as the expected behaviour for a missing
homekey seems reasonable to me. It won't work for copy based environments, but that's a restriction in the old code as well.I do wonder if we could give a meaningful error in the "no
homekey and no executable symlink" case rather than silently ignoring the inconsistent config file.zooba commentedon Jun 23, 2025
Show me the documentation that tells you to create an empty
pyvenv.cfg? The only supported way to create that file is using thevenvmodule, so any other use of it is unsupported. Unsupported things change, they don't "break", and the responsibility is on people using the unsupported features to test, detect, and adapt.Yeah, all too aware of this... environment variables are a pain. If the child processes are sensitive to this, then there are plenty of other ways for them to be broken. Still, I wish we'd made these variables version-sensitive (back in the 1990s or whenever), so at least switching between versions would be okay. The POSIX assumptions run deep...
My request still stands.
zooba commentedon Jun 23, 2025
Out of interest, what happens if you set
home=bin?I'd be more amenable to adding support for relative paths to
pyvenv.cfgthan having a vaguely defined notion of how to handle an invalid one.rickeylev commentedon Jun 24, 2025
Short version: the relative path "bin" gets passed to joinpath() in search_up(), fails to find landmark files, and eventually the compile-time value /usr/local is used for base prefix instead.
The longer version:
Whenn pyvenv.cfg is parsed, two things happen:
executable_dirandreal_executable_dirvariable valuesbase_executableis then set to what theexecutablesymlink points toWhen pyvenv.cfg parsing is exited, the resulting state is e.g.
(of these, executable_dir is the most relevant. The others don't factor in much)
Eventually it ends up in the "calculate: prefix and exec_prefix" logic, where
executable_dir="bin"is the deciding factor. The last-chance thing it looks for is e.g.bin/lib/python3.15/os.py(i.e. that path relative to current working directory), by way of callingsearch_up(<prefix>, <stdlib landmarks>), which is whereisfile(joinpath("bin", landmark))occurs. All those pokes fail, so it falls back to the compile-time /usr/local value (uppercasePREFIXvariable).Up until now,
prefixrefers to the location of the actual python install. It then checks venv_prefix (computed by the pyvenv.cfg parsing earlier), and then sets base_prefix to prefix, and prefix to the venv_prefix. After this, prefix refers to the venv location, while base_prefix refers to the actual location. i.e.prefix=/venv; base_prefix=/usr/localThings continue on, but the remaining code paths aren't really activated (e.g.
_pthfile overriding). The net result is Python looks for its stdlib in e.g. /usr/local, can't find it, and interpreter initialization fails.(That's a no-frills evaluation. There's various env vars and config if-statements that look like they could end up quite differently)
Me too :). I started a thread a couple days ago to help figure that out: https://discuss.python.org/t/making-venvs-relocatable-friendly/96177
Something to work out for 3.15? Do I just need to write a PEP?
rickeylev commentedon Jun 24, 2025
I've created #135831
It has the change I mentioned and a test. CI looks happy (except for NEWS, which should probably be skipped?). I wired it into rules_python and that looks happy, too. I read through the PR life cycle part of the devguide, and I think I got everything in order.
Let me know what else I can do.
In the meantime, I'll file a feature request about relative paths for pyvenv.cfg's home keyedit -- actually, I'm not sure if the discussion in the thread meets the FR text of "most people [on python ideas] support your idea", so I'll hold off another day or so to see where else that thread goes.rickeylev commentedon Jun 27, 2025
Feature request issue created: #136051
gh-135773: have pyvenv.cfg without home key anchor a venv and deduce …
pythongh-135773: have pyvenv.cfg without home key anchor a venv and d…
[3.14] gh-135773: have pyvenv.cfg without home key anchor a venv and …
ncoghlan commentedon Jul 4, 2025
PEP 796 (linked from the feature request) has been written to provide a supported approach to handling this in 3.15+.
I've merged the PRs to restore the legacy implementation-defined behaviour for 3.14 and the 3.15 dev branch so 3.14 won't be a special case that differs from both 3.15+ and 3.13 and earlier.
pythongh-135773: have pyvenv.cfg without home key anchor a venv and d…
pythongh-135773: have pyvenv.cfg without home key anchor a venv and d…
pythongh-135773: have pyvenv.cfg without home key anchor a venv and d…
pythongh-135773: have pyvenv.cfg without home key anchor a venv and d…
pythongh-135773: have pyvenv.cfg without home key anchor a venv and d…