Repository navigation
Warning about the use of site-specific path when the site initialization has been disabled #126793
Description
Activity
I'm not sure I agree with the assertion that
purelibandplatlibshould be considered unavailable when running under-S. All that should mean is that they're not actually added tosys.pathautomatically, not that they're totally off limits. In fact, there's nothing stopping one from runningpython3 -i -S -c 'import site; site.main()'as a weird way to start up a regular REPL with no headers :)I would agree with that if the
-Sflag was as simple as you describe it, but it isn't 😅What makes things messy is that on top of the customizations you mentioned, the
sitemodule is also responsible for implementing virtual environments. Disabling thesiteinitialization also disables the virtual environment activation mechanism.
If we were to think conceptually of an environment resultant of-Ssimply as a "locked down" version of another existing environment, then that existing environment would be the installation-wide environment (eg./usr/lib/python3.14t/site-packages), rather than the virtual environment (eg.~/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages).In practice, this means that
python -S -m ensurepip,python -S -m pip install ..., or other commands that customize a "fully-fledged" environment would be acting on the installation-wide environment, even when running these commands inside a virtual environment. IMO this is a major footgun and, to me, is an undesirable behavior left over from of historic implementation decisions.I think we should think conceptually of
-Smore as an environment-less way of running Python, than a limited environment.
I am happy to hear other points of view though.I think we should think conceptually of
-Smore as an environment-less way of running Python, than a limited environment.This sounds reasonable to me, however...
Disabling the
siteinitialization also disables the virtual environment activation mechanism.Not entirely (a lot is handled in getpath, IIRC independent of the
-Soption), but if the above definition sticks, then it should disable it entirely. Effectively you should only get the stdlib (and any explicit additions), regardless of whether you launch through a venv or not. And so...this means that
python -S -m ensurepip,python -S -m pip install ..., or other commands that customize a "fully-fledged" environment would be acting on the installation-wide environment, even when running these commands inside a virtual environmentHere's where it gets complicated. There's strictly nothing wrong with
ensurepiporpip installalways installing into the default "site", regardless of whether it's active or not, but that is going to mean something weird if we don't clearly define what launching a venv with-Smeans. Hopefully we can define that in a way that makes it consistent and doesn't actually change existing behaviour.Perhaps this works:
- launching a venv overrides the default site directory
site.main()adds the default site directory (and etc...) when called-Sskips callingsite.main()[ensure]pipinstalls to the default site directory regardless of whether it's active or not
I'm sure there are new edge cases introduced by a definition like that (if it works at all, I haven't thought it all the way through yet), but perhaps those are easier to fix/handle?
Not entirely (a lot is handled in getpath, IIRC independent of the
-Soption), but if the above definition sticks, then it should disable it entirely. Effectively you should only get the stdlib (and any explicit additions), regardless of whether you launch through a venv or not. And so...(I realized I went a bit off track when writing the reply below, but I am keeping it since it's useful)
Right, I am aware that
getpathhas some code that covers virtual environments, but AFAIK nothing that changes any other aspects of the initialization, at least not in any meaningful way anymore. That may have been the case in the past, but the only code left now is for findingsys._base_executablein niche situations where it is unknown.Here's my understanding of what happens:
getpathlooks for apyvenv.cfg(the indicator of a virtual environment) near the executable or, if unknown, the current directory- If
base_executablefromPyConfigis not set ahomekey is specified inpyvenv.cfg, it searches for the interpreter executable there and sets itbase_executableto its path if found - If no process executable was provided,
base_executableis used instead when looking for the._pthfile syssetsys._base_executableto the value inPyConfig
So, there may be a possibility for
sys._base_executableto be set to an interpreter from a virtual environment, though this looks like a very niche case, and am not sure if there's are any relevant real world use-case.
Even in that possibility, it only seems to affect._pthfile results in-Eand-S, I don't think it's relevant here.
Considering this,If there's anything else that I am missing, let me know.
Here's where it gets complicated. There's strictly nothing wrong with
ensurepiporpip installalways installing into the default "site", regardless of whether it's active or not, but that is going to mean something weird if we don't clearly define what launching a venv with-Smeans. Hopefully we can define that in a way that makes it consistent and doesn't actually change existing behaviour.Perhaps this works:
* launching a venv overrides the default site directory * `site.main()` adds the default site directory (and etc...) when called * `-S` skips calling `site.main()` * `[ensure]pip` installs to the default site directory regardless of whether it's active or notI'm sure there are new edge cases introduced by a definition like that (if it works at all, I haven't thought it all the way through yet), but perhaps those are easier to fix/handle?
I think what we are running into here is the fact that virtual environments probably shouldn't be implemented by the
sitemodule. IMO, the base mechanism (prefix relocation viapyvenv.cfg) should be provided as a core feature, andsitecould still keep being responsible for the environment customization.My proposal to move forward:
- Start setting
prefixandbase_prefixto thepyvenv.cfgdirectory ingetpathagain - Document the
pyvenv.cfgdetection as part of the interpreter initialization, instead of part of thesiteimport - Update the all other documentation to reflect this change
Even in that possibility, it only seems to affect
._pthfile results in-Eand-S, I don't think it's relevant here.FWIW, the mere presence of a
._pthfile implies both-Eand-S(and animport sitein the._pthfile removes-Sagain).
Bug report
Bug description:
Even when disabling the
siteinitialization (passing-S),sysconfig.get_paths()still returns site-specific paths, as there's no way in the current API to express the unavailability of such paths. A lot of code does not take this use-case into account, and assumes these are paths that are currently in-use for the active environment.An example of this is
ensurepip, which will still proceed and install pip to the site directories, despite thesitemodule being disabled.IMO, it should refuse to install when the
sitemodule is disabled, as there are no valid paths to install pip to under the current environment.To make things worse, the result of
sysconfig.get_paths(), currently, is inconsistent, and does not always return the same result (see #126789).Newer APIs (#103481), should handle this scenario directly, by expressing directly the unavailability of
purelibandplatlib, for example.CPython versions tested on:
3.9, 3.10, 3.11, 3.12, 3.13, 3.14, CPython main branch
Operating systems tested on:
Linux