Repository navigation
sysconfig data inconsistencies caused by the disabling the site initialization on virtual environments #126789
Description
Activity
Things are a bit messy, as the virtual environment mechanism is technically part of the
sitecustomization. In reality, thepyvenv.cfgdetection is implemented on the interpreter initialization, but the customization itself is done bysite. I feel like this should probably be reconsidered.Anyway, to match this model, if
sitehas been initialized,sysconfig.get_paths()should always return the virtual environment paths. So, the correct result should be as follows:$ python -S Python 3.14.0a1+ experimental free-threading build (heads/main:de0d5c6e2e1, Oct 23 2024, 15:37:46) [GCC 14.2.1 20240910] on linux >>> import sysconfig >>> sysconfig.get_paths() {'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/usr/local/lib/python3.14t', 'purelib': '/usr/local/lib/python3.14t/site-packages', 'platlib': '/usr/local/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/usr/local/bin', 'data': '/usr/local'} >>> import site >>> site.main() >>> sysconfig.get_paths() {'stdlib': '/usr/local/lib/python3.14t', 'platstdlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t', 'purelib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'platlib': '/home/anubis/.virtualenvs/test-sysconfig-paths/lib/python3.14t/site-packages', 'include': '/usr/local/include/python3.14td', 'platinclude': '/usr/local/include/python3.14td', 'scripts': '/home/anubis/.virtualenvs/test-sysconfig-paths/bin', 'data': '/home/anubis/.virtualenvs/test-sysconfig-paths'}
Correction:
sysconfigdoes not cache thesysconfig.get_paths(), it cachessysconfig.get_config_vars(), which is used to calculate the paths.- changed the title
[-]Fix `sysconfig.get_paths()` inconsistencies caused by the disabling the `site` initialization on virtual environments[/-][+]`sysconfig` data inconsistencies caused by the disabling the `site` initialization on virtual environments[/+]on Nov 13, 2024 While fixing the main bug, I have noticed that the
prefixandexec_prefixvariables get overwritten by_init_posix, so on virtual environmentssysconfig.get_config_vars('prefix')andsys.prefixwill have different values, the same goes forexec_prefix.IMO this is worth to fix, but we probably shouldn't backport it as it is a significant behavior change, and there's a possibility it would break existing user code.
Correction:
sysconfigdoes not cache thesysconfig.get_paths(), it cachessysconfig.get_config_vars(), which is used to calculate the paths.Just some more background on this,
sysconfig.get_paths()uses an incomplete copy of the config vars directory to expand the path placeholder, which is what threw me off initially. This makes it a little bit more tricky to fix the issue, as we need to make sure we update the config vars cache without creating a recursive loop, but wasn't too difficult to get the implementation right.- added a commit that references this issue
on Nov 17, 2024 5 remaining items
- added a commit that references this issue
on Nov 18, 2024 - added a commit that references this issue
on Nov 19, 2024 @hugovk, sorry I just say this now. I should have mentioned it in the issue text, but as per the title this only affects virtual environments.
The issue is that currently
siteis responsible for the virtual environment initialization (settingsys.prefixandsys.exec_prefix), so when you run a virtual environment with-S,sys.prefixandsys.exec_prefixpoint to the base installation, meaning that, canonically, you are not in a virtual environment. Insite.main(),sys.prefixandsys.exec_prefixare updated to point to the virtual environment.
If thesysconfig.get_config_vars()cache, which is used when calculatingsysconfig.get_paths(), is created before thesiteinitialization, it still holds the old prefix values, resulting insysconfigreporting different paths depending on whether the cache was generated before or after thesiteinitialization.
The issue only affects virtual environments, because that's the only case wheresys.prefixandsys.exec_prefixchange after initialization.I hope that helps clarify things. Let me know if you have any more questions.
Thanks for the clarification!
The change for this seems to have regressed building Firefox with Python 3.12.8 and Python 3.13.1: https://bugzilla.mozilla.org/show_bug.cgi?id=1935621
I don't know enough about Firefox's build system, mach, to know if it is an issue on their side or not.
Reacted by Filipe Laíns@thesamesam, I investigated the issue, the source of the issue is that they are overwriting
sys.prefixand were relying onsysconfigdata using the original prefix. GH-126812 fixedsysconfigdata not matchingsys.prefix/sys.exec_prefixif they were updated after import.sysconfigused to cache their values at import, so if they were updated afterwards, the data wouldn't reflect the change.
In GH-126987, we moved the virtual environment activation from
siteto the interpreter path initialization, which means that the reproducible above would no longer trigger thesysconfigcache issue, assys.prefix/sys.exec_prefixare no longer updated by thesitemodule. Because of this, I also reverted the cache bugfix, but seeing that some folks are updatingsys.prefixmanually, I think we should re-apply the bugfix.Reacted by Sam James- added a commit that references this issue
on Dec 19, 2024 - added a commit that references this issue
on Dec 20, 2024
Bug report
Bug description:
Currently, running the Python interpreter with
-S(disabling thesitemodule initialization) causes several inconsistenciessysconfig.get_paths()depends on whethersysconfigwas imported before or after thesiteinitializationCurrently,
sysconfigcalculates the paths at import time and caches them. This results in it resulting incorrect paths if thesiteinitialization happens aftersysconfigwas imported.Expected output when the
sitemodule has been initialized:And just to make sure this is not dependent on the interpreter being run with
-S, and rather on thesysconfigimport time.CPython versions tested on:
3.9, 3.10, 3.11, 3.12, 3.13, 3.14
Operating systems tested on:
Linux
Linked PRs