getpath miscalculates sys.path from second initialization with PYTHONHOME on Windows #91985
Description
When I ran the code below in a python build directory:
#include <Python.h>
int
main(int argc, char *argv[])
{
PyConfig config;
PyConfig_InitPythonConfig(&config);
for (int i=0; i < 3; i++) {
printf("\n%d\n", i);
Py_InitializeFromConfig(&config);
PyRun_SimpleString("import sys; list(map(print, sys.path))");
Py_Finalize();
}
PyConfig_Clear(&config);
return 0;
}
- Expected results without
PYTHONHOME:
0
C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\PCbuild\amd64
C:\cpython-main
C:\cpython-main\Lib\site-packages
1
C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\PCbuild\amd64
C:\cpython-main
C:\cpython-main\Lib\site-packages
2
C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\PCbuild\amd64
C:\cpython-main
C:\cpython-main\Lib\site-packages
set PYTHONHOME=C:\cpython-main:
0
C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\PCbuild\amd64
C:\cpython-main
C:\cpython-main\Lib\site-packages
1
C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\DLLs # <<<<<<<<<< unexpected
C:\cpython-main
C:\cpython-main\Lib\site-packages
2
C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\DLLs # <<<<<<<<<< unexpected
C:\cpython-main
C:\cpython-main\Lib\site-packages
Currently, test_embed fails due to this. (#32313)
Activity
[-][Windows] getpath miscalculate sys.path from second initialization with PYTHONHOME[/-][+][Windows] getpath miscalculates sys.path from second initialization with PYTHONHOME[/+]neonene commentedon May 4, 2022
[-][Windows] getpath miscalculates sys.path from second initialization with PYTHONHOME[/-][+]getpath miscalculates sys.path with PYTHONHOME/Py_SetPythonHome() on Windows[/+]neonene commentedon May 14, 2022
On
3.10and earlier with or withoutPYTHONHOME, an executable in a build directory hassys.pathas below:[-]getpath miscalculates sys.path with PYTHONHOME/Py_SetPythonHome() on Windows[/-][+]getpath miscalculates sys.path from second initialization with PYTHONHOME on Windows[/+]zooba commentedon May 23, 2022
Okay, so the problem is that
homeis set on the second call, which means getpath skips checking for build directory landmarks.IIRC, other tests were broken if we check against real_executable for the build landmarks and ignore PYTHONHOME (presumably because they used symlinks to a new directory for tests?), so this probably just needs new logic.
However, the fix in #92980 might also affect this, by not recalculating sys.path on subsequent calls. You might want to test again before changing too much.
neonene commentedon Jun 6, 2022
test_getpathpasses the following test cases:I can also start python in their file layouts, but
test_embedwere broken. NeitherPYTHONHOMEnorhomeworked around. New logic may be needed for this. Do I understand what you mean?zooba commentedon Jun 6, 2022
Had a quick look at your changes, but honestly am going to need a lot more time to figure out that the logic is sound. That also concerns me, because I really want to avoid putting any logic for this stuff back in C - moving to Python was to fix that ;-)
Another idea that might work: what if we just add a config field for "running in-tree build"? It can start as "unspecified" and then be updated if we detect landmarks. Then even if home is already set but we know we're in tree, we can go down the path that overrides platlibdir. What do you think?
neonene commentedon Jun 7, 2022
Thanks for your feedback.
IIUC, we have no way to update the original config? getpath updates the config which was duplicated at
pyinit_core(), and the copy disappears after system configuration with the update of global config (_Py_path_config: deprecated?).For example,
module_search_paths_setand_is_python_buildfields never change in the original config when repeatingPy_InitializeFromConfig().I think the flag will work if you would be ok with saving it into the global config.
zooba commentedon Jun 7, 2022
Oh I see, it's the path config state that's keeping the values around. If we copy
_is_python_buildinto there and only ever set it ourselves, that would be fine, right?neonene commentedon Jun 7, 2022
My experiment (#92411) passed the repeating tests, keeping
_is_python_buildas a member of_Py_path_config. I now think it can be saved as a static variable inpathconfig.c.zooba commentedon Jun 8, 2022
All of
_Py_path_configis basically just a static variable - it never gets referenced outside of this module. But we're trying to minimize static variables, so that one day we can get them to be per-runtime rather than per-process. So please, keep it in the struct.gh-91985: Ensure in-tree builds override platstdlib_dir in every path…
pythongh-91985: Ensure in-tree builds override platstdlib_dir in ever…
kumaraditya303 commentedon Jun 19, 2022
@neonene Is this fixed now or there is still work to be done ?
neonene commentedon Jun 19, 2022
I assume we delay 3.11 backporting just in case.
gh-91985: Ensure in-tree builds override platstdlib_dir in every path…
zooba commentedon Jun 20, 2022
This one can be backported immediately (and now has been). It only affects source builds and no public APIs.
Revert 30935 defer bpo45162 to 312 (#94285)