sys.executable can be not normalized

@terryjreedy

BPO	31030
Nosy	@terryjreedy, @vstinner, @serhiy-storchaka, @eryksun

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-07-25.12:41:50.673>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.10', 'tests']
title = 'sys.executable can be not normalized'
updated_at = <Date 2021-02-27.20:06:55.248>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-02-27.20:06:55.248>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Tests']
creation = <Date 2017-07-25.12:41:50.673>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 31030
keywords = []
message_count = 8.0
messages = ['299065', '299067', '299068', '299444', '299456', '387775', '387782', '387787']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'vstinner', 'serhiy.storchaka', 'eryksun']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31030'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

test_executable in Lib/test/test_sys.py tests that sys.executable is absolute. But this test is too strong, it fails if sys.executable is not normalized.

$ Lib/../python -m test test_sys
Run tests sequentially
0:00:00 load avg: 0.39 [1/1] test_sys
test test_sys failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_sys.py", line 649, in test_executable
    self.assertEqual(os.path.abspath(sys.executable), sys.executable)
AssertionError: '/home/serhiy/py/cpython/python' != '/home/serhiy/py/cpython/Lib/../python'
- /home/serhiy/py/cpython/python
+ /home/serhiy/py/cpython/Lib/../python
?                         +++++++

test_sys failed

1 test failed:
test_sys

Total duration: 2 sec
Tests result: FAILURE

I don't know what is wrong: the value of sys.executable or the test.

test_venv fails for the same cause.

$ Lib/../python -m test -m test_defaults test_venv
Run tests sequentially
0:00:00 load avg: 0.31 [1/1] test_venv
test test_venv failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_venv.py", line 103, in test_defaults
    self.assertIn('home = %s' % path, data)
AssertionError: 'home = /home/serhiy/py/cpython/Lib/..' not found in 'home = /home/serhiy/py/cpython\ninclude-system-site-packages = false\nversion = 3.7.0\n'

test_venv failed

1 test failed:
test_venv

Total duration: 166 ms
Tests result: FAILURE

Technically, /home/serhiy/py/cpython/Lib/../python is absolute and is a valid path. I'm not sure that it's possible to normalize sys.executable early in the Python initialization.

If we modify the site module to normalize sys.executable, it would mean that sys.executable wouldn't be normalized when using the python3 -S command line option. I dislike having subtle differences between site and no site.

I suggest to normalize sys.executable in the unit test.

On Windows, sys.executable is normalized before or during startup. C:\Programs\Python36>Lib\..\python -m test test_sys prints
test_executable (main.SysModuleTest) ... ok

C:\Programs\Python36>Lib\..\python -c "import sys; print(sys.executable)"
C:\Programs\Python36\python.exe

Terry, the Windows implementation calls GetModuleFileNameW to get the executable's fully-qualified path from the loader. However, depending on how Python is started, this could be a \\?\ path, which leads to a bug in the startup code that determines the prefix path based on the landmark "lib\os.py". Probably the easiest solution would be to normalize the executable path to remove the \\?\ or \\.\ prefix. This prefix isn't useful here anyway since Windows doesn't properly support running programs from long paths, not even in Windows 10. I'll open a new issue for this.

In Python 3.10 in POSIX, it's still the case that executable, prefix, exec_prefix, base_prefix, and base_exec_prefix in the sys module do not get normalized. For example, in Linux:

    $ .local/bin/../bin/python3.10 -c "import sys; print(sys.executable)"
    /home/someone/.local/bin/../bin/python3.10

In test/test_sys.py, assertEqual(os.path.abspath(sys.executable), sys.executable) fails. In test/test_venv.py, assertIn('home = %s' % path, data) and assertEqual(out.strip(), expected.encode()) both fail, respectively in test_defaults and test_prefixes.

In Windows, prior to Python 3.6, this can be a problem if Python is run via CreateProcessW using a non-normalized path in lpApplicationName (rare, but possible) instead of letting the system search for the executable in lpCommandLine. For example:

    >>> cmd = 'python -c "import sys; print(sys.executable)"'
    >>> exe = r'C:\Program Files\Python35\..\Python35\python.exe'
    >>> subprocess.call(cmd, executable=exe)
    C:\Program Files\Python35\..\Python35\python.exe
    0

It's no longer an issue in Windows with Python 3.6 and above. When getting the program path while initializing, the GetModuleFileNameW(NULL, ..) result gets canonicalized (e.g. via PathCchCanonicalizeEx). For example:

    >>> exe = r'C:\Program Files\Python36\..\Python36\python.exe'
    >>> subprocess.call(cmd, executable=exe)
    C:\Program Files\Python36\python.exe
    0

This issue was opened with 2 related facts of 2017:
a) sys.executable is not normalized
b) as a result, two tests failed
Serhiy then said "I don't know what is wrong: the value of sys.executable or the test." If it was the tests, they seem to have been fixed elsewhere.

If it is the non-normalization of sys.executable, my concern for IDLE is that it starts the execution subprocess with subprocess.Popen([sys.executable, ...]). Could this be a reason for mysterious IDLE failures?

Eryk, you stated above that on Windows a \\?\ path is possible but will not work for startup, and that you would open an issue for this. Have you?

Do any of the non-normalized Linux (or *nix) executable paths fail for startup?

If it was the tests, they seem to have been fixed elsewhere.

The cited tests haven't been changed to work with non-normalized sys.executable, sys.prefix, etc. But it's not common to run the test suite with a non-normalized path such as "Lib/../build/python".

you stated above that on Windows a \\?\ path is possible but
will not work for startup, and that you would open an issue

Sorry, I don't remember the details of the bug that I mentioned. But it became a non-issue in 3.6+, which canonicalizes the executable path. For example:

    >>> exe = r'\\?\C:\Program Files\Python36\python.exe'
    >>> cmd = 'python -c "import sys; print(sys.executable)"'
    >>> subprocess.call(cmd, executable=exe)
    C:\Program Files\Python36\python.exe
    0

Compare the latter to the output of GetModuleFileNameW(NULL, ...) in this case, which returns a \\?\ extended (verbatim) path:

    >>> exe = r'\\?\C:\Program Files\Python36\python.exe'
    >>> cmd = 'python -c "import _winapi; print(_winapi.GetModuleFileName(0))"'
    >>> subprocess.call(cmd, executable=exe)
    \\?\C:\Program Files\Python36\python.exe
    0

Do any of the non-normalized Linux (or *nix) executable paths fail for startup?

Not that I'm aware of. AFAIK, it's just about the particular tests failing in this case.

I think that was fixed by now. On main I see

% Lib/../python.exe -c "import sys; print(sys.executable)"
/Users/iritkatriel/src/cpython/python.exe


% Lib/../python.exe -m test test_sys test_venv            
Using random seed: 233981894
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 3.10 Run 2 tests sequentially
0:00:00 load avg: 3.10 [1/2] test_sys
0:00:02 load avg: 3.09 [2/2] test_venv
Could not find platform dependent libraries <exec_prefix>

== Tests result: SUCCESS ==

All 2 tests OK.

Total duration: 10.8 sec
Total tests: run=109 skipped=10
Total test files: run=2/2
Result: SUCCESS

Navigation Menu

Repository navigation

sys.executable can be not normalized #75213

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

sys.executable can be not normalized #75213

Description

Activity

serhiy-storchaka commented on Jul 25, 2017

serhiy-storchaka commented on Jul 25, 2017

vstinner commented on Jul 25, 2017

terryjreedy commented on Jul 29, 2017

eryksun commented on Jul 29, 2017

eryksun commented on Feb 27, 2021

terryjreedy commented on Feb 27, 2021

eryksun commented on Feb 27, 2021

iritkatriel commented on May 23, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions