Python launcher invoked with command "py" fails to start process

Bug report

When I invoke a python launcher of version 3.11.0 with "py" command, a python launcher fails to start process.

Step to reproduce

Install python 3.11.0 with py launcher for all users.
Invoke "py" in command prompt.

I expect to launch a python interpreter, but got a error as below

>"py"
Unable to create process using 'C:\Python311\py"':  The system cannot find the file specified.

Debug output when enable PYLAUNCHER_DEBUG

>"py"
argv0: py
version: 3.11.0
SearchInfo.originalCmdLine: "py"
SearchInfo.restOfCmdLine:
SearchInfo.executablePath: (null)
SearchInfo.scriptFile: (null)
SearchInfo.executable: py"
SearchInfo.executableArgs: (null)
SearchInfo.company: (null)
SearchInfo.tag: (null)
SearchInfo.oldStyleTag: False
SearchInfo.lowPriorityTag: False
SearchInfo.allowDefaults: False
SearchInfo.allowExecutableOverride: False
SearchInfo.windowed: False
SearchInfo.list: False
SearchInfo.listPaths: False
SearchInfo.help: False
 -V:3.11          C:\Python311\python.exe
 -V:3.10          C:\Python310\python.exe
 -V:3.9           C:\Python39\python.exe
 -V:3.7           C:\Python37\python.exe
env.company: PythonCore
env.tag: 3.11
# about to run: C:\Python311\py"
Unable to create process using 'C:\Python311\py"':  The system cannot find the file specified.

I think this error seems to be similar to #95285. If I invoke py or "py.exe", a python interpreter starts well as I expected.

>py
Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

>"py.exe"
Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

In version 3.10.8, all of "py", py and "py.exe" start a python interpreter correctly.

Your environment

CPython versions tested on: 3.11.0
Operating system and architecture: Windows 10 (10.0.19045.2310)/amd64

Linked PRs

This error causes a failure to build MSI installer between version 3.7 and 3.10. (Example of GitHub Actions build failure in my personal project).

This is because WiX invokes "py" -m pip install -U blurb during build for exe.wixproj and recent GitHub-Hosted runner uses Python Launcher of version 3.11.0.

Confirmed, I am seeing the same issue (see the linked rust issue report above).

Command: "py" "-V"
argv0: py
version: 3.11.0
SearchInfo.originalCmdLine: "py" -V
SearchInfo.restOfCmdLine:  -V
SearchInfo.executablePath: (null)
SearchInfo.scriptFile: (null)
SearchInfo.executable: py"
SearchInfo.executableArgs: (null)
SearchInfo.company: (null)
SearchInfo.tag: (null)
SearchInfo.oldStyleTag: False
SearchInfo.lowPriorityTag: False
SearchInfo.allowDefaults: False
SearchInfo.allowExecutableOverride: False
SearchInfo.windowed: False
SearchInfo.list: False
SearchInfo.listPaths: False
SearchInfo.help: False
 -V:3.11          C:\Users\Gustav\AppData\Local\Programs\Python\Python311\python.exe
 -V:3.10          C:\Users\Gustav\AppData\Local\Programs\Python\Python310\python.exe
 -V:3.9           C:\Users\Gustav\AppData\Local\Programs\Python\Python39\python.exe
 -V:3.8           C:\Users\Gustav\AppData\Local\Programs\Python\Python38\python.exe
env.company: PythonCore
env.tag: 3.11
# about to run: C:\Users\Gustav\AppData\Local\Programs\Python\Python311\py" -V
Unable to create process using 'C:\Users\Gustav\AppData\Local\Programs\Python\Python311\py" -V': The system cannot find the file specified.

In parseCommandLine(), the value of end points to just past the closing double quote if there's no "." found in the tail component. findArgumentEnd() could be modified to exclude the double quote from the end, since the double quote at the end isn't actually part of the command-line argument. It's a syntax character for argument parsing.

cpython/PC/launcher2.c

Lines 565 to 580 in 88385b8

    
           const wchar_t *tail = findArgumentEnd(search->originalCmdLine, -1); 
        
           const wchar_t *end = tail; 
        
           search->restOfCmdLine = tail; 
        
           while (--tail != search->originalCmdLine) { 
        
               if (*tail == L'.' && end == search->restOfCmdLine) { 
        
                   end = tail; 
        
               } else if (*tail == L'\\' || *tail == L'/') { 
        
                   ++tail; 
        
                   break; 
        
               } 
        
           } 
        
           if (tail == search->originalCmdLine && tail[0] == L'"') { 
        
               ++tail; 
        
           } 
        
           // Without special cases, we can now fill in the search struct 
        
           int tailLen = (int)(end ? (end - tail) : wcsnlen_s(tail, MAXLEN));

A couple of issues in findArgumentLength(), which could also be resolved here without creating a new issue for them:

If the buffer doesn't start with a double quote, it needs to search for white space -- at least tab ("\t") in addition to space (" ").
Since findArgumentLength() is used by findArgumentEnd(), which is only ever called to parse the first argument of the command line, it should not implement backslash escaping of double quotes. When parsing the first argument of the command line, both CreateProcessW() and CommandLineToArgvW() enter quote mode if the first character is a double quote, in which case the argument ends at the next double quote, regardless of whether it's followed by whitespace. That's a bit simpler than the C runtime, but even the C runtime doesn't implement escaping of double quotes in the first argument.

Merge branch 'main' into pythongh-99442

gh-99442: Fix handling in py.exe launcher when argv[0] does not inclu…

pythongh-99442: Fix handling in py.exe launcher when argv[0] does not…

The fix appears to have addressed part of my issue, but I'm still getting the same error in a slightly different case. I haven't yet been able to reduce the test case to something smaller, but if I use the just command line utility, with a justfile as follows:

set windows-shell := ["pwsh.exe", "-NoLogo", "-Command"]

test_py:
    #!py
    print(12)

then running just test_py gives the error:

Unable to create process using 'C:\Users\Gustav\AppData\Local\Temp\justc2pOnD\py C:\Users\Gustav\AppData\Local\Temp\justc2pOnD\test_py': The system cannot find the file specified.

If I add some debugging prints, I can see that the command being run was "py" "C:\Users\Gustav\AppData\Local\Temp\justc2pOnD\test_py".

Thanks for the quick extra test! This one looks like it's probably shebang parsing - can you include the whole output you get with PYLAUNCHER_DEBUG set?

Oh, this is probably the deliberate change of behaviour to make shebang lines only search PATH if you use /usr/bin/env in them.

If it falls through to the default case, we don't let you search for arbitrary executables. It will only look adjacent to the script file. The old behaviour was never documented, and isn't safe (the new behaviour is only mildly safer, but at least you can scan for usr/bin/env and recognise when it's going to happen, rather than trying to prove a negative). The new documentation reads:

Shebang lines that do not match any of these patterns are treated as Windows paths that are absolute or relative to the directory containing the script file. This is a convenience for Windows-only scripts, such as those generated by an installer, since the behavior is not compatible with Unix-style shells. These paths may be quoted, and may include multiple arguments, after which the path to the script and any additional arguments will be appended.

gh-99442: Fix handling in py.exe launcher when argv[0] does not inclu…

#!py is a strange shebang, asking the launcher to recursively run itself. There should be a check in _useShebangAsExecutable() to prevent recursively running the launcher, as you did for the "#!/usr/bin/env" case in searchPath(). Or modify checkShebang() to always check for this. Otherwise #!py will blow up if the launcher is named "py.exe", and the script is beside it.

Unfortunately, other people demanded that usr/bin/env python3 should prioritise the PATH search over the version number, so we can't have perfect compatibility that way. But perhaps all these py -3 cases would be quite happy with a python3 on PATH override?

if it's run directly with ./foo.py then I just wanted to make sure it failed early if you only had python 2 available (the script is old enought that that was a concern at the time 😛).

Note that if CMD or PowerShell is used instead of bash, then directly running a .py script will execute the associated command-line template for .py files, which is usually the "py.exe" launcher. If so, the launcher's implementation of "#!/usr/bin/env py" will search for "py.exe" in PATH, find itself, and then opt to ignore the shebang and run the default Python. In most cases that should be 3.x, unless the user has configured 2.x as the default via PY_PYTHON or "py.ini".

Ah, so probably #!/usr/bin/env py -3 is what I'm after then

I just checked this using the MSYS2 version of bash and env. There's a snag. bash passes "py -3" as a single argument to "/usr/bin/env", which in turn requires the option "-S" to split this string into multiple command-line arguments. The launcher's implementation could support "#!/usr/bin/env -S", basically by just consuming the "-S", but currently it doesn't.

There's a limit on how much POSIX emulation we want to support, and I think -S is going too far. Especially since py -3 isn't portable anyway - /usr/bin/env python3 is the portable one, with /usr/bin/python3 and /usr/local/bin/python3 as valid though apparently less portable alternatives (and any of those can have a full tag specified on the end, so /usr/bin/python3.13-arm64 if you really want).

Any other paths not matching those templates can be added to the [commands] section in py.ini alongside the py.exe or in the user's %LocalAppData%\py.ini (which is not strictly the right place to put it, but historical reasons win). We don't need to have them all built in.

Unfortunately, other people demanded that usr/bin/env python3 should prioritise the PATH search over the version number, so we can't have perfect compatibility that way. But perhaps all these py -3 cases would be quite happy with a python3 on PATH override?

The launcher's "env" implementation falls back on a virtual "python[X[.Y[.exe]]]" command if the executable name can't be found on PATH. For example:

argv0: C:\Windows\py.exe
version: 3.12.2
# Read 57 bytes from C:\Temp\test.py to find shebang line
Shebang: /usr/bin/env python3
# Did not find python3.exe on PATH
# Reading from C:\Users\user\AppData\Local\py.ini for commands/python3
# Did not find file C:\Users\user\AppData\Local\py.ini
# Reading from C:\Windows\py.ini for commands/python3
# Did not find file C:\Windows\py.ini
# Treating shebang command '3' as 'py -3'

but may as well reopen to further add a special case for using py instead of the more portable #!/usr/bin/python3 (which would be the equivalent of #!py -3)

"py[.exe] [-X[.Y]]" could be reserved as a virtual command like "python[.X[.Y[.exe]]]". Then handling "#!/usr/bin/env py [-X[.Y]]" could default to a virtual command if either "py" isn't found or if the launcher finds itself.

And just for reference, the reason I'm using py (or py -3) instead of python3 is that that's all I have on my PATH:

$ which python
which: no python in ...

$ which python3
which: no python3 in ...

Note that if CMD or PowerShell is used instead of bash, then directly running a .py script will execute the associated command-line template for .py files, which is usually the "py.exe" launcher. If so, the launcher's implementation of "#!/usr/bin/env py" will search for "py.exe" in PATH, find itself, and then opt to ignore the shebang and run the default Python. In most cases that should be 3.x, unless the user has configured 2.x as the default via PY_PYTHON or "py.ini".

That's a good point. So perhaps the logic (if finding itself in the shebang line) needs to be:

If no selection criteria were used by the current invocation of py then parse the shebang line for selection criteria (eg -3 or whatever).
If the user specified any selection criteria then ignore the shebang line and go with what the user selected.

Or maybe the second case should merge the criteria and flag an error if there's a conflict - that is, if the user specifies py -3 script.py and the script says #!py -3.8 then it should probably pick 3.8 rather than 3.11. Maybe with a --ignore-shebang option or something if you really insist on running it with py -2. (I'm assuming this will add complication to the implementation though, as it presumably doesn't have any way to combine criteria currently?)

There's a limit on how much POSIX emulation we want to support, and I think -S is going too far. Especially since py -3 isn't portable anyway - /usr/bin/env python3 is the portable one, with /usr/bin/python3 and /usr/local/bin/python3 as valid though apparently less portable alternatives (and any of those can have a full tag specified on the end, so /usr/bin/python3.13-arm64 if you really want).

But even /usr/bin/env python3 isn't portable to bash on Windows, by default at least, since I don't think Python adds itself to PATH by default (at least my setup doesn't have it, and I don't think I'd have explicitly unset it). So you'd have to explicitly add it (either in Windows - for which I think there's an option in the installer - or in your .bashrc or something) to make that work there. I guess you could also add a script on your bash PATH called python3 that forwards its arguments to py -3, but again, it won't happen out of the box.

Has anyone ever considered/suggested porting py to non-Windows platforms? That would potentially allow py to (eventually) become the cross-platform way to start a Python script... although to mix arguments like -3 with using /usr/bin/env you'd need to use #!/usr/bin/env -S py -3, but I don't think -S is POSIX-specified 🤷‍♂️.

So perhaps the logic (if finding itself in the shebang line) needs to be:

The logic you describe is basically what's already in place, though it's way more complicated because there are 4-5 different ways to set defaults. Shebangs show up somewhere in the middle.

I'm keen to not make this any more complicated, which is why about the only logic I'd consider is detecting py in the shebang and restarting processing with a new command line (and probably explicitly suppressing shebang handling, since we can't be sure that the original shebang line actually specified an option that would suppress it).

Has anyone ever considered/suggested porting py to non-Windows platforms?

Yes, there is also a py command that is similar. However, its implementation is vastly different, and its roadmap is very different. It's not part of the core project, so there's not necessarily any alignment, and we can't direct its behaviour.

Besides, one major difference is that it doesn't do shebangs at all, because it relies on the system to handle them. If you py script-with-shebang.py with that one, you get the default py behaviour. The shebang handling really is a hack to help out Windows users running scripts that were originally intended for POSIX.

But even /usr/bin/env python3 isn't portable to bash on Windows, by default at least, since I don't think Python adds itself to PATH by default

It depends exactly which emulation you're in. Cygwin brings its own Python build, which should include a python3, and if you installed Python from the Windows Store then you'll get a python3 on PATH (though last I heard, Cygwin and anything based on it - which seems to be every Windows Bash implementation - isn't capable of launching Store apps, so that's not much help). The environment is different enough and far enough out of our control that we don't support it, though we try not to do anything to deliberately break it.

Or maybe the second case should merge the criteria and flag an error if there's a conflict - that is, if the user specifies py -3 script.py and the script says #!py -3.8 then it should probably pick 3.8 rather than 3.11.

I prefer the current behavior, which ignores the shebang if the launcher is executed with a "-X[.Y][-32|-64]" or "-V:tag" command-line option.

But even /usr/bin/env python3 isn't portable to bash on Windows, by default at least, since I don't think Python adds itself to PATH by default

Yes, there are issues for POSIX environments on Windows that have to be resolved manually. For Windows, with the shebang "#!/usr/bin/env python[X[.Y][-32|-64]][.exe]", if the launcher can't find the executable in PATH, it instead handles the virtual "python[X[.Y][-32|-64]]" command. Except I just noticed the following bug in the code that ignores ".exe", given the shebang "#!/usr/bin/env python.exe":

argv0: C:\Windows\py.exe
version: 3.12.2
# Read 60 bytes from C:\Temp\test.py to find shebang line
Shebang: /usr/bin/env python.exe
# Did not find python.exe on PATH
# Reading from C:\Users\user\AppData\Local\py.ini for commands/python.exe
# Did not find file C:\Users\user\AppData\Local\py.ini
# Reading from C:\Windows\py.ini for commands/python.exe
# Did not find file C:\Windows\py.ini
# Treating shebang command '.exe' as 'py -.exe'
# Cannot select defaults for tag '.exe'

I just noticed the following bug in the code that ignores ".exe", given the shebang "#!/usr/bin/env python.exe":

Whoops, yeah, that's a bug.

But at the same time, the intent of this limited shebang support isn't to provide an equivalent to what POSIX shells provide. It is really only meant to be limited to a few options that are compatible with other OSs so that developers who want to be compatible between both are able to achieve it.

I've chatted with Brett, who maintains the POSIX launcher, and he said that the shebang handling in that one is entirely limited to the same basic templates as on Windows in order to restrict Python version. Windows has to handle a few more features, but I'm pretty strongly leaning towards narrowing the scope back to what seems to have been the original intent: basic compatibility with POSIX shells.

What this would mean is that any shebang that doesn't work on a typical/reasonable Linux system is out of scope, even on Windows, and is unsupported. It doesn't mean that any shebang that does work there will be supported, but that the intent of shebang parsing is primarily to infer the -V:x.y argument and nothing more. It also doesn't mean that existing support gets immediately removed, but it does draw the line against additional feature creep, such as that proposed in this issue.

I'm not closing this immediately because I expect further discussion, and if/when some consensus is reached then it needs to be documented somewhere, but I'm no longer intending to change the behaviour in order to support #! py -x.y shebangs.

It depends exactly which emulation you're in. Cygwin brings its own Python build, which should include a python3, and if you installed Python from the Windows Store then you'll get a python3 on PATH (though last I heard, Cygwin and anything based on it - which seems to be every Windows Bash implementation - isn't capable of launching Store apps, so that's not much help). The environment is different enough and far enough out of our control that we don't support it, though we try not to do anything to deliberately break it.

I'm using Git for Windows (which I think is built on top of MSYS2). It apparently doesn't come with a /usr/bin/python by default.

Personally for this use case, I'm not too fussed about py foo.py doing anything fancy with the shebang line at all. But if there's a valid Python install I'd rather it ran something rather than failing. That is, if I do have something weird or fancy in the shebang, it's unexpected that py foo.py can fail to even lookup the interpreter. So maybe if py is unable to handle the shebang correctly it should just fall back to whatever processing it would normally do to locate the version of python to run, and prefer running something rather than nothing? (For example, in the case above, the shebang makes it look like it it's looking for an interpreter called py in the CWD, but there isn't such an executable, so fall back to whatever default python install it can find)

Having said that, I'm not actually sure how/why my shebang was working with ./foo.py. From what I can tell, with a shebang of #!py, bash only looks in the current working directory (so you get Unable to create process using 'C:\test\py ./foo.py': The system cannot find the file specified.) but if you use #!py -3 suddenly it does a PATH search and works - not sure if this is documented behaviour or a bug in bash 🤷‍♂️.

Git Bash for Windows is about the most useless version of Bash, unfortunately, simply because it doesn't come with the full environment around it. As I said, we try not to actively break it, but there's very little we can do to make it work.

I suspect that when you ./foo.py in Git Bash, it's processing the shebang line itself, which will follow its own rules. Most likely, it just does a command line replacement and re-runs it, which means you'll get a normal CreateProcess search.¹ Because py.exe's handling of a bare command doesn't work like this, it will not get the same result.

So what likely happens is that the #!py shebang is used to launch py.exe foo.py which then reprocesses the shebang. But #!py -3 shebang (when launched from Bash) launches py.exe -3 foo.py which then ignores the shebang due to the -3, and so succeeds.

Probably the best fix here is for Get Bash to detect the /usr/bin/env python* pattern and remap it to py.exe itself. I'm not sure if that's possible to configure or if it'd need to be an upstream feature, but fundamentally the issue belongs to them.

Alternatively you could configure a command in py.ini that redirects a py shebang to a Python executable, which will override the script directory search we do for unknown commands. This appears to also be undocumented, unfortunately, but adding a [commands] section with py=<path> ought to do it.

Searches the application directory - bash.exe in this case - first, then PATH, with a few added complexities that depend on your system configuration. ↩

I suspect that when you ./foo.py in Git Bash, it's processing the shebang line itself, which will follow its own rules. Most likely, it just does a command line replacement and re-runs it, which means you'll get a normal CreateProcess search. Because py.exe's handling of a bare command doesn't work like this, it will not get the same result.

So what likely happens is that the #!py shebang is used to launch py.exe foo.py which then reprocesses the shebang. But #!py -3 shebang (when launched from Bash) launches py.exe -3 foo.py which then ignores the shebang due to the -3, and so succeeds.

Oof! Yeah that sounds plausible and would explain the behaviour. Switching to #!/usr/bin/env py seems to be the simplest fix for me in this case as I don't need to worry about Linux support or anything. Thanks for the help 😃.

The launcher referred to in this issue has now been removed (#153511), and so this issue is expired. Please use the Python install manager to install Python, and file any issues on the pymanager repo.

Navigation Menu

Repository navigation

Python launcher invoked with command "py" fails to start process #99442

Bug report

Your environment

Linked PRs

11 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	const wchar_t *tail = findArgumentEnd(search->originalCmdLine, -1);
	const wchar_t *end = tail;
	search->restOfCmdLine = tail;
	while (--tail != search->originalCmdLine) {
	if (*tail == L'.' && end == search->restOfCmdLine) {
	end = tail;
	} else if (tail == L'\\' \|\| tail == L'/') {
	++tail;
	break;
	}
	}
	if (tail == search->originalCmdLine && tail[0] == L'"') {
	++tail;
	}
	// Without special cases, we can now fill in the search struct
	int tailLen = (int)(end ? (end - tail) : wcsnlen_s(tail, MAXLEN));

Python launcher invoked with command "py" fails to start process #99442

Description

Bug report

Your environment

Linked PRs

Activity

kai2nenobu commented on Nov 13, 2022

pfmoore commented on Nov 13, 2022

eryksun commented on Nov 13, 2022

pfmoore commented on Nov 18, 2022

zooba commented on Nov 18, 2022

zooba commented on Nov 18, 2022

eryksun commented on Nov 18, 2022

11 remaining items

zooba commented on Mar 4, 2024

eryksun commented on Mar 4, 2024

eryksun commented on Mar 4, 2024

zooba commented on Mar 4, 2024

eryksun commented on Mar 5, 2024

timabroad commented on Mar 5, 2024

timabroad commented on Mar 5, 2024

zooba commented on Mar 5, 2024

eryksun commented on Mar 5, 2024

zooba commented on Mar 5, 2024

timabroad commented on Mar 6, 2024

zooba commented on Mar 6, 2024

Footnotes

timabroad commented on Mar 7, 2024

zooba commented on Jul 20, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions