Skip to content

py.exe launcher ignores [commands] from py.ini #100247

Closed
Closed
py.exe launcher ignores [commands] from py.ini#100247
Assignees
zooba
Labels
OS-windowstype-bugAn unexpected behavior, bug, or error

Description

@cwalther
cwalther
opened on Dec 14, 2022

Bug report

This is a regression between Python 3.11.0 and 3.11.1. Since 46a3cf4 (88297e2, #98692), py.exe no longer observes the “custom commands” mapping under [commands] in py.ini defining additional “virtual commands” (unless the virtual command starts with the same prefix as one of the four predefined virtual commands), but directly tries to launch the virtual command as an executable.

Steps to reproduce:

%WINDIR%\py.ini:

[commands]
/opt/something/bin/my-python2=C:\something\python27\python.exe

test.py:

#!/opt/something/bin/my-python2
import sys
print('hello from', sys.executable)
%WINDIR%\py.exe test.py

Expected result (and actual up to 3.11.0):

('hello from', 'C:\\something\\python27\\python.exe')

Actual result:

Unable to create process using 'C:\opt\something\bin\my-python2  test.py': The system cannot find the file specified.

I seem to be able to fix this as follows, which satisfies the existing tests, however this code is such a complex tangle of special cases that I have no idea whether it is the right thing to do. (The idea is that the loop over shebangTemplates should always find exactly one match, which was previously (before the regression) ensured by the empty template, so that _findCommand() is always called. Checking for tmpl != &shebangTemplates[0] is needed to satisfy test_recursive_search_path, however it might exclude too much – maybe searchPath() should instead report explicitly that it skipped a recursive call.)

diff --git a/PC/launcher2.c b/PC/launcher2.c
index 9b3db04aa4..ad313c10f3 100644
--- a/PC/launcher2.c
+++ b/PC/launcher2.c
@@ -1001,19 +1001,13 @@ checkShebang(SearchInfo *search)
         L"/usr/bin/env ",
         L"/usr/bin/",
         L"/usr/local/bin/",
-        L"python",
+        L"",
         NULL
     };
 
     for (const wchar_t **tmpl = shebangTemplates; *tmpl; ++tmpl) {
         if (_shebangStartsWith(shebang, shebangLength, *tmpl, &command)) {
             commandLength = 0;
-            // Normally "python" is the start of the command, but we also need it
-            // as a shebang prefix for back-compat. We move the command marker back
-            // if we match on that one.
-            if (0 == wcscmp(*tmpl, L"python")) {
-                command -= 6;
-            }
             while (command[commandLength] && !isspace(command[commandLength])) {
                 commandLength += 1;
             }
@@ -1052,18 +1046,20 @@ checkShebang(SearchInfo *search)
                     debug(L"# Treating shebang command '%.*s' as 'py'\n",
                         commandLength, command);
                 }
+            } else if (tmpl != &shebangTemplates[0]) {
+                // Unrecognised commands are joined to the script's directory and treated
+                // as the executable path
+                return _useShebangAsExecutable(search, shebang, shebangLength);
             } else {
                 debug(L"# Found shebang command but could not execute it: %.*s\n",
                     commandLength, command);
             }
             // search is done by this point
-            return 0;
+            break;
         }
     }
 
-    // Unrecognised commands are joined to the script's directory and treated
-    // as the executable path
-    return _useShebangAsExecutable(search, shebang, shebangLength);
+    return 0;
 }

Your environment

  • CPython versions tested on: 3.9, 3.10, 3.11.0 (good), 3.11.1 (bad)
  • Operating system and architecture: Windows 10 Pro 10.0.19043.2006 AMD64

Linked PRs

Activity

  1. cwalther
    added
    type-bugAn unexpected behavior, bug, or error
    on Dec 14, 2022
  2. AlexWaygood
    added
    OS-windows
    on Dec 14, 2022
  3. AlexWaygood
    assigned
    zooba
    on Dec 14, 2022

  4. cwalther commented on Dec 14, 2022

    @cwalther
    cwalther
    on Dec 14, 2022
    ContributorAuthor

    Thinking about it some more, I question whether the call to _findCommand() should be inside the loop over shebangTemplates at all. That means that with a configuration of

    [commands]
something/apython=C:\something\a\python.exe
/usr/bin/something/apython=C:\something\b\python.exe

    a script with shebang line #!/usr/bin/something/apython will match the first line and be executed with a\python.exe. I see nothing of that sort specified in PEP 397. I would expect it to match the second line and choose b\python.exe. To me, the order of bullet points in Shebang line parsing says that _findCommand should come before before checking for virtual commands with one of the four known prefixes and should consider the entire command.

    (However it seems that this has already worked this way in the 3.9 and 3.10 launcher.)

  5. zooba commented on Jan 10, 2023

    @zooba
    zooba
    on Jan 10, 2023
    Member

    I think what probably needs to happen here (writing my logic down so I can double check once this headache is over) is that the templates should require python (i.e. /usr/bin/python..., not /usr/bin/*), and then the fallback should check for custom commands before assuming it's an executable.

    I agree that the original PEP suggests the custom command should be checked first taking the entire first argument from the shebang, but as pointed out, that's not how it worked in the past. Implementation wins over the original propsal here (since the docs don't even mention this feature).

  6. zooba
    added 2 commits that reference this issue on Jan 11, 2023

    pythongh-100247: Fix py.exe launcher not using entire shebang command…

    a5609e6

    gh-100247: Fix py.exe launcher not using entire shebang command for f…

    468c3bf
  7. zooba
    closed this as completedon Jan 13, 2023
  8. miss-islington
    added 2 commits that reference this issue on Jan 13, 2023

    pythongh-100247: Fix py.exe launcher not using entire shebang command…

    799fec9

    gh-100247: Fix py.exe launcher not using entire shebang command for f…

    6492492

  9. cwalther commented on Jan 13, 2023

    @cwalther
    cwalther
    on Jan 13, 2023
    ContributorAuthor

    Thanks! I’ll give it a test when I’m back at my work Windows desktop, probably some time next week.

    This bit worries me a little, though:

    The name of the command must be a single argument (no spaces),
    and the value substituted is the full path to the executable (no arguments
    may be added).

    If that is true, then there may still be a regression compared to 3.11.0. The actual example we are using in our shipping product to make sure our scripts launch with our own Python installations on both Linux and Windows (which worked with the 3.9 launcher we ship but broke when users installed a 3.11.1 launcher) is

    #!/opt/indel/bin/indel-python2 -u

    with py.ini

    [commands]
/opt/indel/bin/indel-python2=C:\indel\share\python\python27\python.exe
/opt/indel/bin/indel-pythonw2=C:\indel\share\python\python27\pythonw.exe
/opt/indel/bin/indel-python3=C:\indel\share\python\python3\amd64\python.exe
/opt/indel/bin/indel-pythonw3=C:\indel\share\python\python3\amd64\pythonw.exe
/opt/indel/bin/indel-python=C:\indel\share\python\python3\amd64\python.exe
/opt/indel/bin/indel-pythonw=C:\indel\share\python\python3\amd64\pythonw.exe

    where on Linux /opt/indel/bin/indel-python* are symlinks to the appropriate interpreter while the Windows paths are the actual installation locations.

    (I’m not 100% certain we really need the -u, but it could be because some scripts relay output from other processes that may include progress indicators that should not be line-buffered.)

  10. zooba commented on Jan 13, 2023

    @zooba
    zooba
    on Jan 13, 2023
    Member

    Those are all fine. What it means is that you can't use /opt/indel/bin/indel-python2 -u as the command name - it'll stop looking at the first space.

    Also, you can't use C:\indel\share\python\python27\python.exe -u as the right-hand side of the setting, because the whole thing is passed as the executable name, and is quoted when put into the command line.

  11. cwalther commented on Jan 13, 2023

    @cwalther
    cwalther
    on Jan 13, 2023
    ContributorAuthor

    Ah, OK. That makes sense. “name of the command” refers to the the py.ini entry, not to the shebang line.

  12. zooba
    added 2 commits that reference this issue on Jan 16, 2023

    pythongh-100247: Improve documentation for custom shebang commands in…

    362e61a

    gh-100247: Improve documentation for custom shebang commands in py.ex…

    30753c3
  13. miss-islington
    added 2 commits that reference this issue on Jan 16, 2023

    pythongh-100247: Improve documentation for custom shebang commands in…

    fe540fe

    gh-100247: Improve documentation for custom shebang commands in py.ex…

    a3510c3

  14. cwalther commented on Jan 20, 2023

    @cwalther
    cwalther
    on Jan 20, 2023
    ContributorAuthor

    I have tested it now and it works perfectly for our purposes. Thanks again!

    It even fixes the paradoxical behavior I mentioned in #100247 (comment), which makes me happy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

OS-windowstype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions