Repository navigation
subprocess PATH semantics and portability #52803
Description
| BPO | 8557 |
|---|---|
| Nosy | @mark-summerfield, @bitdancer, @briancurtin, @dabrahams, @eryksun, @henryiii, @asottile |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2010-04-28.10:44:53.142>
labels = ['type-feature', '3.8', '3.9', '3.10', 'docs']
title = 'subprocess PATH semantics and portability'
updated_at = <Date 2021-05-13.20:49:33.930>
user = 'https://github.com/dabrahams'bugs.python.org fields:
activity = <Date 2021-05-13.20:49:33.930>
actor = 'Henry Schreiner'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2010-04-28.10:44:53.142>
creator = 'dabrahams'
dependencies = []
files = ['17180']
hgrepos = []
issue_num = 8557
keywords = []
message_count = 21.0
messages = ['104422', '104429', '104437', '104527', '104611', '104646', '104647', '104738', '104752', '104766', '104902', '104904', '104908', '104909', '104912', '104925', '105867', '262382', '262399', '320098', '388155']
nosy_count = 10.0
nosy_names = ['mark', 'r.david.murray', 'brian.curtin', 'docs@python', 'dabrahams', 'RubyTuesdayDONO', 'eryksun', 'pepalogik', 'Henry Schreiner', 'Anthony Sottile']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue8557'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
Activity
dabrahams commentedon Apr 28, 2010
On POSIX systems, the PATH environment variable is always used to
look up directory-less executable names passed as the first argument to Popen(...), but on Windows, PATH is only considered when shell=True is also passed.
Actually I think it may be slightly weirder than that when
shell=False, because the following holds for me:
C:\>rem ##### Prepare minimal PATH #####
C:\>set "PATH=C:\Python26\Scripts;C:\Python26;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem"
C:\>rem ##### Prepare a minimal, clean environment #####
C:\>virtualenv --no-site-packages e:\zzz
New python executable in e:\zzz\Scripts\python.exe
Installing setuptools................done.
That is, it looks like the environment at the time Python is invoked is what counts unless I pass shell=True. I don't even seem to be able to override this behavior by changing os.environ: you can clear() it and pass env={} and subprocess.Popen(['python']) still succeeds.
This is a very important problem for portable code and one that took me hours to suss out. I think:
a) the current behavior needs to be documented
b) it needs to be fixed if possible
c) otherwise, shell=True should be the default
dabrahams commentedon Apr 28, 2010
It's worse than I thought; there isn't even one setting for shell that works everywhere. This is what happens on POSIX (tested on Mac and Ubuntu):
mark-summerfield commentedon Apr 28, 2010
IMO there's another problem with subprocess portablity---the lack of control over encodings: see bpo-6135.
bitdancer commentedon Apr 29, 2010
Changing the default value of shell is not an option anyway.
The behavior you describe is exactly what one should expect: the environment in which the executable is located is the environment of the process calling Popen, not the environment passed to Popen. The environment passed to Popen is the environment in which the subprocess executes. When using shell=True, this is the environment in which the shell executes, and the *shell* then looks up the executable in that new environment. As far as I know this behavior is the same on both Windows and Unix, and thus is not a portability issue. (How the shell works can be a portability issue, though.)
I'm not sure that this needs to be documented explicitly, as it is a logical consequence of how subprocesses work, but if you want to suggest a doc update I'll take a look at it.
I suspect your Unix example is about the fragility of the rules for computing sys.executable (there's an issue in the tracker about that...you may get a different result on trunk), but I haven't checked it.
[-]subprocess portability issue[/-][+]subprocess PATH semantics[/+]dabrahams commentedon Apr 30, 2010
I wrote a Python script (enclosed) to methodically test how these things work, that doesn't rely on peculiarities of sys.executable. The tests did reveal some notable differences on *nix and 'doze:
When shell=False on windows you must launch the process using a full filename (e.g. "foo.exe", not just "foo", pass --invoke-filename to the script to enable that). This may seem obvious to you, but for me it was surprising that one executable lookup function (looking in PATH) is in effect but not the other (extending unqualified executable names). This should be spelled out in the docs.
On *nix, with shell=False and the executable is neither in the PATH in the environment at the time of Python's launch nor in os.environ at the time of Popen, passing Popen an explicit env whose PATH includes the executable is enough to cause it to be found. Not so on 'doze.
On 'doze, when the executable is in the PATH of os.environ but not in that of Popen's explicit env argument, even with shell=False, no Exception is raised (but returncode is nonzero)
bitdancer commentedon Apr 30, 2010
Well, it seems I was mistaken when I thought I knew how this worked :)
Checking the os.exec documentation linked from the subprocess page, I see that when an environment is supplied PATH is indeed checked in it. The documentation for CreateProcess, however, indicates that PATH is ignored, and any extension must be supplied explicitly.
At the very least the docs should be updated to clarify that execvpe is used when an environment is passed on posix, and to link to the CreateProcess docs. A discussion of PATH could perhaps be put in a note or footnote (probably footnote, there are enough notes already in those docs!)
I'm not sure how one creates a good portability story out of these pieces. It doesn't seem as though there is any way to harmonize the two, since we are dealing with the semantics of system calls over which we have no control.
For reference, here is (a?) link to CreateProcess docs that I found via Google:
http://msdn.microsoft.com/en-us/library/ms682425(VS.85).aspx
It doesn't look like the kind of link that one could trust to be stable, though, so I'm not sure if we should include it in the docs.
I'm adding Brian Curtin as nosy to see if he knows whether or not there are permalink-like links to the relevant Microsoft documentation that we could use.
[-]subprocess PATH semantics[/-][+]subprocess PATH semantics and portability[/+]briancurtin commentedon Apr 30, 2010
You could take the "(VS8.5)" part out of the link which will give the latest version, which may not always be the relevant version (although I doubt this specific API would change).
That's about the best permalink-like feature you'll find, but overall, leaving the link as-is is pretty safe in my experience.
dabrahams commentedon May 1, 2010
@r.david.murray: did you try running my test? I think it shows that we are pretty darned close to fully portable. I believe we could fix Popen to make it fully portable pretty easily. In fact, there may be a pure-python fix. Documenting the differences would also not be hard. I would discourage you from relying *solely* on a description such as "uses execvpe on POSIX" to describe the semantics. Aside from being a nonportable description, it doesn't help anyone who isn't familiar with the POSIX system calls.
bitdancer commentedon May 1, 2010
I didn't run the script. I have now, but I'm not clear from its output what each test is actually doing, and don't really have the time to figure it out from the code right now.
I think it is probably more efficient to just ask you what your suggestion is for making things more portable?
As for the docs, the docs link to the os.exec python docs, which explain the PATH semantics. Linking to the Microsoft documentation would equivalently explain the Windows semantics. An explicit footnote discussing the differences in PATH behavior in the subprocess context would probably be helpful.
dabrahams commentedon May 2, 2010
I've uploaded a new probe.py that contains a win32 Popen wrapper that I think acts just like *nix's Popen w.r.t. PATH and environment (pass --fix to demonstrate). I suggest using this or an equivalent wrapper for Win32, and documenting the fact that with shell=False, filename extensions need to be supplied explicitly on windows.
12 remaining items
eryksun commentedon Mar 5, 2021
The Popen() docs begin by explaining that it has "os.execvp()-like" behavior in POSIX and uses CreateProcess() in Windows. Personally, I do not think it's proper for Python's documentation to discuss details of how CreateProcess() handles lpCommandLine (args), lpApplicationName (executable), lpCurrentDirectory (cwd), and lpEnvironment (env). So maybe all this needs is to clearly map Popen() parameters to the corresponding CreateProcess() parameters.
If Popen() implements a parameter on its own, then it makes sense to me to document the behavior. For example, in POSIX the behavior of
cwdis implemented by Popen(), and documented as follows:This claim is not always true in POSIX since a base filename without a slash in it, which is a relative path, is not searched for in the current directory unless "." is in PATH. But the main problem with the above sentence is the lack of a disclaimer that it only applies to POSIX. In Windows,
cwdis passed through directly as the lpCurrentDirectory of CreateProcess(). This parameter sets the working directory of the child process and has nothing to do with searching for an executable parsed out of lpCommandLine or resolving a relative path in lpApplicationName. It may affect the result with shell=True, but even in that case there are caveats. Regardless, Python doesn't do anything withcwdin Windows except pass it to CreateProcess(), so the cwd -> lpCurrentDirectory parameter mapping is all there is to document.