use os.posix_spawn in subprocess

@gpshead

BPO	35537
Nosy	@gpshead, @vstinner, @serhiy-storchaka, @koobs, @izbyshev, @pablogsal, @miss-islington, @nanjekyejoannah, @kevans91
PRs	bpo-35537: use os.posix_spawn in subprocess #11242 bpo-35537: subprocess uses os.posix_spawn in some cases #11452 bpo-35537: subprocess can use posix_spawn with pipes #11575 bpo-35537: subprocess can now use os.posix_spawnp #11579 Revert "bpo-35537: subprocess can now use os.posix_spawnp (GH-11579)" #11582 bpo-35537: Add setsid parameter to os.posix_spawn() and os.posix_spawnp(). #11608 bpo-35537: Document posix_spawn() change in subprocess #11668 bpo-35537: Fix docstring typos in _use_posix_spawn() #11684 subprocess: close pipes/fds by using ExitStack #11686 bpo-35537: Skip test_start_new_session() of posix_spawn #11718 [WIP] bpo-35537: Fix function name in os.posix_spawnp() errors #11719 bpo-35537: Rewrite setsid test for os.posix_spawn #11721 bpo-35537: Change subprocess to use posix_spawnp instead of posix_spawn #11917 [3.8] bpo-35537: Rewrite setsid test for os.posix_spawn (GH-11721) #14093 bpo-35537: Add a comment to _Py_RestoreSignals() #18792
Files	subprocess_bench.py subprocess_bench_stdout.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2019-06-05.08:27:39.280>
created_at = <Date 2018-12-19.16:47:59.385>
labels = ['3.8', 'type-feature', 'library']
title = 'use os.posix_spawn in subprocess'
updated_at = <Date 2020-03-05.16:41:09.872>
user = 'https://github.com/nanjekyejoannah'

bugs.python.org fields:

activity = <Date 2020-03-05.16:41:09.872>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2019-06-05.08:27:39.280>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2018-12-19.16:47:59.385>
creator = 'nanjekyejoannah'
dependencies = []
files = ['48011', '48057']
hgrepos = []
issue_num = 35537
keywords = ['patch', 'patch', 'patch']
message_count = 61.0
messages = ['332151', '332204', '332210', '332212', '332213', '332216', '332217', '332219', '332222', '332223', '332235', '332236', '332237', '332238', '332239', '332266', '332270', '332272', '332393', '332565', '333123', '333126', '333127', '333131', '333571', '333629', '333646', '333656', '333657', '333738', '333739', '333740', '333744', '333745', '333749', '333771', '333794', '333795', '333799', '333800', '333803', '333805', '333820', '333825', '333839', '333889', '333902', '333994', '334268', '334269', '334288', '334292', '334305', '334337', '334661', '334662', '334685', '340838', '344683', '345617', '345621']
nosy_count = 9.0
nosy_names = ['gregory.p.smith', 'vstinner', 'serhiy.storchaka', 'koobs', 'izbyshev', 'pablogsal', 'miss-islington', 'nanjekyejoannah', 'kevans']
pr_nums = ['11242', '11452', '11575', '11579', '11582', '11608', '11668', '11684', '11686', '11718', '11719', '11721', '11917', '14093', '18792']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue35537'
versions = ['Python 3.8']

On Linux, posix_spawn() uses vfork() instead of fork() which blocks
the parent process. The child process executes exec() early and so we
don't pay the price of duplicating the memory pages (the thing which
tracks memory pages of a process).

On macOS, posix_spawn() is a system call, so the kernel is free to use
fast-paths to optimize it as they want.

posix_spawn() is faster but it's also safer: it allows us to do a lot of
"actions" before exec(), before executing the new program. For
example, you can close files and control signals. Again, on macOS,
these actions are "atomic" since it's a system call. On Linux, glibc uses a very good implementation which has no race condition.

Currently, Python uses a C extension _posixsubprocess which more or
less reimplements posix_spawn(): close file descriptors, make some
file descriptors inheritable or not, etc. It is very tricky to write
correct code: code run around fork() is very fragile. In theory, the
POSIX specification only allows to use "async-signal-safe" functions
after fork()...

So it would be great to avoid _posixsubprocess whenever possible for
(1) speed (2) correctness.

"posix_spawn() is faster"

This assumption needs a benchmark!

I ran a benchmark on Linux (Fedora 29, Linux kernel 4.19.9-300.fc29.x86_64, glibc 2.28) using attached subprocess_bench.py.

I rebased the PR 11242 on master and run the benchmark in virtual environment:

git co pr/11242
# current branch: PR 11242
make distclean
./configure
make
./python -m venv env
env/bin/python -m pip install perf
env/bin/python subprocess_bench.py -v -o posix_spawn.json
git co master
env/bin/python subprocess_bench.py -v -o fork_exec.json
env/bin/python -m perf compare_to fork_exec.json posix_spawn.json

The result is quite explicit: the PR makes subprocess.Popen 61x faster!!!

$ env/bin/python -m perf compare_to fork_exec.json posix_spawn.json 
Mean +- std dev: [fork_exec] 27.1 ms +- 0.4 ms -> [posix_spawn] 447 us +- 163 us: 60.55x faster (-98%)

That's the best case:

The parent process (Python) allocated 2 GiB of memory: that's not uncommon for large application. On OpenStack for example, it's common that a process takes more than 1 GiB.
The child process has a short execution time and allocates few memory.

On my config (Fedora 29, Linux kernel 4.19.9-300.fc29.x86_64, glibc 2.28), os.posix_spawn() uses vfork():

$ strace -o trace ./python -c 'import subprocess; subprocess.run(["/bin/true"], close_fds=False, restore_signals=False)'
$ grep clone trace
clone(child_stack=0x7fab28ac0ff0, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 23073

I guess that the 61x speedup mostly comes from vfork().

See also bpo-34663 for previous discussion about vfork() and os.posix_spawn(). In short, the glibc is smart and detects when vfork() can be used or not.

Benchmark on macOS Mojave (10.14.2), x86_64:

$ ./python.exe -m perf compare_to fork_exec.json posix_spawn.json 
Mean +- std dev: [fork_exec] 2.79 ms +- 0.06 ms -> [posix_spawn] 1.64 ms +- 0.03 ms: 1.70x faster (-41%)

The speedup is "only" 1.7x faster, it's less impressive. Maybe fork()+exec() is more efficient on macOS than on Linux.

Benchmark on FreeBSD:

$ env/bin/python -m perf compare_to fork_exec.json posix_spawn.json 
Mean +- std dev: [fork_exec] 52.8 ms +- 4.7 ms -> [posix_spawn] 499 us +- 45 us: 105.91x faster (-99%)

Wow! 106x faster!

See also:

"How a fix in Go 1.9 sped up our Gitaly service by 30x" (hint: posix_spawn)
https://about.gitlab.com/2018/01/23/how-a-fix-in-go-19-sped-up-our-gitaly-service-by-30x/

fork/exec vs. posix_spawn benchmarks:
https://github.com/rtomayko/posix-spawn#benchmarks

We need to add tests for all corner cases. I.e. disabling any of conditions for posix_spawn should cause the failure.

We need to do also something with PyOS_BeforeFork/PyOS_AfterFork_Child/PyOS_AfterFork_Parent.

We need to add tests for all corner cases. I.e. disabling any of conditions for posix_spawn should cause the failure.

Do you mean running tests twice, one with posix_spawn(), one without?

We need to do also something with PyOS_BeforeFork/PyOS_AfterFork_Child/PyOS_AfterFork_Parent.

Can you please elaborate? posix_spawn() may or may not use fork() internally.

Do you mean running tests twice, one with posix_spawn(), one without?

I mean that after writing tests they can be tested manually by disabling conditions for posix_spawn one by one. I.e. some tests should fail if remove "stdout is None" and some tests should fail if remove "not close_fds", etc.

Can you please elaborate? posix_spawn() may or may not use fork() internally.

_posixsubprocess.fork_exec() calls PyOS_BeforeFork/PyOS_AfterFork_Child/PyOS_AfterFork_Parent. If use os.posix_spawn(), these calls will be omitted. This is a behavior change. We should either call these functions manually before/after os.posix_spawn() (but I do not know what to do with PyOS_AfterFork_Child), or disable using os.posix_spawn() if non-trivial callbacks were added, or use other methods for calling registered callbacks. But the stdlib already adds callbacks in the threading and random modules.

Serhiy, PyOS_* functions are called only if preexec_fn != None. But it will never be possible to implement support for preexec_fn (and some other subprocess features, e.g. close_fds) if processes are run via posix_spawn, so I don't see why anything should be done with those functions.

Oh, nice! So this is not an obstacle for using os.posix_spawn().

Victor and Joannah, thanks for working on adding vfork() support to subprocess. Regarding speedups in the real world, I can share a personal anecdote. Back at the time when AOSP was built with make (I think it was AOSP 5) I've observed ~2x slowdown (or something close to that) if fork() is used instead of vfork() in make. This is the slowdown of the *whole* build (not just process creation time), and it's dramatic given the amount of pure computation (compilation of C++/Java code) involved in building AOSP. The underlying reason was that make merged all AOSP subproject makefiles into a single gigantic one, so make consumed more than 1 GB of RAM, and each shell invocation in recipes resulted in copying a large page table.

That said, I'm not sure that the chosen approach of adding posix_spawn() "fast paths" for particular combinations of arguments is an optimal way to expose vfork() benefits to most users. For example, close_fds is True by default in subprocess, and I don't think there is a reliable way to use posix_spawn() in this case. So users would need to use "magic" combinations of arguments to subprocess.Popen to benefit from vfork(). Another example is "cwd" parameter: there is no standard posix_spawn file action to change the working directory in the child, so such a seemingly trivial operation would trigger the "slow" fork-exec path.

Another approach would be to use vfork-exec instead fork-exec in _posixsubprocess. I know that previous discussions were resolved against vfork(), but I'm still not convinced and suggest to reevaluate those decisions again. Some arguments:

While POSIX contain scare-wording forbidding to do pretty anything in a vfork-spawned child, real systems where posix_spawn() is not a system call call vfork() and then execute all specified actions in the child. Cases where vfork() is avoided seem to be a quality-of-implementation issue, not a fundamental issue (for example, until recently glibc used fork() in some cases because of heap memory allocations, but they could be avoided). In practice, there is no problem with calling at least async-signal-safe functions in vfork-children, and there is no technical reason why there would be any problem.
_posixsubprocess is already very careful and calls only async-signal-safe functions in almost all cases (an obvious exception is preexec_fn, which is discouraged anyway).
Our use case for vfork() is restricted, so some concerns don't apply to us. For example, the setuid() problem outlined in Rich Felker's post (https://ewontfix.com/7) doesn't seem to apply. The problem with signal handlers from the same post should be possible to avoid except for preexec_fn, but we'd have to fallback to fork() in this case anyway due to memory allocation.
In the standard example of fork() unsafety in multithreaded processes (state of memory-based locks is copied to the child, and there could be nobody to unlock them), vfork() is *safer* than fork() because it still shares memory with the parent, and all threads other than the parent one are running. In particular, it should be perfectly safe to use any memory allocator that protects its state with something like futexes and atomics in a vfork-child even if other threads of the parent are using it concurrently.
Other language runtimes are already using vfork(). Java has been doing it for ages, and Victor referenced Go.

Some possible counter-arguments:

We seem to be reimplementing posix_spawn(). In fact, due to (2) above, we can fully reuse the existing fork-exec code, with additional tweaks that doesn't seem hard at this point.
My points above are based on Linux, but I don't know much about other Unix-likes, so in theory additional complications could exist. I can't, however, imagine any fundamental complication.

In the end, I think that migrating subprocess to vfork-exec would have more impact for users than adding "fast paths" and have consistent performance regardless of subprocess.Popen arguments (with few exceptions). Please consider it. Thanks!

New changeset f6243ac by Victor Stinner in branch 'master':
bpo-35537: subprocess can use posix_spawn with pipes (GH-11575)
f6243ac

Another problem with posix_spawn() on glibc: it doesn't report errors to the parent process when run under QEMU user-space emulation and Windows Subsystem for Linux. This is because starting with commit [1] (glibc 2.25) posix_spawn() relies on address space sharing semantics of clone(CLONE_VM) to report errors, but it's not implemented in QEMU and WSL, so clone(CLONE_VM) and vfork() behave like fork(). See also [2], [3].

This can be easily demonstrated:
$ cat test.c
#include <spawn.h>

int main(int argc, char *argv[], char *envp[]) {
    return posix_spawn(0, "non-existing", 0, 0, argv, envp);
}

$ gcc test.c
$ ./a.out
$ echo $?
2
$ aarch64-linux-gnu-gcc -static test.c
$ qemu-aarch64 ./a.out
$ echo $?
0

There is no problem with musl (it doesn't rely on address space sharing).

[1] https://sourceware.org/git/?p=glibc.git;a=commit;h=4b4d4056bb154603f36c6f8845757c1012758158
[2] https://bugs.launchpad.net/qemu/+bug/1673976
[3] https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg04890.html

Another problem with posix_spawn() on glibc: it doesn't report errors to the parent process when run under QEMU user-space emulation and Windows Subsystem for Linux. This is because starting with commit [1] (glibc 2.25) posix_spawn() relies on address space sharing semantics of clone(CLONE_VM) to report errors, but it's not implemented in QEMU and WSL, so clone(CLONE_VM) and vfork() behave like fork(). See also [2], [3].

Oh, you know a lot of stuff about vfork and posix_spawn, you are impressive :-)

Is sys.platform equal to 'linux' on WSL? Sorry, I don't know WSL. If it's equal, is it possible to explicitly exclude WSL in the subprocess test, _use_posix_spawn()?

For QEMU, I was very surprised that a full VM wouldn't implement vfork. So I tried Fedora Rawhide and I didn't notice the bug that you described.

---

$ cat test.c
#include <spawn.h>

int main(int argc, char *argv[], char *envp[]) {
    return posix_spawn(0, "non-existing", 0, 0, argv, envp);
}

$ gcc test.c
$ ./a.out
$ echo $?
2

In the VM, I get exit status 2 as expected.

You wrote:

---

$ aarch64-linux-gnu-gcc -static test.c
$ qemu-aarch64 ./a.out
$ echo $?
0

So the bug only affects "QEMU User Space". I never used that before. I understand that it's a bug in the glibc and in "QEMU User Emulation" which doesn't implement vfork "properly".

There is no problem with musl (it doesn't rely on address space sharing).

I'm not interested to support musl at this point because I don't see any obvious way to detect that we are running musl nor get the musl version.

The history of this issue shows the posix_spawn() only works in some very specific cases, so we have to be careful and only opt-in for specific libc, on specific platform with specific libc version (read at runtime if possible).

I discussed the following issue with Florian Weimer (who works on the glibc):
---

$ cat test.c
#include <spawn.h>

int main(int argc, char *argv[], char *envp[]) {
    return posix_spawn(0, "non-existing", 0, 0, argv, envp);
}

$ aarch64-linux-gnu-gcc -static test.c
$ qemu-aarch64 ./a.out
$ echo $?
0

While the parent gets a "success", in subprocess we get the pid and later read the exit status of the child process. Even if posix_spawn() doesn't report a failure, waitpid() will still report a failure. It's a subtle behavior change. I'm not sure yet if it's acceptable for Python in term of semantics:

without posix_spawn(), subprocess.Popen constructor raises FileNotFoundError: the parent knows that the execution failed because the program doesn't exist
with posix_spawn() with the vfork bug, subprocess.Popen succeed but Popen.wait() returns something like exit code 127. The parent doesn't know if the child process explcitly used exit(127) or if execv() failed.

Does the difference matters? The bug only occurs in some very specific cases:

WSL
QEMU User Emulation

Are these use cases common enough to block the whole idea of using posix_spawn() on Linux. I don't think so. I really want to use posix_spawn() for best performances! Moreover, I expect that glibc implementation of posix_spawn() is safer than Python _posixsubprocess. The glibc has access to low-level stuff like it's internal signals, cancellation points, etc. _posixsubprocess is more generic and doesn't worry about such low-level stuff, whereas they might cause bad surprised.

I wrote PR 11668 to propose to *document* the subtle behavior change when posix_spawn() is used on QEMU User Emulation and WSL platforms, rather than trying to opt-out for posix_spawn() on these platforms.

Is sys.platform equal to 'linux' on WSL? Sorry, I don't know WSL. If it's equal, is it possible to explicitly exclude WSL in the subprocess test, _use_posix_spawn()?

I don't have immediate access to WSL right now, but I'll try to get one and investigate what we have there wrt. posix_spawn() and vfork().

So the bug only affects "QEMU User Space". I never used that before.

Yes, I was specifically referring to QEMU user-space. One use case that heavily relies on it is Tizen. Its packages are built in a chroot jail containing the build environment with binaries native to the target architecture, making an illusion that packages are built on the target system and are not cross-compiled. So the binaries are run under QEMU user-space emulation. But in reality, because of unacceptable performance of binary translation, many frequently-used binaries, like coreutils and compilers, are replaced with host-native binaries in a way transparent to the build system (so it has no idea whether it runs host-native or target-native binaries).

Does the difference matters? The bug only occurs in some very specific cases:

WSL

QEMU User Emulation

Are these use cases common enough to block the whole idea of using posix_spawn() on Linux. I don't think so. I really want to use posix_spawn() for best performances! Moreover, I expect that glibc implementation of posix_spawn() is safer than Python _posixsubprocess. The glibc has access to low-level stuff like it's internal signals, cancellation points, etc. _posixsubprocess is more generic and doesn't worry about such low-level stuff, whereas they might cause bad surprised.

It's true that a C library is in a better position to implement something like posix_spawn(), but I still think that vfork()/exec() is worth to consider at least on Linux. See bpo-35823, which should also work under QEMU user-mode and WSL (but needs testing).

New changeset 80c5dfe by Victor Stinner (Joannah Nanjekye) in branch 'master':
bpo-35537: Add setsid parameter to os.posix_spawn() and os.posix_spawnp() (GH-11608)
80c5dfe

New changeset 1e39b83 by Victor Stinner in branch 'master':
bpo-35537: Skip test_start_new_session() of posix_spawn (GH-11718)
1e39b83

New changeset 325e4ba by Victor Stinner in branch 'master':
bpo-35537: Fix function name in os.posix_spawnp() errors (GH-11719)
325e4ba

New changeset d7befad by Victor Stinner in branch 'master':
bpo-35537: Document posix_spawn() change in subprocess (GH-11668)
d7befad

Python 3.8 beta 1 is released with this feature. I documented the change. Thnaks everybody who was involved to make this possible. I close the issue. Open new issues for follow-up.

New changeset 5884043 by Victor Stinner in branch 'master':
bpo-35537: Rewrite setsid test for os.posix_spawn (GH-11721)
5884043

New changeset e696b15 by Miss Islington (bot) in branch '3.8':
bpo-35537: Rewrite setsid test for os.posix_spawn (GH-11721)
e696b15

Navigation Menu

Repository navigation

use os.posix_spawn in subprocess #79718

37 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

use os.posix_spawn in subprocess #79718

Description

Activity

nanjekyejoannah commented on Dec 19, 2018

vstinner commented on Dec 20, 2018

vstinner commented on Dec 20, 2018

vstinner commented on Dec 20, 2018

vstinner commented on Dec 20, 2018

serhiy-storchaka commented on Dec 20, 2018

vstinner commented on Dec 20, 2018

serhiy-storchaka commented on Dec 20, 2018

izbyshev commented on Dec 20, 2018

serhiy-storchaka commented on Dec 20, 2018

izbyshev commented on Dec 20, 2018

37 remaining items

vstinner commented on Jan 23, 2019

izbyshev commented on Jan 23, 2019

vstinner commented on Jan 24, 2019

vstinner commented on Jan 24, 2019

vstinner commented on Jan 24, 2019

izbyshev commented on Jan 25, 2019

vstinner commented on Feb 1, 2019

vstinner commented on Feb 1, 2019

vstinner commented on Feb 1, 2019

vstinner commented on Apr 25, 2019

vstinner commented on Jun 5, 2019

vstinner commented on Jun 14, 2019

miss-islington commented on Jun 14, 2019

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions