Repository navigation
expose posix_spawn(p) #64303
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/gpshead'
closed_at = <Date 2018-09-09.11:40:06.390>
created_at = <Date 2014-01-01.18:13:20.774>
labels = ['extension-modules', 'type-feature', '3.8']
title = 'expose posix_spawn(p)'
updated_at = <Date 2019-01-07.00:47:02.950>
user = 'https://github.com/benjaminp'bugs.python.org fields:
activity = <Date 2019-01-07.00:47:02.950>
actor = 'vstinner'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2018-09-09.11:40:06.390>
closer = 'vstinner'
components = ['Extension Modules']
creation = <Date 2014-01-01.18:13:20.774>
creator = 'benjamin.peterson'
dependencies = ['33191', '33441']
files = []
hgrepos = []
issue_num = 20104
keywords = ['patch']
message_count = 70.0
messages = ['207137', '207139', '213344', '222570', '222571', '222572', '269684', '289758', '289769', '289775', '289791', '311040', '311041', '311051', '311081', '311086', '311087', '311089', '311094', '311100', '311102', '311144', '311164', '311165', '311166', '314732', '314770', '314774', '314775', '314777', '314779', '314781', '314783', '314784', '314826', '315795', '315995', '315997', '316014', '316046', '316063', '316068', '316088', '316150', '316235', '316238', '316246', '316247', '316268', '316282', '316489', '316495', '316503', '316522', '316525', '316551', '316588', '316592', '316595', '317071', '317076', '317084', '317510', '323639', '324765', '324832', '324833', '324881', '324883', '333129']
nosy_count = 13.0
nosy_names = ['gregory.p.smith', 'vstinner', 'benjamin.peterson', 'ned.deily', 'alex', 'dhduvall', 'neologix', 'gennad', 'martin.panter', 'serhiy.storchaka', 'John Jones', 'pablogsal', 'miss-islington']
pr_nums = ['5109', '5109', '5418', '6332', '6334', '6336', '6673', '6693', '6725', '6794']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20104'
versions = ['Python 3.8']
Activity
benjaminp commentedon Jan 1, 2014
posix_spawn is a nice, efficient replacement for fork()/exec(). We should expose it and possibly use it in subprocess where possible.
gpshead commentedon Jan 1, 2014
Unless it could replace the fork+exec code path in its entirety, which I do not believe is possible, I see posix_spawn() as a distraction and additional maintenance burden with no benefit.
http://pubs.opengroup.org/onlinepubs/7999959899/functions/posix_spawn.html
Read the RATIONALE section. The posix_spawn API was not created to make subprocess creation easier (i'd argue that it is the same burden to setup a proper call to posix_spawn as it is to do everything right for fork and exec).
One notable thing posix_spawn() does not support: setsid() (start_new_session=True) of the child process. Obviously it also couldn't handle the arbitrary preexec_fn but preexec_fn is in general considered harmful.
benjaminp commentedon Mar 13, 2014
Okay, this seems like a bad idea.
dhduvall commentedon Jul 8, 2014
Our project (the Solaris packaging system, IPS), relies on posix_spawn() primarily for the ability to fork without making a large memory reservation (and possibly failing) because the forking process was itself very large. That's the (a?) bug benefit of posix_spawn() -- it's not a benefit for the programmer using it (who might have to fall back to fork/exec), but for the end-user that benefits from its streamlined operation.
You're right that it doesn't handle everything that subprocess.Popen() does -- though at least on Solaris there's a way to change the cwd in the file actions, and I'm sure we'd consider adding a way to do the setsid() as well. The rest should be possible cross-platform.
alex commentedon Jul 8, 2014
Danek, you might find https://github.com/dreid/posix_spawn/ useful, it provides bindings and a public API over posix_spawn (it's not complete yet, but if there's stuff missing, feel free to file a ticket!)
dhduvall commentedon Jul 8, 2014
Cool. We implemented our own version as a straight-up native module (https://java.net/projects/ips/sources/pkg-gate/content/src/modules/pspawn.c), and our Popen replacement is not at present a complete replacement for the one in the stdlib, but it does what we need it to do. We'd absolutely switch if it came in to the stdlib, though, and I think there would be plenty of programs that would benefit from it (certainly a number of large Python programs in Solaris have run into exactly this problem).
dhduvall commentedon Jul 1, 2016
Oh, for what it's worth, Solaris added setsid support to posix_spawn a few years ago, as a result of this conversation. I still think it would be worthwhile supporting this in the stdlib, since we keep running into processes which have a lot of memory reserved and can't fork some small program because they're running in a memory-limited environment.
JohnJones commentedon Mar 17, 2017
To prevent subprocess/os.fork() doubling my memory after loading a large numpy array into memory, I now have to start my script with 650 calls to subprocess.Popen(), which just sit their waiting for some stdin to start doing something. This doesn't happen until after the numpy array is loaded, else I quickly run out of memory.
I think this sort of situation results in more unnecessary complexity compared to using posix_spawn, in spite of the concerns about using posix_spawn. Perhaps subprocess.Popen just needs a "posix_spawn=True" parameter?
gpshead commentedon Mar 17, 2017
I think someone wanting this will need to put forward a patch adding it to be reviewed and mulled over. As Alex mentioned in msg22571 - https://github.com/dreid/posix_spawn/ exists as does the code Danek pointed at in the next comment. try those.
I suggest someone who actively cares about a limited available process address space environment contribute this. (ie: not your typical 64-bit system)
fork()+exec() does not cause significant memory allocation, only a brief ~doubling of mapped address space, with backing pages being the originals just marked copy on write, but never written to by the child. The exec undoes that mapping.
It is technically possible to cause the copy on writes to happen if you immediately write to a large amount of memory in the parent process after the fork has happened before the exec has, but that seems like a rare timing problem that could even be worked around by monitoring the forked child to see that the exec has occurred before continuing.
JohnJones commentedon Mar 17, 2017
I agree with everything you're saying Gregory, however I don't think the significance of the memory doubling is as inconsequential as you might first think. For example, i have on my 64bit Linux system 128Gb of RAM, and a numpy table that's around 70Gb. Spawning a subprocess, even though memory is doubled for a very short period of time, is enough to raise a MemoryError, despite the subprocess i'm spawning using only 2 or 3Mb after the exec().
I do appreciate that for most Python users however, they will not see much benefit from what I imagine is quite a lot of development work.
FWIW, I did try the posix_spawn module, but i couldn't figure out how to write data to the stdin of a posix_spawn subprocess, and gave up in place of the commonly recommended solution to this problem (via StackExchange) of spawning lots of subprocesses before you put stuff in memory. Fortunately, for my problem, this was a possible solution. For others I think they're going to have to use posix_spawn, or an entirely different programming language if that doesn't work.
gpshead commentedon Mar 18, 2017
All I'm really saying is that someone who wants this should provide a
patch/PR with unittests. :) I can help review and go from there.
It does make sense to me for it to be available as part of the subprocess
API if it is available at all, likely an alternative implementation of or
behavior flag to _posixsubprocess.fork_exec() with appropriate autoconf and
conditional compilation based on availability ifdefs.
On Fri, Mar 17, 2017 at 11:49 AM John Jones <report@bugs.python.org> wrote:
66 remaining items
vstinner commentedon May 18, 2018
posix_spawn can still be found in Python 3.7:
configure:11243: posix_fallocate posix_fadvise posix_spawn pread preadv preadv2 \
configure.ac:3470: posix_fallocate posix_fadvise posix_spawn pread preadv preadv2 \
pyconfig.h.in:710:/* Define to 1 if you have the `posix_spawn' function. */
This check has been added by the commit 6c6ddf9 and should be removed from Python 3.7 as well, no? It's not a release blocker, but it may be good to remove posix_spawn from configure.ac (and related generated files).
pablogsal commentedon May 19, 2018
I originally removed it from the configure script in PR6794 but it was reintroduced in commit 5700952. Check the PR for Greg's rationale.
vstinner commentedon May 19, 2018
Oh ok.
vstinner commentedon May 24, 2018
Tests fail on PPC64 Fedora 3.x: bpo-33630.
serhiy-storchaka commentedon Aug 17, 2018
Benjamin, Gregory, could you please look at PR 6693? Is it what you want?
pablogsal commentedon Sep 7, 2018
New changeset 254a466 by Pablo Galindo in branch 'master':
bpo-20104: Add flag capabilities to posix_spawn (GH-6693)
254a466
serhiy-storchaka commentedon Sep 8, 2018
New changeset d700f97 by Serhiy Storchaka in branch 'master':
bpo-20104: Change the file_actions parameter of os.posix_spawn(). (GH-6725)
d700f97
serhiy-storchaka commentedon Sep 8, 2018
Thank you for your contribution Pablo. This issue have appeared much more complex and less obvious than it looked initially.
vstinner commentedon Sep 9, 2018
Yeah, thanks Pablo for your tenacy! This function is interesting in term of
performance and correctness ("atomic" function, signal safe?).
vstinner commentedon Sep 9, 2018
I close the issue, it's now done.
If someone wants to experiment in posix_spawn() in subprocess, please open a new issue.
vstinner commentedon Jan 7, 2019
Serhiy:
I created bpo-35674 to see how to expose this feature.