Repository navigation
asyncio: support fork #66197
Description
| BPO | 21998 |
|---|---|
| Nosy | @pitrou, @vstinner, @tiran, @1st1, @Martiusweb |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/1st1'
closed_at = <Date 2020-02-10.23:51:31.002>
created_at = <Date 2014-07-17.16:10:45.820>
labels = ['expert-asyncio']
title = 'asyncio: support fork'
updated_at = <Date 2020-02-10.23:51:31.001>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2020-02-10.23:51:31.001>
actor = 'vstinner'
assignee = 'yselivanov'
closed = True
closed_date = <Date 2020-02-10.23:51:31.002>
closer = 'vstinner'
components = ['asyncio']
creation = <Date 2014-07-17.16:10:45.820>
creator = 'vstinner'
dependencies = []
files = ['37306', '37385', '38016', '38019', '38164']
hgrepos = []
issue_num = 21998
keywords = ['patch']
message_count = 56.0
messages = ['223344', '223431', '225973', '226861', '226889', '226892', '226899', '226932', '231819', '231845', '231919', '231921', '231941', '232301', '232308', '235409', '235410', '235429', '236065', '236131', '236134', '236136', '236145', '236149', '236150', '236151', '244111', '244112', '244113', '244114', '244116', '244117', '244118', '244119', '244120', '244121', '244123', '244130', '244135', '249722', '249724', '249732', '256889', '297117', '297207', '297218', '297219', '297228', '297230', '297232', '297233', '297235', '313196', '325820', '325824', '361755']
nosy_count = 9.0
nosy_names = ['pitrou', 'vstinner', 'christian.heimes', 'zmedico', 'neologix', 'yselivanov', 'Adam.Bishop', 'martius', 'Christian H']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue21998'
versions = ['Python 3.6']
Activity
vstinner commentedon Jul 17, 2014
It looks like asyncio does not suppport fork() currently: on fork(), the event loop of the parent and the child process share the same "self pipe".
Example:
---
Output:
---
parent 6 5
child 6 5
---
I'm not sure that it's revelant to use asyncio with fork, but I wanted to open an issue at least as a reminder.
I found the "self pipe" + fork issue when reading the issue bpo-12060, while working on a fix for the signal handling race condition on FreeBSD: issue bpo-21645.
vstinner commentedon Jul 18, 2014
Oh, I wanted to use the atfork module for that, but then I saw that it does not exist :-(
vstinner commentedon Aug 27, 2014
A first step would be to document the issue in the "developer" section of asyncio documentation. Mention that the event loop should be closed and a new event loop should be created.
1st1 commentedon Sep 14, 2014
It's not that it doesn't work after fork, right? Should we add a recipe with pid monitoring a self-pipe re-initialization?
gvanrossum commentedon Sep 14, 2014
Actually I expect that if you share an event loop across different processes via form, everything's a mess -- whenever a FD becomes ready, won't both the parent and the child be woken up? Then both would attempt to read from it. One would probably get EWOULDBLOCK (assuming all FDs are actually in non-blocking mode) but it would still be a mess. The specific mess for the self-pipe would be that the race condition it's intended to solve might come back.
It's possible that some polling syscall might have some kind of protection against forking, but the Python data structures that map FDs to handlers don't know that, so it would still be a mess.
Pretty much the only thing you should expect to be able to do safely after forking is closing the event loop -- and I'm not even 100% sure that that's safe (I don't know what happens to a forked executor).
Is there a use case for sharing an event loop across forking?
1st1 commentedon Sep 15, 2014
I don't think so. I use forking mainly for the following two use-cases:
Socket sharing for web servers. Solution: if you want to have a shared sockets between multiple child processes, just open them in master process, fork as many times as you want, and start event
loops in child processes only.
If sometimes you want to spawn some child process "on demand". Solution: fork before having a loop running, and use the child process as a "template", i.e. when you need a new child process - just ask the first child to fork it.
It would certainly be handy to have an ability to fork while the loop is running and safely close (and reopen) it in the forked child process. But now I can see that it's a non-trivial thing to do properly. Probably it's ~somewhat~ safe to re-initialize epoll (or whatever selector we use), re-open self-pipe etc, drop all queued callbacks and clear Task.all_tasks collection, but even then it's easy to miss something.
I think we just need to make sure that we have documented that asyncio loops are not fork safe, and forks in running loops should be avoided by all means.
vstinner commentedon Sep 15, 2014
I don't know if asyncio must have a builtin support for this use case. The minimum is to document the behaviour, or maybe even suggest a recipe to support it.
For example, an event loop of asyncio is not thread-safe and we don't want to support this use case. But I wrote a short documentation with code snippets to show how to workaround this issue:
https://docs.python.org/dev/library/asyncio-dev.html#concurrency-and-multithreading
We need a similar section to explain how to use asyncio with the os.fork() function and the multiprocesing module.
gvanrossum commentedon Sep 15, 2014
That sounds about right -- it's a doc issue. Let me propose a paragraph:
"""
NOTE: It is not safe to share an asyncio event loop between processes that are related by os.fork(). If an event loop exists in a process, and that process is forked, the only safe operation on the loop in the child process is to call its close() method.
"""
(I don't want to have to research what the various polling primitives do on fork(), so I don't want to suggest that it's okay to close the loop in the parent and use it in the child.)
A similar note should probably be added to the docs for the selectors module.
Martiusweb commentedon Nov 28, 2014
Hi,
Actually, closing and creating a new loop in the child doesn't work either, at least on Linux.
When, in the child, we call loop.close(), it performs:
self.remove_reader(self._ssock)
(in selector_events.py, _close_self_pipe() around line 85)
Both the parent and the child still refer to the same underlying epoll structure, at that moment, and calling remove_reader() has an effect on the parent process too (which will never watch the self-pipe again).
I attached a test case that demonstrates the issue (and the workaround, commented).
gvanrossum commentedon Nov 29, 2014
Martin, what is the magic call to make in the child (or in the parent pre-fork???) to disable the epoll object in the child without disabling it in the parent?
(Perhaps just closing the selector and letting the unregister calls fail would work?)
Martiusweb commentedon Dec 1, 2014
Guido,
Currently in my program, I manually remove and then re-adds the reader to the loop in the parent process right after the fork(). I also considered a dirty monkey-patching of remove_reader() and remove_writer() which would act as the original versions but without removing the fds from the epoll object (ensuring I don't get bitten by the same behavior for an other fd).
The easiest fix, I think, is indeed to close the selector without unregistering the fds, but I don't know if doing so would have undesired side effects on an other platform than Linux (resources leak, or the close call failing maybe).
Martiusweb commentedon Dec 1, 2014
I said something wrong in my previous comment: removing and re-adding the reader callback right after the fork() is obviously subject to a race condition.
I'll go for the monkey patching.
gvanrossum commentedon Dec 1, 2014
How about not doing anything in the parent but in the child, closing the
selector and then the event loop?
On Mon, Dec 1, 2014 at 1:29 AM, Martin Richard <report@bugs.python.org>
wrote:
Martiusweb commentedon Dec 8, 2014
Currently, this is what I do in the child after the fork:
It replaces unregister() by _BaseSelectorImpl.unregister(), so "our" data structures are still cleaned (the dict _fd_to_key, for instance).
If a fix for this issue is desired in tulip, the first solution proposed by Guido (closing the selector and let the unregister call fail, see the -trivial- patch attached) is probably good enough.
37 remaining items
1st1 commentedon Jun 28, 2017
We can do this but only in debug mode.
vstinner commentedon Jun 28, 2017
A compromise for the short term would be to detect fork in debug mode
and raise an exception, and explain how to fix such bug. What do you
think?
1st1 commentedon Jun 28, 2017
I'd prefer to fix it properly in 3.7.
pitrou commentedon Jun 28, 2017
I'm not sure why it would be debug-only. You usually don't fork() often, and you don't have many event loops around, so the feature sounds cheap.
In any case, I'm not directly affected by this issue, I'm merely suggesting options.
1st1 commentedon Jun 28, 2017
I think you're right. If it's low or zero overhead we can have the check always enabled.
zmedico commentedon Mar 4, 2018
I'm not sure about possible use cases that might conflict with this approach, but using a separate event loop for each pid seems very reasonable to me, as follows:
vstinner commentedon Sep 19, 2018
Give up, document that fork() is not supported and close the issue :-) IMHO it's not worth it.
1st1 commentedon Sep 19, 2018
I'll revisit this later.
vstinner commentedon Feb 10, 2020
There is no activity for 2 years. Asyncio is mature now. It seems like users learnt how to work around this issue, or don't use fork() with asyncio.
I close the issue.
kristjanvalur commentedon Aug 24, 2023
As far as I can tell, the problems of fork() and asyncio are not documented. It would be useful at least to have it mentioned in the docs that forking a running event loop is not supported.
pitrou commentedon Aug 24, 2023
Well, fork() is more generally discouraged especially in networking applications. If you're using multiprocessing, the "forkserver" method is a reliable and safe alternative.
That said, I agree it would be useful to document. Perhaps you would like to open a separate issue for that?
kristjanvalur commentedon Aug 24, 2023
sounds fine
fix: handle some webtiles race conditions