asyncio.new_event_loop() hangs #70707

Activity

1. ErwinMayer commented on Mar 9, 2016

   

   ErwinMayermannequin

   on Mar 9, 2016

   MannequinAuthor

   More actions

   I am using the following function to force a coroutine to run synchronously:

   def await_sync(coro: types.CoroutineType, timeout_s: int=None):
       """

   :param coro: a coroutine or lambda loop: coroutine(loop)
   :param timeout_s:
   :return:
   """
   loop = asyncio.new_event_loop()  # type: BaseEventLoop
   if not is_awaitable(coro):
       coro = coro(loop)
   if timeout_s is None:
       fut = asyncio.ensure_future(coro, loop=loop)
   else:
       fut = asyncio.ensure_future(asyncio.wait_for(coro, timeout=timeout_s, loop=loop), loop=loop)
   loop.run_until_complete(fut)
   return fut.result()
   

   def is_awaitable(coro_or_future):
       if isinstance(coro_or_future, futures.Future):
           return coro_or_future
       elif asyncio.coroutines.iscoroutine(coro_or_future):
           return True
       elif asyncio.compat.PY35 and inspect.isawaitable(coro_or_future):
           return True
       else:
           return False

   However, intermittently, it will freeze upon simply trying to create a new loop: loop = asyncio.new_event_loop(). Inspecting the stack traces shows me the exact location where it hangs:

   File: "/src\system\utils.py", line 34, in await_sync
   loop = asyncio.new_event_loop() # type: BaseEventLoop
   File: "\lib\asyncio\events.py", line 636, in new_event_loop
   return get_event_loop_policy().new_event_loop()
   File: "\lib\asyncio\events.py", line 587, in new_event_loop
   return self._loop_factory()
   File: "\lib\asyncio\selector_events.py", line 55, in __init__
   self._make_self_pipe()
   File: "\lib\asyncio\selector_events.py", line 116, in _make_self_pipe
   self._ssock, self._csock = self._socketpair()
   File: "\lib\asyncio\windows_events.py", line 295, in _socketpair
   return windows_utils.socketpair()
   File: "\lib\socket.py", line 515, in socketpair
   ssock, _ = lsock.accept()
   File: "\lib\socket.py", line 195, in accept
   fd, addr = self._accept()

   I am using Python 3.5.1.

   StackOverflow question: http://stackoverflow.com/questions/35861175/what-can-cause-asyncio-to-hang-upon-simply-creating-a-new-loop
2. ErwinMayer

   added

   topic-asyncio

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   on Mar 9, 2016
3. vstinner commented on Mar 9, 2016

   

   vstinner

   on Mar 9, 2016

   Member

   More actions

   Do you have a firewall? Maybe the firewall blocks programs listening on sockets? It would be silly, the socket is only listening on 127.0.0.1 (local link) (or ::1 in IPv6, also local link).

   Extract of the socketpair() function:

               try:
                   csock.connect((addr, port))
               except (BlockingIOError, InterruptedError):
                   pass
   

   Can you please try to replace this code with:

               try:
                   csock.connect((addr, port))
               except (BlockingIOError, InterruptedError) as exc:
                   print("CONNECT ERROR! %r" % (exc,))
   

   You can copy socket.py from the Python standard library to the current directory and modify the local copy.
4. ErwinMayer commented on Mar 9, 2016

   

   ErwinMayermannequin

   on Mar 9, 2016

   MannequinAuthor

   More actions

   This cannot be a firewall issue as a loop can be successfully created thousands of time, for hours, before a creation attempt will hang (with no change from the firewall). The rest of the program (other threads) continues to run and there is memory issue.

   If I understand correctly, the sockets used by asyncio are internal only and would not be exposed to the world through any local port, so I don't see how a firewall could intercept them (that would be worrisome for all users of asyncio).
5. gvanrossum commented on Mar 9, 2016

   

   gvanrossum

   on Mar 9, 2016

   Member

   More actions

   Why are you creating a new event loop for this purpose? There are all sorts of things that can go wrong with that.

   This is not appropriate for a bug report, you should continue to seek help on StackOverflow.
6. gvanrossum

   closed this as completedon Mar 9, 2016
7. ErwinMayer commented on Mar 9, 2016

   

   ErwinMayermannequin

   on Mar 9, 2016

   MannequinAuthor

   More actions

   I simply need to run a coroutine synchronously (which may or may not have a loop parameter), so I tried to do it by the book. With .NET it is easy to call .Wait() on a Task (but .NET by default uses a threadpool, not a single-threaded event loop).

   I am intrigued: how is it possible to achieve this at all without creating an event loop, and what kind of things are you referring to that could go wrong with my implementation?

   I am a bit scared as well as if this canonical way to use loops fails for no good reason, it may imply other things in asyncio could go wrong. Unless proven otherwise (I do not rule out an implementation mistake), there could be a bug here, hence this post (StackOverflow is generally of little help when it comes to bugs).
8. gvanrossum commented on Mar 9, 2016

   

   gvanrossum

   on Mar 9, 2016

   Member

   More actions

   Asyncio is not .NET. Don't assume that what's "by the book" according
   to .NET applies to asyncio.
9. ErwinMayer commented on Mar 9, 2016

   

   ErwinMayermannequin

   on Mar 9, 2016

   MannequinAuthor

   More actions

   I understand that asyncio is not .NET and did my best to implement "by the book" according to the asyncio documentation I read extensively.
   My await_async function has in fact nothing in common with what I would do in .NET but is hopefully as close as possible to the canonical way to create and consume a loop.
   The only maybe non standard thing I do is create a lot of short-lived loops, but nothing in the documentation indicates this heavy usage can cause instability/freezing.

   The case I am reporting can therefore be seen as an asyncio stress test and is worrisome as if there is indeed a bug, it could occur randomly when creating a single loop as well.

   So far, I cannot see anything in my implementation that would be fundamentally wrong (for example: do we need to always call close() on a loop, or is it taken care of by the GC). I can provide remote access to my screen when the issue occurs if it helps, and will post a bounty on my StackOverflow question tomorrow.
10. gvanrossum commented on Mar 9, 2016

    

    gvanrossum

    on Mar 9, 2016

    Member

    More actions

    Closing the loop is not taken care of by GC, so I would start there.
    AFAIK that's covered by the docs, but if you think the docs could be
    clearer please do submit a separate issue with a documentation update
    (you can easily submit a patch for that!).

    It would be more useful as a bug report if you boiled it down to a
    small stand-alone program that exhibits the bug, rather than saying
    "in production, after many hours, I get a traceback" (the code you
    posted by itself is not enough to repro the issue, since you never
    show what the coroutine does).
11. ErwinMayer commented on Mar 10, 2016

    

    ErwinMayermannequin

    on Mar 10, 2016

    MannequinAuthor

    More actions

    Thank you Guido; I am now trying to always close a loop explicitly and see if it fixes the issue (so far so good in the last 24h).

    I could not locate this requirement in the EventLoop documentation page (https://docs.python.org/3/library/asyncio-eventloop.html), however it was indeed mentioned in the "Develop with asyncio" documentation page (https://docs.python.org/3/library/asyncio-dev.html#asyncio-close-transports), saying that otherwise a Resource warning would be emitted (if debug mode was turned on).
    It could be beneficial to remind the need to close the loop explicitly in the EventLoop page as well. I saw a lot of people posting code snippets not doing it. I'll check how I can submit a patch for that.

    I agree with your comments on the reproductibility of my issue, I was hoping the stack trace could be enough to pinpoint reasons of why a socket creation could be hanging (and maybe diagnose a bug).
12. ezio-melotti

    transferred this issue fromon Apr 10, 2022