iscoroutine returns False for native coroutines in python 3.5+ #2092

Activity

1. scoder commented on Feb 10, 2018

   

   scoder

   on Feb 10, 2018

   Contributor

   More actions

   That's correct, yes. Cython compiled coroutines do not have byte code, and they are not Python generators. They implement the same protocol, but they are technically not the same thing (in the same way that a list iterator is not a dict iterator, even though both are iterators).

   The inspect module should definitely return False. Whether the asyncio module should also return False is debatable. It could be argued that that's a bug in asyncio, but deciding that would require a deep look into the internals of asyncio to figure out the exact semantics that is intended there. Please reopen if you have reason to believe that Cython should do something about this, rather than fixing it in asyncio.
2. scoder

   closed this as completedon Feb 10, 2018
3. danigosa commented on May 5, 2018

   

   danigosa

   on May 5, 2018

   More actions

   @scoder the implementation in asyncio is ridiculous

   def iscoroutinefunction(func):
       """Return True if func is a decorated coroutine function."""
       return (inspect.iscoroutinefunction(func) or
               getattr(func, '_is_coroutine', None) is _is_coroutine)
   

   So once we agreed inspect returns False, it happens to be that CPython just add a simple attribute to the function _is_coroutine, I think if Cython does this adding up this attribute it will make everything work

   Here: https://github.com/python/cpython/blob/789e359f51d2b27bea01b8c6c3bf090aaedf8839/Lib/asyncio/coroutines.py#L152

   And here:
   https://github.com/python/cpython/blob/789e359f51d2b27bea01b8c6c3bf090aaedf8839/Lib/asyncio/coroutines.py#L160

   So the solution as it can't be another it results in this, from my test in iPython:

   In [46]: fut = init_connection()
   In [47]: asyncio.iscoroutine(fut)
   Out[47]: True
   
   In [49]: init_connection.__class__
   Out[49]: cython_function_or_method
   
   In [50]: asyncio.iscoroutinefunction(init_connection)
   Out[50]: False
   
   In [56]: from asyncio.coroutines import _is_coroutine
   
   In [57]: init_connection._is_coroutine = _is_coroutine
   
   In [58]: asyncio.iscoroutinefunction(init_connection)
   Out[58]: True
   

   Dirty but easy. Don't know if this is a 5 seconds issue or a hara-kiri, theoretically it should be easy to plug the attribute runtime from from asyncio.coroutines import _is_coroutine, or not.

   This is a super annoying problem (I don't blame anyone with the word bug) that make cythonized code to be very dramatic in several asyncio frameworks. asyncio.iscoroutine work well with cythonized funcs for the silliest reason I think.
4. danigosa commented on May 5, 2018

   

   danigosa

   on May 5, 2018

   Last edited by danigosa

   More actions

   I found a filthy workaround just in case anyone falls into desperation:

   Following the cyfunctions above, create_token is NOT a coroutine, while init_connection is an async function, this workaround works in both so it's ok if you can schedule the functions whether are coros or not, no way to easily fix this with asyncio.iscoroutinefunction or at least easier than that:

   In [59]: async def test():
       ...:     await create_token({})
       ...:
   In [61]: asyncio.get_event_loop().run_until_complete(test())
   TypeError: object str can't be used in 'await' expression
   
   In [73]: async def test():
       ...:     return await init_connection()
   Out[74]: <asyncpg.pool.Pool at 0x7f78e04752c8>
   

   Here comes the magic:

   In [69]: async def test():
       ...:     return await asyncio.coroutine(create_token)({})
   In [70]: asyncio.get_event_loop().run_until_complete(test())
   Out[70]: 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzUxMiJ9.eyJleHAiOjE1MjU1NTk0NTB9.v8wRZGIke6RYizIpJhl4oaypKyvuVARKaiq0KdpJg6XJ0qTB0o76BuTLera6kSQ_5qXnAb7_9DQSadwdRqgPmw'
   
   In [71]: async def test():
       ...:     return await asyncio.coroutine(init_connection)()
   In [72]: asyncio.get_event_loop().run_until_complete(test())
   Out[72]: <asyncpg.pool.Pool at 0x7f78df48af48>
   

   With that the workaround:

   def nasty_iscoroutinefunction(func):
         is_coro = asyncio. iscoroutinefunction(func)
         if func.__class__.__name__ == 'cython_function_or_method':
               # It's cythonized, danger!
               is_coro = True  # We'll make it possible
               func = asyncio.coroutine(func)
         return is_coro

   This works!
5. graingert commented on Jul 18, 2022

   

   graingert

   on Jul 18, 2022

   More actions

   inspect.iscoroutinefunction() uses inspect._signature_is_functionlike so that duck-type functions like Cython functions and AsyncMock can support it

   this was backported to 3.10 and is released in 3.11b4
6. graingert commented on Jul 18, 2022

   

   graingert

   on Jul 18, 2022

   More actions

   @graingert looking at the implementation, it still requires us to set f.__code__.co_flags to have a certain value.

   We've apparently been reluctant to do this because of the possibility that CPython might try to use the flags to process the code object #2273 (comment)

   Originally posted by @da-woods in #3849 (comment)
7. scoder commented on Jul 19, 2022

   

   scoder

   on Jul 19, 2022

   Contributor

   More actions

   Thanks @graingert.

   Note that the fact that unittest.mock.AyncMock uses a fake code object and sets a flag on it does not mean that the same thing will work for us at the C level with a real code object using the CO_COROUTINE flag. However, this should at least work safely in CPython <= 3.10, probably up to 3.11, and from those versions on, the inspect change in python/cpython#94050 would take over (from the PoV of a user) if we ever need to remove the flag again. So I'd deem it ok to generally enable the flag for coroutines in Cython 3.0. Until further notice. :)
8. scoder

   reopened this on Jul 19, 2022
9. scoder

   added

   defect

   Python SemanticsGeneral Python (3) language semanticsGeneral Python (3) language semantics

   on Jul 19, 2022
10. scoder

    added this to the 3.0 milestone on Jul 19, 2022
11. scoder commented on Jul 19, 2022

    

    scoder

    on Jul 19, 2022

    Contributor

    More actions

    Given that there was a lot more activity in ticket #2273, I'll reopen that one instead.
12. scoder

    closed this as completedon Jul 19, 2022