Repository navigation
asyncio.Runner allows async generators to skip gracefull finalization #95584
Description
reproducer:
import gc
import asyncio
import weakref
async def agenfn():
for i in range(100):
try:
yield i
finally:
await asyncio.sleep(0)
with asyncio.Runner() as runner:
agen = agenfn()
print(runner.run(anext(agen)))
print(runner.run(anext(agen)))
print(runner.run(anext(agen)))
agen_wr = weakref.ref(agen)
del agen
gc.collect()
assert agen_wr() is None
async def run_through():
agen = agenfn()
print(await anext(agen))
print(await anext(agen))
print(await anext(agen))
agen_wr = weakref.ref(agen)
del agen
gc.collect()
assert agen_wr() is None
asyncio.run(run_through())
gc.collect()output:
0
1
2
Exception ignored in: <async_generator object agenfn at 0x7f29a019d000>
Traceback (most recent call last):
File "/home/graingert/projects/gc_agen_runner.py", line 19, in <module>
del agen
^^^^
RuntimeError: async generator ignored GeneratorExit
0
1
2
Activity
graingert commentedon Aug 4, 2022
I think this is a fundamental problem with allowing an event loop to be paused and resumed.
currently pausing and resuming a loop can impact other threads relying on the main thread loop to handle processes
and as above async generator cleanup relies on an eventloop that remains in full control of the thread until it closes and shuts down, eg:
it makes sense for low-level hooks like loop.run_until_complete and loop.close to permit this, but high-level APIs shouldn't expose users to these sorts of edge cases
cc @njsmith
kumaraditya303 commentedon Nov 26, 2022
Is this an issue in practice though? I don't see any point in using a async generator across multiple runs. It should be iterated over once in the cycle.
gvanrossum commentedon Nov 29, 2022
@graingert Do you have a fix in mind?
kumaraditya303 commentedon Dec 2, 2022
FWIW I consider this to be a misuse of the API here not as a bug, you are not supposed to create async generator without a running event loop otherwise you end up not triggering async hooks. @graingert Do you have any use case for this? otherwise I'll close this a won't fix. See my comment #95584 (comment)
gvanrossum commentedon Dec 3, 2022
But that feels like an arbitrary restriction. I could imagine doing exactly that to control some test case, for example. (The advantage of doing it this way rather than the other way would be that the test itself can remain synchronous.)
kumaraditya303 commentedon Dec 3, 2022
I guess you are not aware of how async gen initialization and finalization works. See https://docs.python.org/3/library/sys.html#sys.set_asyncgen_hooks, using it without a running loop is inviting issues.
gvanrossum commentedon Dec 3, 2022
That sounds a little condescending. :-)
I sometimes try to see things from the POV of an unsophisticated user (which honestly I still am, most of the time :-).
The docs there don't actually explain a thing, the semantics are in PEP 525. I could look that up, but basically my point is that nobody who isn't an asyncio implementation expert understands how async generators really work, so the restrictions feel arbitrary (even though I remember that when I approved the PEP I understood why they existed and couldn't think of a better way either).
At the very least the docs should explain the restrictions of async generators. Not sure where that should go though -- the
sysfunction seems pretty obscure, but the language reference doesn't usually talk about specific async frameworks like asyncio, and the asyncio docs don't have a place to discuss this since when you use it as intended they "just work".kumaraditya303 commentedon Dec 3, 2022
I imagine that this isn't something an unsophisticated user could ever hit. You can only ever hit this if you are trying hard to do it. The general case is that an
agenis created inside a coroutine, it is exhausted and then it is closed. If you are creating it outside a running event loop, and using it across multiple event loops orasyncio.runcycles then you cannot expect it to work at all and in fact if it will work, it will promote wrong idiom of usingagen.+1 for better documentation.
kumaraditya303 commentedon Dec 8, 2022
I created #100108 to add better documentation for async generators.
kumaraditya303 commentedon Apr 19, 2023
Closing as #100108 covers this.