Skip to content

Commit e3687d4

Browse files
chrisguidryclaude
andauthored
Propagate external cancellation through worker waits on Python 3.10 and 3.11 (#456)
asyncio.wait_for on Python 3.10 and 3.11 swallows a cancellation that is delivered to the caller in the same event-loop tick that its inner future completes (python/cpython#86296; the 3.12 rewrite fixed it). The worker waits on events between polls with `_wait_for_event()`, so cancelling `run_forever()` at that moment lost the cancellation and the worker ran forever. #448 made the startup window easy to hit by deferring cancellation readiness until the pub/sub confirmation arrives, which is how FastMCP's worker shutdown started hanging deterministically on docket 0.23.1 (the Python 3.10 and lowest-direct jobs on PrefectHQ/fastmcp#4802). `cancel_task()` had the same class of defect: awaiting the task directly cannot tell a caller-delivered cancellation from the subtask's own when both land at once. Both helpers now live in `_cancellation.py` and use `asyncio.wait`, which never raises the awaited task's CancelledError and never absorbs the caller's. The new tests in `tests/test_worker_cancellation.py` cancel at a range of event-loop tick offsets; before the fix they fail on 3.10 and 3.11 at the offsets that land inside the swallow window, at the unit level and against a real worker. Not fixed here, same class of hazard on 3.10/3.11, each on a different code path: the `asyncio.wait_for` calls at `execution.py:1142`, `cli/_support.py:41`, and `dependencies/_timeout.py:89`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 84bbc3a commit e3687d4

3 files changed

Lines changed: 178 additions & 27 deletions

File tree

‎src/docket/_cancellation.py‎

Lines changed: 38 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,19 @@
1-
"""Cancellation utilities for cleaning up background tasks.
1+
"""Cancellation-correct waiting primitives.
22
33
When we cancel one of our own background tasks (a heartbeat, a monitor, a
44
renewal loop) and await it, the task's CancelledError surfaces in the awaiter
55
and must not propagate: we asked for it. A CancelledError should only
66
propagate when it was delivered to the *caller* -- an external cancellation
7-
that arrived while we were waiting.
8-
9-
Telling the two apart by the cancel message (task.cancel(msg=...)) is
10-
unreliable: CPython drops the message when the task's awaited future completes
11-
in the same event-loop tick as the cancellation (python/cpython#91048), and on
12-
Python 3.10 the message never propagates at all. The awaited task's final
13-
state is reliable on every supported version, so cancel_task() checks that
14-
instead.
7+
that arrived while we were waiting. The helpers here guarantee both halves
8+
of that contract, which plain awaits and asyncio.wait_for do not.
9+
10+
Awaiting the task directly cannot tell those two apart: the cancel message
11+
(task.cancel(msg=...)) is dropped when the awaited future completes in the
12+
same event-loop tick as the cancellation (python/cpython#91048), and the
13+
task's final state is ambiguous when both cancellations happen at once.
14+
asyncio.wait() never raises the awaited task's CancelledError at all, so the
15+
only CancelledError that can escape cancel_task() is one delivered to the
16+
caller, which must propagate.
1517
"""
1618

1719
import asyncio
@@ -21,21 +23,39 @@
2123
CANCEL_MSG_CLEANUP = "docket:cleanup"
2224

2325

26+
async def _wait_for_event(event: asyncio.Event, timeout: float) -> bool:
27+
"""Wait up to ``timeout`` seconds for ``event``; True if it was set.
28+
29+
asyncio.wait_for on Python 3.10 and 3.11 swallows a cancellation that is
30+
delivered to the caller in the same event-loop tick that the inner future
31+
completes (python/cpython#86296; the 3.12 rewrite fixed it). The worker
32+
waits on events between polls, so a worker cancelled at that moment would
33+
lose the cancellation and run forever. asyncio.wait never returns the
34+
inner task's result through the caller and never absorbs the caller's
35+
cancellation.
36+
"""
37+
if event.is_set():
38+
return True
39+
waiter = asyncio.ensure_future(event.wait())
40+
try:
41+
done, _ = await asyncio.wait([waiter], timeout=timeout)
42+
return bool(done)
43+
finally:
44+
waiter.cancel()
45+
46+
2447
async def cancel_task(task: "asyncio.Task[Any]", reason: str) -> None:
2548
"""Cancel a task and await its completion, suppressing its cancellation.
2649
27-
If the awaited task ended cancelled, the CancelledError came from it, and
28-
we swallow it -- we initiated that cancellation. If the task did not end
29-
cancelled, the CancelledError was delivered to the caller instead, so it
30-
propagates.
50+
A CancelledError raised here was delivered to the caller, not the awaited
51+
task, and propagates. If the task ended with a real exception instead of
52+
cancelling, that exception is re-raised.
3153
3254
Args:
3355
task: The task to cancel
3456
reason: A description of why we're cancelling (e.g., CANCEL_MSG_CLEANUP)
3557
"""
3658
task.cancel(reason)
37-
try:
38-
await task
39-
except asyncio.CancelledError:
40-
if not task.cancelled(): # pragma: no cover - caller cancelled mid-await
41-
raise
59+
await asyncio.wait([task])
60+
if not task.cancelled() and (error := task.exception()) is not None:
61+
raise error

‎src/docket/worker.py‎

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@
3434
from opentelemetry import trace
3535
from opentelemetry.trace import Status, StatusCode, Tracer
3636

37-
from ._cancellation import CANCEL_MSG_CLEANUP, cancel_task
37+
from ._cancellation import CANCEL_MSG_CLEANUP, _wait_for_event, cancel_task
3838
from ._lua import Arg, Key, redis_script
3939
from ._redis import RedisClient
4040
from ._telemetry import suppress_instrumentation
@@ -109,14 +109,6 @@
109109
TaskKey: TypeAlias = str
110110

111111

112-
async def _wait_for_event(event: asyncio.Event, timeout: float) -> bool:
113-
try:
114-
await asyncio.wait_for(event.wait(), timeout=timeout)
115-
except asyncio.TimeoutError:
116-
return False
117-
return True
118-
119-
120112
class PubSubMessage(TypedDict):
121113
"""Message received from Redis pub/sub pattern subscription."""
122114

‎tests/test_worker_cancellation.py‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
"""Regression tests: cancelling a worker externally must always shut it down.
2+
3+
asyncio.wait_for on Python 3.10 and 3.11 swallows a cancellation that is
4+
delivered to the caller in the same event-loop tick that the inner future
5+
completes (python/cpython#86296; the 3.12 rewrite fixed it). The worker
6+
waits on events between polls, so a worker cancelled at exactly that moment
7+
lost the cancellation and ran forever. These tests cancel at a range of
8+
event-loop tick offsets so at least one case lands inside that window.
9+
"""
10+
11+
import asyncio
12+
13+
import pytest
14+
15+
from docket import Docket, Worker
16+
from docket._cancellation import (
17+
_wait_for_event, # pyright: ignore[reportPrivateUsage]
18+
cancel_task,
19+
)
20+
21+
OFFSETS = [0, 1, 2, 3]
22+
23+
24+
async def test_wait_for_event_returns_true_for_a_set_event():
25+
"""An already-set event returns True without waiting"""
26+
event = asyncio.Event()
27+
event.set()
28+
29+
assert await _wait_for_event(event, 10.0) is True
30+
31+
32+
async def test_wait_for_event_returns_false_on_timeout():
33+
"""An event that never sets returns False after the timeout"""
34+
event = asyncio.Event()
35+
36+
assert await _wait_for_event(event, 0.01) is False
37+
38+
39+
async def test_cancel_task_reraises_a_real_exception():
40+
"""A subtask that fails instead of cancelling re-raises its exception"""
41+
42+
async def defiant() -> None:
43+
try:
44+
await asyncio.Event().wait()
45+
except asyncio.CancelledError:
46+
raise RuntimeError("cleanup went sideways") from None
47+
48+
task = asyncio.create_task(defiant())
49+
await asyncio.sleep(0)
50+
51+
with pytest.raises(RuntimeError, match="cleanup went sideways"):
52+
await cancel_task(task, "test cleanup")
53+
54+
55+
async def test_cancel_task_tolerates_a_task_that_finishes_anyway():
56+
"""A subtask that absorbs the cancellation and returns is left alone"""
57+
58+
async def stubborn() -> None:
59+
try:
60+
await asyncio.Event().wait()
61+
except asyncio.CancelledError:
62+
pass
63+
64+
task = asyncio.create_task(stubborn())
65+
await asyncio.sleep(0)
66+
67+
await cancel_task(task, "test cleanup")
68+
69+
assert task.done()
70+
assert not task.cancelled()
71+
72+
73+
async def _wait_for_worker_readiness(worker: Worker) -> None:
74+
"""Yield until the worker's cancellation listener reports readiness."""
75+
deadline = asyncio.get_running_loop().time() + 5
76+
while asyncio.get_running_loop().time() < deadline:
77+
session = worker._processing_session # pyright: ignore[reportPrivateUsage]
78+
if session is not None and session.cancellation_ready.is_set():
79+
return
80+
await asyncio.sleep(0)
81+
raise TimeoutError("worker never became ready")
82+
83+
84+
@pytest.mark.parametrize("offset", OFFSETS)
85+
async def test_wait_for_event_honors_caller_cancellation(offset: int):
86+
"""A cancellation that races the event being set still cancels the caller"""
87+
event = asyncio.Event()
88+
task = asyncio.create_task(_wait_for_event(event, 10.0))
89+
await asyncio.sleep(0)
90+
91+
event.set()
92+
for _ in range(offset):
93+
await asyncio.sleep(0)
94+
cancel_requested = task.cancel()
95+
96+
await asyncio.wait([task], timeout=5)
97+
98+
assert task.done(), "the wait outlived its cancellation"
99+
assert not cancel_requested or task.cancelled(), (
100+
"the caller's cancellation was swallowed"
101+
)
102+
103+
104+
@pytest.mark.parametrize("offset", OFFSETS)
105+
async def test_cancel_task_propagates_caller_cancellation(offset: int):
106+
"""A cancellation that races the subtask's own cancellation still propagates"""
107+
forever = asyncio.Event()
108+
inner = asyncio.create_task(forever.wait())
109+
await asyncio.sleep(0)
110+
111+
outer = asyncio.create_task(cancel_task(inner, "test cleanup"))
112+
for _ in range(offset):
113+
await asyncio.sleep(0)
114+
cancel_requested = outer.cancel()
115+
116+
await asyncio.wait([outer], timeout=5)
117+
118+
assert outer.done(), "cancel_task outlived its cancellation"
119+
assert not cancel_requested or outer.cancelled(), (
120+
"the caller's cancellation was swallowed"
121+
)
122+
123+
124+
@pytest.mark.parametrize("offset", OFFSETS)
125+
async def test_cancelling_run_forever_at_readiness_shuts_down(
126+
docket: Docket, offset: int
127+
):
128+
"""run_forever cancelled the moment the worker becomes ready still exits"""
129+
async with Worker(docket, name=f"cancel-me-{offset}") as worker:
130+
run_task = asyncio.create_task(worker.run_forever())
131+
await _wait_for_worker_readiness(worker)
132+
133+
for _ in range(offset):
134+
await asyncio.sleep(0)
135+
run_task.cancel()
136+
137+
_, pending = await asyncio.wait([run_task], timeout=5)
138+
139+
assert not pending, "the worker did not shut down after cancellation"

0 commit comments

Comments
 (0)