Repository navigation
Fix asyncio lock waiter deadlocks after cancellation - #1145
Conversation
2ccf10e to
cde0728
Compare
|
Are these super complicated tests really required to reproduce the issue? |
|
Hm, I'm starting to see the logic behind the tests. Could they not use AnyIO APIs though, and also run on Trio? |
cde0728 to
dbb003e
Compare
|
Please don't force-push changes. That way I can't see what you've changed. |
|
Thanks, I'll review when I get the time. |
It is the minimal tests I could come up with that reproduces the race: a cancelled head waiter must still be queued when release runs, and a later waiter must queue before the cancelled task's cleanup removes itself. We could drop |
Ah, sorry. I will stop doing that. |
…ncelled-waiters # Conflicts: # docs/versionhistory.rst
|
I'm currently spending some time trying to understand the conditions under which this problem can occur. My own attempts to reproduce the problem have all failed. I can move forward with this PR once I fully understand what's going on. |
I agree; the tests do seem overly complicated. In fact, the issue can be reproduced with just two tasks: #!/usr/bin/env python3
import anyio
async def main():
lock = anyio.Lock()
lock_acquire = lock.acquire
lock_release = lock.release
await lock_acquire()
async with anyio.create_task_group() as tg:
task1 = tg.create_task(lock_acquire())
await anyio.wait_all_tasks_blocked()
task2 = tg.create_task(lock_acquire())
task1.cancel()
lock_release()
with anyio.fail_after(1):
await task2
if __name__ == "__main__":
anyio.run(main)I tested #!/usr/bin/env python3
import asyncio
async def main():
future = asyncio.get_running_loop().create_future()
async def work():
try:
await future
except asyncio.CancelledError:
print("task: cancelled")
task = asyncio.create_task(work())
await asyncio.sleep(0) # switch to the task
task.cancel()
print(f"future: {'cancelled' if future.cancelled() else 'pending'}")
await task
if __name__ == "__main__":
asyncio.run(main())As a result:
The solution in this PR is similar to how my primitives work, except that my Note that asyncio takes a different approach and modifies the fast-path condition. This is also a possible solution, but it seems far less optimal to me (it is effectively O(n) just to check for the possibility of acquiring without waiting!). However, the real solution lies not in that, but in the fact that a cancelled task always wakes up the first one in the queue if the lock has not been acquired. In theory, we could repeat that behavior by moving the main part of the ...the same example, but with just one task: #!/usr/bin/env python3
import anyio
async def main():
lock = anyio.Lock()
lock_acquire = lock.acquire
lock_release = lock.release
async with anyio.create_task_group() as tg:
await lock_acquire()
task = tg.create_task(lock_acquire())
await anyio.wait_all_tasks_blocked()
task.cancel()
lock_release()
with anyio.fail_after(1):
await lock_acquire()
if __name__ == "__main__":
anyio.run(main)The explanation is the same as well. |
| try: | ||
| self._waiters.remove(item) | ||
| except ValueError: | ||
| pass |
There was a problem hiding this comment.
This would also result in O(n²) iterations, since self._waiters.remove(item) takes O(n) time for a missing item. Hypothetically, we could mark visited items in the release() method and only perform the removal when there is no such marker, but I am not sure how to do this efficiently in terms of memory for asyncio futures.
It would likely be worth changing the items (queue tokens) slightly so that they include one mutable value (for example, replacing a two-element tuple with a three-element list).
>>> from sys import getsizeof
>>> getsizeof((1, 2)) # immutable
64
>>> getsizeof([1, 2, 3]) # mutable
88
>>> class LockWaiter:
... __slots__ = ("task", "future", "removed")
>>> getsizeof(LockWaiter()) # also mutable
56 # without slots: 328 == getsizeof(LockWaiter()) + getsizeof(LockWaiter().__dict__)|
Perhaps understanding the conditions under which the problem occurs will help in understanding it. For the issue to be reproduced, four conditions must be met:
|
|
It took me a while, but I now understand that it's here where things go wrong: if self._owner_task is None and not self._waiters:What is there is no current owner but there are waiters, all of which have been cancelled? The latest task would be put in a waiting state, waiting for a release that never comes, as the cancelled waiters only do that if they've received the lock. |
|
@x42005e1f I pushed some simplifications to this branch, including the unconditional |
|
I removed the other semaphore test which was already passing without these changes. I hope that was okay. In any case, I'm ready to merge unless anyone watching this has objections. |
|
I also took another look at this block in the lock and semaphore: self._waiters.append(item)
try:
await fut
except CancelledError:
...It looks inconsistent, and in the case of the semaphore, it could be optimized. I suggest using the following implementation for both: self._waiters.append(item)
try:
await fut
except CancelledError:
if fut.cancelled():
try:
self._waiters.remove(fut)
except ValueError: # already removed (in release())
pass
else: # notified
self.release()
raiseIt can be further improved by adding markers, but this way it will no longer attempt to remove a non-existent item when the notification has already happened (and thus perform the expensive O(n) operation on every semaphore release!). After all, if the notification has happened, then due to the unconditional |
Under what circumstances would |
|
Does the concern about correctness have to do only with the fact that we only handle |
|
Yes, of course. The code seems to handle direct task cancellations, and as a result, it contains redundant logic. At the same time, it does not handle other types of exceptions. And my concerns about performance remain. |
|
A side effect of this PR: |
|
Alright, I've pushed your optimizations. All good now? |
|
Hmm, wait a minute. If |
|
I think you'll have to elaborate. My brain is quite tired at this time of night. |
|
The In the case of locks, the release finds the first uncancelled task (if there is one) and transfers ownership to it. If there is no such task, the queue will be empty, and there will be no owner. A situation where there is no owner but the queue is not empty cannot occur. In the case of semaphores, a similar thing happens, but when there is no uncancelled task, the value is incremented. It is impossible to imagine a situation where the value is non-zero but there is still someone in the queue. In both cases, checking whether the queue is non-empty ( |
|
I'm not 100% whether this is true, but the performance impact of said check is negligible so I don't think I'm going to touch that check at this point. This is the only PR left blocking the v4.14 release so we can revisit this later if there are no more impactful changes or concerns. |
|
I agree. |
|
Thanks @rasmusfaber for the PR and @x42005e1f for the in-depth review! |
|
Interestingly, my comment about redundant checks still holds true even before the changes in this PR, which hints at one possibility: in fact, removing It likely came to us from asyncio, where |
PR #403 pinned a temporary fork of AnyIO 4.13.0 to fix runner deadlocks on cancelled Lock/Semaphore waiters, because the upstream fix wasn't released yet. AnyIO 4.14.0 now ships that fix (agronholm/anyio#1145), so drop the fork override and pin the official release across all lockfiles. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
anyio before 4.14 can leave an asyncio Lock or Semaphore stuck when a waiter is cancelled mid-acquire: release() skips the cancelled waiter but leaves it queued, so later waiters block forever on an unlocked lock (agronholm/anyio#1145, fixed in 4.14.0). The stale-flush logging in this PR acquires _flush_pending_lock in cancellation-heavy paths (idle-timer teardown, sample retries), so on the asyncio backend an eval_set retry run can corrupt the lock and hang. That is the Python 3.10 CI "worker crashed" in test_eval_set_identifiers (the pytest --timeout added here turns the hang into a killed xdist worker). Bump the floor to >=4.14.0 and relock (4.13.0 -> 4.14.0). The branch already adopted anyio 4.14's stricter start_soon typing; this completes that upgrade. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
anyio before 4.14 can leave an asyncio Lock or Semaphore stuck when a waiter is cancelled mid-acquire: release() skips the cancelled waiter but leaves it queued, so later waiters block forever on an unlocked lock (agronholm/anyio#1145, fixed in 4.14.0). The stale-flush logging in this PR acquires _flush_pending_lock in cancellation-heavy paths (idle-timer teardown, sample retries), so on the asyncio backend an eval_set retry run can corrupt the lock and hang. That is the Python 3.10 CI "worker crashed" in test_eval_set_identifiers (the pytest --timeout added here turns the hang into a killed xdist worker). Bump the floor to >=4.14.0 and relock (4.13.0 -> 4.14.0). The branch already adopted anyio 4.14's stricter start_soon typing; this completes that upgrade. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Flush pending eval samples while running * fix(log): drop unused stale-flush event, shield logger cleanup on cancel Address review feedback on #4204: - Remove the write-only `_stale_flush_stopped` field; stale-flush synchronization runs entirely through the `_stale_flush_stops` set. - Cancel-shield the logger cleanup loop in the run.py BaseException handler so a cancellation can't interrupt cleanup()'s wait for an in-flight stale flush, which would leave flush timers running against torn-down logging state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(log): shield stale-flush cleanup so cancellation can't hang teardown The stale-flush worker's finally acquired _flush_pending_lock (a cancellation checkpoint) before stopped.set(); under cancellation the acquire could raise and skip stopped.set(), leaving the shielded await stopped.wait() in _stop_stale_flush_timer() hung forever. This manifested as the Python 3.10 CI test job hanging (3.11 unaffected). Shield the finally so stopped.set() always runs. Also harden CI: add timeout-minutes to the test job and a thread-method pytest --timeout so a future hang dumps a traceback (through anyio shields, which the conftest async timeout cannot) instead of riding the 360-minute default. Adds pytest-timeout to dev deps. * fix(types): annotate background coroutine fns for anyio 4.14 start_soon anyio 4.14 tightened TaskGroup.start_soon to require Callable[..., Coroutine[Any, Any, T_co]]; the prior Awaitable[None] annotations were too broad and failed mypy. Coroutine[Any, Any, None] is a subtype of Awaitable[Any], so this is compatible with older anyio too. Also relocks uv.lock to include pytest-timeout (dev dep). * fix(tests): drop stale eval_sandbox kwarg from eval_run test call The main merge removed the eval_sandbox parameter from eval_run (sandbox startup is now internal via startup_sandbox_environments), leaving this test passing a kwarg that no longer exists. mypy on 3.11 flagged it. * fix(deps): require anyio>=4.14.0 for asyncio Lock cancellation fix anyio before 4.14 can leave an asyncio Lock or Semaphore stuck when a waiter is cancelled mid-acquire: release() skips the cancelled waiter but leaves it queued, so later waiters block forever on an unlocked lock (agronholm/anyio#1145, fixed in 4.14.0). The stale-flush logging in this PR acquires _flush_pending_lock in cancellation-heavy paths (idle-timer teardown, sample retries), so on the asyncio backend an eval_set retry run can corrupt the lock and hang. That is the Python 3.10 CI "worker crashed" in test_eval_set_identifiers (the pytest --timeout added here turns the hang into a killed xdist worker). Bump the floor to >=4.14.0 and relock (4.13.0 -> 4.14.0). The branch already adopted anyio 4.14's stricter start_soon typing; this completes that upgrade. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(log): shield stale-flush arm rollback so cancellation can't strand a waiter _arm_stale_flush_timer registers `stopped` in _stale_flush_stops before starting the background task. When run_in_background raises (e.g. start_soon on an already-closing task group during teardown), the rollback except is the only path that signals `stopped` — the task never started, so its shielded finally never runs. That rollback acquired _flush_pending_lock (a cancellation checkpoint) and called stopped.set() unshielded, so a cancellation there could skip stopped.set() and hang a shielded `await stopped.wait()` in _stop_stale_flush_timer(). Shield it, mirroring the teardown in _stale_flush_after_delay, so a registered stopped event is always signaled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(log): make log_finish_stops_stale_flush_timer actually assert the stop The test armed no timer (start=False) and only asserted `_buffer_db is None`, so it proved nothing about its namesake and would pass even if log_finish never stopped the timer (the _running_stale_flush_timer CM stops it in its finally regardless). Arm a sub-threshold stale-flush timer, assert it's running, then assert log_finish stopped it before the CM's finally runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Eric Patey <eric.patey@gmail.com>
* Flush pending eval samples while running * fix(log): drop unused stale-flush event, shield logger cleanup on cancel Address review feedback on UKGovernmentBEIS#4204: - Remove the write-only `_stale_flush_stopped` field; stale-flush synchronization runs entirely through the `_stale_flush_stops` set. - Cancel-shield the logger cleanup loop in the run.py BaseException handler so a cancellation can't interrupt cleanup()'s wait for an in-flight stale flush, which would leave flush timers running against torn-down logging state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(log): shield stale-flush cleanup so cancellation can't hang teardown The stale-flush worker's finally acquired _flush_pending_lock (a cancellation checkpoint) before stopped.set(); under cancellation the acquire could raise and skip stopped.set(), leaving the shielded await stopped.wait() in _stop_stale_flush_timer() hung forever. This manifested as the Python 3.10 CI test job hanging (3.11 unaffected). Shield the finally so stopped.set() always runs. Also harden CI: add timeout-minutes to the test job and a thread-method pytest --timeout so a future hang dumps a traceback (through anyio shields, which the conftest async timeout cannot) instead of riding the 360-minute default. Adds pytest-timeout to dev deps. * fix(types): annotate background coroutine fns for anyio 4.14 start_soon anyio 4.14 tightened TaskGroup.start_soon to require Callable[..., Coroutine[Any, Any, T_co]]; the prior Awaitable[None] annotations were too broad and failed mypy. Coroutine[Any, Any, None] is a subtype of Awaitable[Any], so this is compatible with older anyio too. Also relocks uv.lock to include pytest-timeout (dev dep). * fix(tests): drop stale eval_sandbox kwarg from eval_run test call The main merge removed the eval_sandbox parameter from eval_run (sandbox startup is now internal via startup_sandbox_environments), leaving this test passing a kwarg that no longer exists. mypy on 3.11 flagged it. * fix(deps): require anyio>=4.14.0 for asyncio Lock cancellation fix anyio before 4.14 can leave an asyncio Lock or Semaphore stuck when a waiter is cancelled mid-acquire: release() skips the cancelled waiter but leaves it queued, so later waiters block forever on an unlocked lock (agronholm/anyio#1145, fixed in 4.14.0). The stale-flush logging in this PR acquires _flush_pending_lock in cancellation-heavy paths (idle-timer teardown, sample retries), so on the asyncio backend an eval_set retry run can corrupt the lock and hang. That is the Python 3.10 CI "worker crashed" in test_eval_set_identifiers (the pytest --timeout added here turns the hang into a killed xdist worker). Bump the floor to >=4.14.0 and relock (4.13.0 -> 4.14.0). The branch already adopted anyio 4.14's stricter start_soon typing; this completes that upgrade. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(log): shield stale-flush arm rollback so cancellation can't strand a waiter _arm_stale_flush_timer registers `stopped` in _stale_flush_stops before starting the background task. When run_in_background raises (e.g. start_soon on an already-closing task group during teardown), the rollback except is the only path that signals `stopped` — the task never started, so its shielded finally never runs. That rollback acquired _flush_pending_lock (a cancellation checkpoint) and called stopped.set() unshielded, so a cancellation there could skip stopped.set() and hang a shielded `await stopped.wait()` in _stop_stale_flush_timer(). Shield it, mirroring the teardown in _stale_flush_after_delay, so a registered stopped event is always signaled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(log): make log_finish_stops_stale_flush_timer actually assert the stop The test armed no timer (start=False) and only asserted `_buffer_db is None`, so it proved nothing about its namesake and would pass even if log_finish never stopped the timer (the _running_stale_flush_timer CM stops it in its finally regardless). Arm a sub-threshold stale-flush timer, assert it's running, then assert log_finish stopped it before the CM's finally runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Eric Patey <eric.patey@gmail.com>
--------- Co-authored-by: Alex Grönholm <alex.gronholm@nextday.fi>
The harness surfaced a production-reachable deadlock in the pinned anyio
4.13.0 but left it as prose: `Lock.release()` skips a waiter whose future is
already cancelled yet leaves its entry queued, so when the bridge reader is
raw-cancelled (`_await_cancelled_task`) while parked on
`session.pending_lock` in the same tick as the holder's release, the lock is
left unowned with a stale entry, an already-runnable acquirer takes the slow
path and parks, and the cancelled waiter's cleanup finds no owner to hand
over from. anyio 4.14.0 fixed it ("Fixed asyncio Lock and Semaphore
deadlocks caused by cancelled waiters left queued during release",
agronholm/anyio#1145). Without a pin, anyone widening `_SCHEDULE_COUNT` gets
"schedule did not quiesce" and cannot tell harness rot from a product or
dependency defect.
Add a minimal stdlib+anyio reproduction
(`test_anyio_lock_cancelled_waiter.py`, ~10 loop turns, no real sleep) and
pin the production-turn seeds that wedge `pending_lock` under the default
profile in the first 3000 (`_ANYIO_LOCK_WEDGE_SEEDS`, currently 1234; the
default 200-seed run and the other latency profiles' 200 seeds are free of
wedges), both as `xfail(strict=True, raises=AssertionError)` conditioned on
`anyio < 4.14`, so a dependency bump turns them into required passes. Seed
1234 passes the full invariant set with the 4.14 `Lock.release` fix
monkeypatched in. The dependency bump itself is a separate decision.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ck (#47378) anyio below 4.14.0 can leave a cancelled waiter queued on an asyncio Lock or Semaphore, so the next task to acquire waits forever on a lock nobody holds (agronholm/anyio#1145, fixed in 4.14.0). httpcore takes these locks for every httpx request, including the prisma query-engine client litellm imports anyio directly but never declared it, so pip and uv kept an already installed anyio 4.13.0 when installing litellm and accepted an explicit anyio==4.13.0 next to it. uv.lock already resolves 4.14.2, so Docker and CI installs do not change. Both the litellm and litellm-core manifests get the floor, and the lock only records the new direct dependency Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
NOTE Erasing or replacing the contents of this template will result in your pull
request being summarily closed without consideration!
Changes
Cancelled asyncio lock/semaphore waiters can be observed by
release()before theiracquire()cleanup has removed them from the wait queue. In that window,release()skipped the cancelled waiter but left it queued, allowing a later live waiter to become stranded with no future release to wake it.This PR prunes cancelled waiters during asyncio
LockandSemaphorerelease, while preserving the existing cancellation-after-handoff behavior. It also adds regression coverage for the interleaving where a cancelled head waiter would otherwise orphan a later waiter.Checklist
If this is a user-facing code change, like a bugfix or a new feature, please ensure that
you've fulfilled the following conditions (where applicable):
tests/) which would fail without your patchdocs/), in case of behavior changes or newfeatures
docs/versionhistory.rst).If this is a trivial change, like a typo fix or a code reformatting, then you can ignore
these instructions.
Updating the changelog
If there are no entries after the last release, use
**UNRELEASED**as the version.If, say, your patch fixes issue #123, the entry should look like this:
If there's no issue linked, just link to your pull request instead by updating the
changelog after you've created the PR.