Skip to content

Fix asyncio lock waiter deadlocks after cancellation - #1145

Merged
agronholm merged 11 commits into
agronholm:masterfrom
rasmusfaber:fix-asyncio-sync-cancelled-waiters
Jun 15, 2026
Merged

agronholm merged 11 commits into
agronholm:masterfrom
rasmusfaber:fix-asyncio-sync-cancelled-waiters

Conversation

@rasmusfaber

Copy link
Copy Markdown
Contributor

NOTE Erasing or replacing the contents of this template will result in your pull
request being summarily closed without consideration!

Changes

Cancelled asyncio lock/semaphore waiters can be observed by release() before their acquire() cleanup has removed them from the wait queue. In that window, release() skipped the cancelled waiter but left it queued, allowing a later live waiter to become stranded with no future release to wake it.

This PR prunes cancelled waiters during asyncio Lock and Semaphore release, while preserving the existing cancellation-after-handoff behavior. It also adds regression coverage for the interleaving where a cancelled head waiter would otherwise orphan a later waiter.

Checklist

If this is a user-facing code change, like a bugfix or a new feature, please ensure that
you've fulfilled the following conditions (where applicable):

  • You've added tests (in tests/) which would fail without your patch
  • You've updated the documentation (in docs/), in case of behavior changes or new
    features
  • You've added a new changelog entry (in docs/versionhistory.rst).

If this is a trivial change, like a typo fix or a code reformatting, then you can ignore
these instructions.

Updating the changelog

If there are no entries after the last release, use **UNRELEASED** as the version.
If, say, your patch fixes issue #123, the entry should look like this:

- Fix big bad boo-boo in task groups
  (`#123 <https://github.com/agronholm/anyio/issues/123>`_; PR by @yourgithubaccount)

If there's no issue linked, just link to your pull request instead by updating the
changelog after you've created the PR.

@rasmusfaber
rasmusfaber force-pushed the fix-asyncio-sync-cancelled-waiters branch from 2ccf10e to cde0728 Compare May 11, 2026 20:48
@agronholm

Copy link
Copy Markdown
Owner

Are these super complicated tests really required to reproduce the issue?

@agronholm

Copy link
Copy Markdown
Owner

Hm, I'm starting to see the logic behind the tests. Could they not use AnyIO APIs though, and also run on Trio?

Comment thread tests/test_synchronization.py Outdated
@rasmusfaber
rasmusfaber force-pushed the fix-asyncio-sync-cancelled-waiters branch from cde0728 to dbb003e Compare May 11, 2026 22:06
@agronholm

Copy link
Copy Markdown
Owner

Please don't force-push changes. That way I can't see what you've changed.

@agronholm

Copy link
Copy Markdown
Owner

Thanks, I'll review when I get the time.

@rasmusfaber

Copy link
Copy Markdown
Contributor Author

Are these super complicated tests really required to reproduce the issue?

It is the minimal tests I could come up with that reproduces the race: a cancelled head waiter must still be queued when release runs, and a later waiter must queue before the cancelled task's cleanup removes itself.

We could drop test_cancelled_head_waiter_does_not_overrelease(). It just guards against an incomplete semaphore fix.

@rasmusfaber

Copy link
Copy Markdown
Contributor Author

Please don't force-push changes. That way I can't see what you've changed.

Ah, sorry. I will stop doing that.

@agronholm

Copy link
Copy Markdown
Owner

I'm currently spending some time trying to understand the conditions under which this problem can occur. My own attempts to reproduce the problem have all failed. I can move forward with this PR once I fully understand what's going on.

@x42005e1f

x42005e1f commented Jun 14, 2026 •

Copy link
Copy Markdown

Are these super complicated tests really required to reproduce the issue?

I agree; the tests do seem overly complicated. In fact, the issue can be reproduced with just two tasks:

#!/usr/bin/env python3

import anyio


async def main():
    lock = anyio.Lock()
    lock_acquire = lock.acquire
    lock_release = lock.release
    await lock_acquire()

    async with anyio.create_task_group() as tg:
        task1 = tg.create_task(lock_acquire())
        await anyio.wait_all_tasks_blocked()

        task2 = tg.create_task(lock_acquire())
        task1.cancel()
        lock_release()

        with anyio.fail_after(1):
            await task2


if __name__ == "__main__":
    anyio.run(main)

I tested asyncio.Lock, anyio.Lock, and aiologic.Lock, and only anyio.Lock proved to be affected. As I see it, the problem is that the asyncio future object is cancelled immediately, but the task will not resume until its callback is invoked:

#!/usr/bin/env python3

import asyncio


async def main():
    future = asyncio.get_running_loop().create_future()

    async def work():
        try:
            await future
        except asyncio.CancelledError:
            print("task: cancelled")

    task = asyncio.create_task(work())
    await asyncio.sleep(0)  # switch to the task

    task.cancel()
    print(f"future: {'cancelled' if future.cancelled() else 'pending'}")

    await task


if __name__ == "__main__":
    asyncio.run(main())
future: cancelled
task: cancelled

As a result:

  1. The second task is created before task1.cancel() is called, so the second acquire will precede the finalization of the cancelled first task (callbacks are invoked by the asyncio event loop in FIFO order).
  2. At the moment lock_release() is called, the second task has not yet started execution, and the future object of the first task has already been cancelled, so self._owner_task = None, but the queue still contains the future object of the cancelled task.
  3. The second task enters lock_acquire() and sees that the self.owner_task is None and not self.waiters condition (fast path) is false (the queue is not empty!), so it joins the wait queue.
  4. The first task resumes and sees that it has been cancelled, but the self.owner_task is task condition is false, so it does nothing except remove its future object from the queue. No self.release() is called.
  5. The second task remains in the queue forever.

The solution in this PR is similar to how my primitives work, except that my popleft() is unconditional (and there are other differences with state passing, since I have to support threads). Is there a strong reason to leave the queue untouched when ownership is successfully transferred? If we make it unconditional, we can eliminate the second self._waiters.remove(item) in the acquire() method.


Note that asyncio takes a different approach and modifies the fast-path condition. This is also a possible solution, but it seems far less optimal to me (it is effectively O(n) just to check for the possibility of acquiring without waiting!).

However, the real solution lies not in that, but in the fact that a cancelled task always wakes up the first one in the queue if the lock has not been acquired. In theory, we could repeat that behavior by moving the main part of the release() method into a separate private method and adding the elif self._owner_task is None branch that calls it. But I still think that scanning the entire queue every time to find an uncancelled task is not a very good idea from a performance standpoint. If we have n tasks and they are all cancelled, the total number of iterations will be O(n²).


...the same example, but with just one task:

#!/usr/bin/env python3

import anyio


async def main():
    lock = anyio.Lock()
    lock_acquire = lock.acquire
    lock_release = lock.release

    async with anyio.create_task_group() as tg:
        await lock_acquire()

        task = tg.create_task(lock_acquire())
        await anyio.wait_all_tasks_blocked()

        task.cancel()
        lock_release()

        with anyio.fail_after(1):
            await lock_acquire()


if __name__ == "__main__":
    anyio.run(main)

The explanation is the same as well.

Comment thread src/anyio/_backends/_asyncio.py Outdated
Comment on lines +1883 to +1886
try:
self._waiters.remove(item)
except ValueError:
pass

@x42005e1f x42005e1f Jun 14, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would also result in O(n²) iterations, since self._waiters.remove(item) takes O(n) time for a missing item. Hypothetically, we could mark visited items in the release() method and only perform the removal when there is no such marker, but I am not sure how to do this efficiently in terms of memory for asyncio futures.

It would likely be worth changing the items (queue tokens) slightly so that they include one mutable value (for example, replacing a two-element tuple with a three-element list).

>>> from sys import getsizeof
>>> getsizeof((1, 2))  # immutable
64
>>> getsizeof([1, 2, 3])  # mutable
88
>>> class LockWaiter:
...     __slots__ = ("task", "future", "removed")
>>> getsizeof(LockWaiter())  # also mutable
56  # without slots: 328 == getsizeof(LockWaiter()) + getsizeof(LockWaiter().__dict__)

@x42005e1f

x42005e1f commented Jun 14, 2026 •

Copy link
Copy Markdown

Perhaps understanding the conditions under which the problem occurs will help in understanding it.

For the issue to be reproduced, four conditions must be met:

  1. The lock must be acquired.
  2. The wait queue must contain only items of cancelled tasks, with at least one such item present. If there is even one uncancelled task or the queue is empty, this condition is not met.
  3. The lock must be released before the last cancelled task is resumed. That is, the queue must not be empty when the lock is released.
  4. There must be at least one attempt to acquire the lock before the last cancelled task is resumed, and it must not precede the release. That is, the queue must not be empty when the lock is acquired.

@agronholm

Copy link
Copy Markdown
Owner

It took me a while, but I now understand that it's here where things go wrong:

        if self._owner_task is None and not self._waiters:

What is there is no current owner but there are waiters, all of which have been cancelled? The latest task would be put in a waiting state, waiting for a release that never comes, as the cancelled waiters only do that if they've received the lock.

@agronholm

Copy link
Copy Markdown
Owner

@x42005e1f I pushed some simplifications to this branch, including the unconditional popleft() you suggested.

@agronholm

Copy link
Copy Markdown
Owner

I removed the other semaphore test which was already passing without these changes. I hope that was okay. In any case, I'm ready to merge unless anyone watching this has objections.

@x42005e1f

x42005e1f commented Jun 15, 2026 •

Copy link
Copy Markdown

I also took another look at this block in the lock and semaphore:

self._waiters.append(item)
try:
    await fut
except CancelledError:
    ...

It looks inconsistent, and in the case of the semaphore, it could be optimized. I suggest using the following implementation for both:

self._waiters.append(item)
try:
    await fut
except CancelledError:
    if fut.cancelled():
        try:
            self._waiters.remove(fut)
        except ValueError:  # already removed (in release())
            pass
    else:  # notified
        self.release()

    raise

It can be further improved by adding markers, but this way it will no longer attempt to remove a non-existent item when the notification has already happened (and thus perform the expensive O(n) operation on every semaphore release!). After all, if the notification has happened, then due to the unconditional popleft(), the item is definitely no longer in the queue. Also, in the case of locks, the notification always happens with ownership transfer, so this implementation is valid for locks as well.

@agronholm

agronholm commented Jun 15, 2026 •

Copy link
Copy Markdown
Owner

I also took another look at this block in the lock and semaphore:

self._waiters.append(item)
try:
    await fut
except CancelledError:
    ...

It looks inconsistent, and in the case of the semaphore, it could be optimized. I suggest using the following implementation for both:

self._waiters.append(item)
try:
    await fut
except CancelledError:
    if fut.cancelled():
        try:
            self._waiters.remove(fut)
        except ValueError:  # already removed (in release())
            pass
    else:  # notified
        self.release()

    raise

It can be further improved by adding markers, but this way it will no longer attempt to remove a non-existent item when the notification has already happened (and thus perform the expensive O(n) operation on every semaphore release!). After all, if the notification has happened, then due to the unconditional popleft(), the item is definitely no longer in the queue. Also, in the case of locks, the notification always happens with ownership transfer, so this implementation is valid for locks as well.

Under what circumstances would await fut get a CancelledError without fut being cancelled? That makes no sense.

@agronholm

Copy link
Copy Markdown
Owner

Does the concern about correctness have to do only with the fact that we only handle CancelledError?

@x42005e1f

Copy link
Copy Markdown

Yes, of course. The code seems to handle direct task cancellations, and as a result, it contains redundant logic. At the same time, it does not handle other types of exceptions. And my concerns about performance remain.

@x42005e1f

x42005e1f commented Jun 15, 2026 •

Copy link
Copy Markdown

A side effect of this PR: lock.acquire_nowait() will behave as expected when all waiters have been cancelled (the same self._owner_task is None and not self._waiters check).

@agronholm

agronholm commented Jun 15, 2026 •

Copy link
Copy Markdown
Owner

Alright, I've pushed your optimizations. All good now?

@x42005e1f

Copy link
Copy Markdown

Hmm, wait a minute. If release() always transfers ownership, then what is the point of checking whether the queue is not empty? And as I noticed, semaphore.acquire() and semaphore.acquire_nowait() have different checks (and not self._waiters in the first, but not in the second).

@agronholm

Copy link
Copy Markdown
Owner

I think you'll have to elaborate. My brain is quite tired at this time of night.

@x42005e1f

Copy link
Copy Markdown

The acquire() method can do two things: either acquire the lock/semaphore immediately, or join the queue until the lock/semaphore is released.

In the case of locks, the release finds the first uncancelled task (if there is one) and transfers ownership to it. If there is no such task, the queue will be empty, and there will be no owner. A situation where there is no owner but the queue is not empty cannot occur.

In the case of semaphores, a similar thing happens, but when there is no uncancelled task, the value is incremented. It is impossible to imagine a situation where the value is non-zero but there is still someone in the queue.

In both cases, checking whether the queue is non-empty (and not self._waiters) is redundant.

@agronholm

agronholm commented Jun 15, 2026 •

Copy link
Copy Markdown
Owner

I'm not 100% whether this is true, but the performance impact of said check is negligible so I don't think I'm going to touch that check at this point. This is the only PR left blocking the v4.14 release so we can revisit this later if there are no more impactful changes or concerns.

@x42005e1f

Copy link
Copy Markdown

I agree.

@agronholm

Copy link
Copy Markdown
Owner

Thanks @rasmusfaber for the PR and @x42005e1f for the in-depth review!

@agronholm
agronholm merged commit f8b9f01 into agronholm:master Jun 15, 2026
19 checks passed
@x42005e1f

x42005e1f commented Jun 15, 2026 •

Copy link
Copy Markdown

Interestingly, my comment about redundant checks still holds true even before the changes in this PR, which hints at one possibility: in fact, removing and not self._waiters from the checks was enough to solve the problem (if there is no owner / value > 0, why should the fact that the queue is not empty prevent the primitive from being acquired?). But I think it is still good that we switched to popleft(). From a performance (and statistics) standpoint, this looks much better than what we had before.

It likely came to us from asyncio, where self._locked = True is set in acquire() instead of release(), and the AnyIO implementation was inspired by this, but then moved the action to release(). In asyncio, the waiters update the attributes themselves (that is, release always sets self._locked = False, even when notifying a task in the queue, and therefore the primitive always looks unlocked after the call), so that check is necessary and sufficient there (even cancelled waiters can notify tasks in the queue if the lock has not been acquired). But when the attributes are updated by release(), the check is useless and harmful.

rasmusfaber added a commit to METR/hawk that referenced this pull request Jun 17, 2026
PR #403 pinned a temporary fork of AnyIO 4.13.0 to fix runner deadlocks
on cancelled Lock/Semaphore waiters, because the upstream fix wasn't
released yet. AnyIO 4.14.0 now ships that fix (agronholm/anyio#1145), so
drop the fork override and pin the official release across all lockfiles.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rasmusfaber added a commit to METR/inspect_ai that referenced this pull request Jun 22, 2026
anyio before 4.14 can leave an asyncio Lock or Semaphore stuck when a
waiter is cancelled mid-acquire: release() skips the cancelled waiter but
leaves it queued, so later waiters block forever on an unlocked lock
(agronholm/anyio#1145, fixed in 4.14.0).

The stale-flush logging in this PR acquires _flush_pending_lock in
cancellation-heavy paths (idle-timer teardown, sample retries), so on the
asyncio backend an eval_set retry run can corrupt the lock and hang. That
is the Python 3.10 CI "worker crashed" in test_eval_set_identifiers (the
pytest --timeout added here turns the hang into a killed xdist worker).

Bump the floor to >=4.14.0 and relock (4.13.0 -> 4.14.0). The branch
already adopted anyio 4.14's stricter start_soon typing; this completes
that upgrade.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rasmusfaber added a commit to METR/inspect_ai that referenced this pull request Jun 22, 2026
anyio before 4.14 can leave an asyncio Lock or Semaphore stuck when a
waiter is cancelled mid-acquire: release() skips the cancelled waiter but
leaves it queued, so later waiters block forever on an unlocked lock
(agronholm/anyio#1145, fixed in 4.14.0).

The stale-flush logging in this PR acquires _flush_pending_lock in
cancellation-heavy paths (idle-timer teardown, sample retries), so on the
asyncio backend an eval_set retry run can corrupt the lock and hang. That
is the Python 3.10 CI "worker crashed" in test_eval_set_identifiers (the
pytest --timeout added here turns the hang into a killed xdist worker).

Bump the floor to >=4.14.0 and relock (4.13.0 -> 4.14.0). The branch
already adopted anyio 4.14's stricter start_soon typing; this completes
that upgrade.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
epatey added a commit to UKGovernmentBEIS/inspect_ai that referenced this pull request Jun 23, 2026
* Flush pending eval samples while running

* fix(log): drop unused stale-flush event, shield logger cleanup on cancel

Address review feedback on #4204:
- Remove the write-only `_stale_flush_stopped` field; stale-flush
  synchronization runs entirely through the `_stale_flush_stops` set.
- Cancel-shield the logger cleanup loop in the run.py BaseException
  handler so a cancellation can't interrupt cleanup()'s wait for an
  in-flight stale flush, which would leave flush timers running against
  torn-down logging state.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(log): shield stale-flush cleanup so cancellation can't hang teardown

The stale-flush worker's finally acquired _flush_pending_lock (a
cancellation checkpoint) before stopped.set(); under cancellation the
acquire could raise and skip stopped.set(), leaving the shielded
await stopped.wait() in _stop_stale_flush_timer() hung forever. This
manifested as the Python 3.10 CI test job hanging (3.11 unaffected).
Shield the finally so stopped.set() always runs.

Also harden CI: add timeout-minutes to the test job and a thread-method
pytest --timeout so a future hang dumps a traceback (through anyio
shields, which the conftest async timeout cannot) instead of riding the
360-minute default. Adds pytest-timeout to dev deps.

* fix(types): annotate background coroutine fns for anyio 4.14 start_soon

anyio 4.14 tightened TaskGroup.start_soon to require
Callable[..., Coroutine[Any, Any, T_co]]; the prior Awaitable[None]
annotations were too broad and failed mypy. Coroutine[Any, Any, None]
is a subtype of Awaitable[Any], so this is compatible with older anyio
too. Also relocks uv.lock to include pytest-timeout (dev dep).

* fix(tests): drop stale eval_sandbox kwarg from eval_run test call

The main merge removed the eval_sandbox parameter from eval_run (sandbox
startup is now internal via startup_sandbox_environments), leaving this
test passing a kwarg that no longer exists. mypy on 3.11 flagged it.

* fix(deps): require anyio>=4.14.0 for asyncio Lock cancellation fix

anyio before 4.14 can leave an asyncio Lock or Semaphore stuck when a
waiter is cancelled mid-acquire: release() skips the cancelled waiter but
leaves it queued, so later waiters block forever on an unlocked lock
(agronholm/anyio#1145, fixed in 4.14.0).

The stale-flush logging in this PR acquires _flush_pending_lock in
cancellation-heavy paths (idle-timer teardown, sample retries), so on the
asyncio backend an eval_set retry run can corrupt the lock and hang. That
is the Python 3.10 CI "worker crashed" in test_eval_set_identifiers (the
pytest --timeout added here turns the hang into a killed xdist worker).

Bump the floor to >=4.14.0 and relock (4.13.0 -> 4.14.0). The branch
already adopted anyio 4.14's stricter start_soon typing; this completes
that upgrade.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(log): shield stale-flush arm rollback so cancellation can't strand a waiter

_arm_stale_flush_timer registers `stopped` in _stale_flush_stops before
starting the background task. When run_in_background raises (e.g. start_soon
on an already-closing task group during teardown), the rollback except is the
only path that signals `stopped` — the task never started, so its shielded
finally never runs. That rollback acquired _flush_pending_lock (a cancellation
checkpoint) and called stopped.set() unshielded, so a cancellation there could
skip stopped.set() and hang a shielded `await stopped.wait()` in
_stop_stale_flush_timer().

Shield it, mirroring the teardown in _stale_flush_after_delay, so a registered
stopped event is always signaled.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(log): make log_finish_stops_stale_flush_timer actually assert the stop

The test armed no timer (start=False) and only asserted `_buffer_db is None`,
so it proved nothing about its namesake and would pass even if log_finish
never stopped the timer (the _running_stale_flush_timer CM stops it in its
finally regardless). Arm a sub-threshold stale-flush timer, assert it's
running, then assert log_finish stopped it before the CM's finally runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Eric Patey <eric.patey@gmail.com>
rasmusfaber added a commit to METR/inspect_ai that referenced this pull request Jun 25, 2026
* Flush pending eval samples while running

* fix(log): drop unused stale-flush event, shield logger cleanup on cancel

Address review feedback on UKGovernmentBEIS#4204:
- Remove the write-only `_stale_flush_stopped` field; stale-flush
  synchronization runs entirely through the `_stale_flush_stops` set.
- Cancel-shield the logger cleanup loop in the run.py BaseException
  handler so a cancellation can't interrupt cleanup()'s wait for an
  in-flight stale flush, which would leave flush timers running against
  torn-down logging state.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(log): shield stale-flush cleanup so cancellation can't hang teardown

The stale-flush worker's finally acquired _flush_pending_lock (a
cancellation checkpoint) before stopped.set(); under cancellation the
acquire could raise and skip stopped.set(), leaving the shielded
await stopped.wait() in _stop_stale_flush_timer() hung forever. This
manifested as the Python 3.10 CI test job hanging (3.11 unaffected).
Shield the finally so stopped.set() always runs.

Also harden CI: add timeout-minutes to the test job and a thread-method
pytest --timeout so a future hang dumps a traceback (through anyio
shields, which the conftest async timeout cannot) instead of riding the
360-minute default. Adds pytest-timeout to dev deps.

* fix(types): annotate background coroutine fns for anyio 4.14 start_soon

anyio 4.14 tightened TaskGroup.start_soon to require
Callable[..., Coroutine[Any, Any, T_co]]; the prior Awaitable[None]
annotations were too broad and failed mypy. Coroutine[Any, Any, None]
is a subtype of Awaitable[Any], so this is compatible with older anyio
too. Also relocks uv.lock to include pytest-timeout (dev dep).

* fix(tests): drop stale eval_sandbox kwarg from eval_run test call

The main merge removed the eval_sandbox parameter from eval_run (sandbox
startup is now internal via startup_sandbox_environments), leaving this
test passing a kwarg that no longer exists. mypy on 3.11 flagged it.

* fix(deps): require anyio>=4.14.0 for asyncio Lock cancellation fix

anyio before 4.14 can leave an asyncio Lock or Semaphore stuck when a
waiter is cancelled mid-acquire: release() skips the cancelled waiter but
leaves it queued, so later waiters block forever on an unlocked lock
(agronholm/anyio#1145, fixed in 4.14.0).

The stale-flush logging in this PR acquires _flush_pending_lock in
cancellation-heavy paths (idle-timer teardown, sample retries), so on the
asyncio backend an eval_set retry run can corrupt the lock and hang. That
is the Python 3.10 CI "worker crashed" in test_eval_set_identifiers (the
pytest --timeout added here turns the hang into a killed xdist worker).

Bump the floor to >=4.14.0 and relock (4.13.0 -> 4.14.0). The branch
already adopted anyio 4.14's stricter start_soon typing; this completes
that upgrade.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(log): shield stale-flush arm rollback so cancellation can't strand a waiter

_arm_stale_flush_timer registers `stopped` in _stale_flush_stops before
starting the background task. When run_in_background raises (e.g. start_soon
on an already-closing task group during teardown), the rollback except is the
only path that signals `stopped` — the task never started, so its shielded
finally never runs. That rollback acquired _flush_pending_lock (a cancellation
checkpoint) and called stopped.set() unshielded, so a cancellation there could
skip stopped.set() and hang a shielded `await stopped.wait()` in
_stop_stale_flush_timer().

Shield it, mirroring the teardown in _stale_flush_after_delay, so a registered
stopped event is always signaled.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(log): make log_finish_stops_stale_flush_timer actually assert the stop

The test armed no timer (start=False) and only asserted `_buffer_db is None`,
so it proved nothing about its namesake and would pass even if log_finish
never stopped the timer (the _running_stale_flush_timer CM stops it in its
finally regardless). Arm a sub-threshold stale-flush timer, assert it's
running, then assert log_finish stopped it before the CM's finally runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Eric Patey <eric.patey@gmail.com>
agronholm added a commit that referenced this pull request Aug 4, 2026
---------

Co-authored-by: Alex Grönholm <alex.gronholm@nextday.fi>
Soju06 added a commit to Soju06/codex-lb that referenced this pull request Sep 5, 2026
The harness surfaced a production-reachable deadlock in the pinned anyio
4.13.0 but left it as prose: `Lock.release()` skips a waiter whose future is
already cancelled yet leaves its entry queued, so when the bridge reader is
raw-cancelled (`_await_cancelled_task`) while parked on
`session.pending_lock` in the same tick as the holder's release, the lock is
left unowned with a stale entry, an already-runnable acquirer takes the slow
path and parks, and the cancelled waiter's cleanup finds no owner to hand
over from. anyio 4.14.0 fixed it ("Fixed asyncio Lock and Semaphore
deadlocks caused by cancelled waiters left queued during release",
agronholm/anyio#1145). Without a pin, anyone widening `_SCHEDULE_COUNT` gets
"schedule did not quiesce" and cannot tell harness rot from a product or
dependency defect.

Add a minimal stdlib+anyio reproduction
(`test_anyio_lock_cancelled_waiter.py`, ~10 loop turns, no real sleep) and
pin the production-turn seeds that wedge `pending_lock` under the default
profile in the first 3000 (`_ANYIO_LOCK_WEDGE_SEEDS`, currently 1234; the
default 200-seed run and the other latency profiles' 200 seeds are free of
wedges), both as `xfail(strict=True, raises=AssertionError)` conditioned on
`anyio < 4.14`, so a dependency bump turns them into required passes. Seed
1234 passes the full invariant set with the 4.14 `Lock.release` fix
monkeypatched in. The dependency bump itself is a separate decision.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
yucheng-berri added a commit to BerriAI/litellm that referenced this pull request Oct 19, 2026
…ck (#47378)

anyio below 4.14.0 can leave a cancelled waiter queued on an asyncio Lock or
Semaphore, so the next task to acquire waits forever on a lock nobody holds
(agronholm/anyio#1145, fixed in 4.14.0). httpcore takes these locks for every
httpx request, including the prisma query-engine client

litellm imports anyio directly but never declared it, so pip and uv kept an
already installed anyio 4.13.0 when installing litellm and accepted an
explicit anyio==4.13.0 next to it. uv.lock already resolves 4.14.2, so Docker
and CI installs do not change. Both the litellm and litellm-core manifests get
the floor, and the lock only records the new direct dependency

Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants