wait_for usage is unsafe on CPython 3.11 #1413

Activity

1. dvarrazzo commented on Sep 17, 2026

   

   dvarrazzo

   on Sep 17, 2026

   Member

   More actions

   Hello,

   I don't think that fixing this is our responsibility. Python 3.11 is on its way out so replace a core builtin function seems at the same time dangerous and of limited utility, considering that a program might use Psycopg together with other users of wait_for() in other libraries.

   Also note that we have just fixed a few cases of mishandling of async cancellation (#1371 chiefly): maybe there's a chance that what was biting was something else? Planning to release 3.3.6 today or tomorrow.

   I think it would be safe to just suggest your user to use Python 3.12, which has been around for 3 years already. People who use Python 3.11 should be aware that they are missing at least 3 releases of improvement at this point and this issue is not different.
2. dvarrazzo

   closed this as not plannedon Sep 17, 2026
3. TkTech commented on Sep 17, 2026

   

   TkTech

   on Sep 17, 2026

   Author

   More actions

   Also note that we have just fixed a few cases of mishandling of async cancellation (#1371 chiefly): maybe there's a chance that what was biting was something else?

   Nope, definitely this issue - wait_for() is genuinely unsafe to use in 3.11 and will eat cancellations.

   Planning to release 3.3.6 today or tomorrow.

   Is 3.3.6 dropping 3.11 support?
4. dvarrazzo commented on Sep 17, 2026

   

   dvarrazzo

   on Sep 17, 2026

   Member

   More actions

   Is 3.3.6 dropping 3.11 support?

   No, we usually drop old versions support on minor releases, not on bugfix releases.

   Nope, definitely this issue - wait_for() is genuinely unsafe to use in 3.11 and will eat cancellations.

   This has not been shown us in the past 3 years that Python 3.11 has been released. Can you write a test to show it?
5. dvarrazzo commented on Sep 18, 2026

   

   dvarrazzo

   on Sep 18, 2026

   Member

   More actions

   In a surprising turn of events, working on the optimization in wait_async() in #1416, the use of Event and wait_for() has been dropped in favour of using a Future to coordinate, so the issue you are experiencing should probably be solved with the release 3.3.6 (to be released as soon as #1416 is merged).
6. TkTech commented on Sep 19, 2026

   

   TkTech

   on Sep 19, 2026

   Author

   More actions

   Can you write a test to show it?

   Sure, here's a minimal example that just shows the flaw in waiting.wait_async. It'll swallow cancellations in 3.11 and work on all other versions. This will deliver a cancellation in the same frame an I/O wait finishes.

     import asyncio
     import socket
     from unittest.mock import patch
   
     from psycopg import waiting
   
   
     async def main():
         class CancelOnReady(asyncio.Event):
             async def wait(self):
                 await super().wait()
                 task.cancel()  # Cancel exactly when the I/O wait completes.
   
         def operation():
             yield waiting.WAIT_R
             return "BUG: cancellation was swallowed"
   
         reader, writer = socket.socketpair()
         with reader, writer, patch.object(waiting, "Event", CancelOnReady):
             writer.send(b"x")
             task = asyncio.create_task(
                 waiting.wait_async(operation(), reader.fileno(), interval=1)
             )
             try:
                 print(await task)
             except asyncio.CancelledError:
                 print("OK: cancellation propagated")
   
   
     asyncio.run(main())

   This has not been shown us in the past 3 years that Python 3.11 has been released.

   It's very annoying to pin down to psycopg, since it simply discards the cancellation. For most use cases I imagine it wouldn't look like an issue in psycopg at all, since it continues - it would just manifest as a wait on a future that uses psycopg never getting cancelled - stuck forever when you try to clean it up.

   In a surprising turn of events, working on the optimization in wait_async() in #1416, the use of Event and wait_for() has been dropped in favour of using a Future to coordinate, so the issue you are experiencing should probably be solved with the release 3.3.6 (to be released as soon as #1416 is merged).

   Fantastic!