test_deadlock & test_threads sometimes deadlock on Windows refleaks buildbot

The “AMD64 Windows11 Refleaks 3.12” sometimes deadlocks with:

3:58:18 load avg: 0.07 running (1): test.test_concurrent_futures.test_deadlock (2 hour 34 min)
3:58:48 load avg: 0.05 running (1): test.test_concurrent_futures.test_deadlock (2 hour 35 min)
3:59:18 load avg: 0.03 running (1): test.test_concurrent_futures.test_deadlock (2 hour 35 min)
3:59:48 load avg: 0.02 running (1): test.test_concurrent_futures.test_deadlock (2 hour 36 min)
command timed out: 14400 seconds elapsed running [b'Tools\\buildbot\\test.bat', b'-p', b'x64', b'-j2', b'-R', b'3:3', b'-u-cpu', b'-j2', b'--timeout', b'12000'], attempting to kill
4:00:18 load avg: 0.03 running (1): test.test_concurrent_futures.test_deadlock (2 hour 36 min)
program finished with exit code 1
elapsedTime=14430.167703

On some builds, a different test fails:

3:58:59 load avg: 0.03 running (1): test.test_multiprocessing_spawn.test_threads (1 hour 43 min)
3:59:29 load avg: 0.06 running (1): test.test_multiprocessing_spawn.test_threads (1 hour 44 min)
command timed out: 14400 seconds elapsed running [b'Tools\\buildbot\\test.bat', b'-p', b'x64', b'-j2', b'-R', b'3:3', b'-u-cpu', b'-j2', b'--timeout', b'12000'], attempting to kill
3:59:59 load avg: 0.04 running (1): test.test_multiprocessing_spawn.test_threads (1 hour 44 min)
program finished with exit code 1
elapsedTime=14430.167536

I am able to reproduce locally, by running these two (or this and test_concurrent_futures.test_shutdown) in parallel. The deadlock occurs in test_crash_big_data.

Linked PRs

[3.12] gh-114440: Close writer pipe in multiprocessing.Queue, not concurrent.futures #114489

[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…

In the 3.12 backports for gh-109047, gh-109370, and gh-107219, a backporting/rebasing error snuck in:

test_concurrent_futures.test_deadlock: test_crash_big_data() hangs randomly on Windows #107219 adds self.call_queue._writer.close() to _ExecutorManagerThread in concurrent.futures.
Unexpected traceback output in test_concurrent_futures and crash #109370 changes this to be only called on Windows.
test_concurrent_futures.test_shutdown: test_interpreter_shutdown() fails randomly (race condition) #109047 moves the call to multiprocessing.Queue's _terminate_broken.

The backports to 3.12 were applied in a different order (and with different bases), leaving the Windows-specific close call in concurrent.futures. Moving it to multiprocessing.Queue fixes test_concurrent_futures.test_deadlock for me.

[3.12] gh-114440: Close writer pipe in multiprocessing.Queue, not con…

Thanks Victor & Serhiy!

I'll close the issue but continue monitoring buildbots.

Victor pointed out that this might need a backport to 3.11.
( I haven't seen it on the 3.11 buildbots, and didn't try to reproduce there.)

I cannot reproduce on 3.11. (I am using a simplified reproducer. But I also haven't seen this fail on a 3.11 buildbot yet.)
If it comes up in the future, #109047 (356de02) probably needs to be backported before this.

[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…

Navigation Menu

Repository navigation

test_deadlock & test_threads sometimes deadlock on Windows refleaks buildbot #114440

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test_deadlock & test_threads sometimes deadlock on Windows refleaks buildbot #114440

Description

Linked PRs

Activity

encukou commented on Jan 23, 2024

encukou commented on Jan 24, 2024

encukou commented on Jan 26, 2024

encukou commented on Jan 29, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions