Bug report

Bug description:

A daemon thread that is waiting for the GIL during interpreter shutdown can leave the main thread blocked forever in drop_gil().

In take_gil(), the _PyThreadState_MustExit() branch releases gil->mutex first (Python/ceval_gil.c:346) and only then clears the drop request it set earlier on the GIL holder (:354). Between these two lines the holder can go through drop_gil(): it still sees _PY_GIL_DROP_REQUEST_BIT set (:260, read without the mutex) and waits on gil->switch_cond with no timeout (:271). The only thread that could take the GIL and signal switch_cond is the daemon thread, and it goes straight to _PyThreadState_HangThread(). The process never exits.

(Line numbers are from main at a5a4659.)

Sequence:

1. The main thread holds the GIL in C code for longer than the switch interval. The daemon thread times out in take_gil() and sets the drop request bit on the main thread.
2. Finalization marks the daemon thread as shutting down (_PyThreadState_SetShuttingDown()).
3. The main thread keeps holding the GIL (shutdown GC). The daemon thread times out again and enters the _PyThreadState_MustExit() branch: MUTEX_UNLOCK(gil->mutex) at :346.
4. Before the daemon thread reaches :354, the main thread runs a __del__ from the shutdown GC, handles the eval breaker and calls drop_gil(). The bit is still set, so it waits on switch_cond.
5. The daemon thread clears the bit (too late) and hangs. Nobody signals switch_cond.

This needs exactly one thread waiting for the GIL. If another thread were waiting, it would take the GIL and signal switch_cond.

Reproducer

gc_bug.py:

import atexit
import threading
import time


class Finalizer:
    def __del__(self):
        pass


def spin():
    while True:
        time.sleep(0)


threading.Thread(target=spin, daemon=True).start()
time.sleep(0.1)

# Many tracked objects make the shutdown gc.collect() hold the GIL for a while.
junk = [[] for _ in range(10**6)]

# A cycle with __del__: shutdown GC runs Python code, which releases the GIL.
f = Finalizer()
f.cycle = f
del f

# Hold the GIL in C, with no eval-breaker checks, right before finalization.
atexit.register(sum, range(10**7))

The race window is a few instructions wide, so a plain ./python gc_bug.py exits normally almost every time. To make it deterministic, stop the daemon thread inside the window with a debugger and let only the main thread run:

$ PYTHON_JIT=0 lldb ./python
(lldb) breakpoint set -f ceval_gil.c -l 353 -c drop_requested
(lldb) process launch -- gc_bug.py
(lldb) thread continue 1

The process never exits. Main thread backtrace after interrupting (PYTHON_JIT=0 only keeps the debugger from stopping on JIT code registration):

pthread_cond_wait
drop_gil                Python/ceval_gil.c:271
_Py_HandlePending       Python/ceval_gil.c:1418
check_periodics
_PyEval_EvalFrameDefault
...
call_unbound_noarg      (Finalizer.__del__)
finalize_garbage
PyGC_Collect
_Py_Finalize
Py_RunMain

If the breakpoint is set at :357 instead (after the bit is cleared), the process exits normally every time. So the hang comes from clearing the bit after releasing the mutex.

Build

CC=clang-21 CXX=clang++-21 LDFLAGS='-fuse-ld=lld-21' ./configure --enable-experimental-jit=yes
make

./python -VV: Python 3.16.0a0 (heads/main:a5a4659548e) [Clang 21.1.8]

CPython versions tested on:

CPython main branch, 3.16

Operating systems tested on:

Linux

Linked PRs

• gh-158090: Fix hang at shutdown when a daemon thread resets the GIL drop request #158101