ThreadSanitizer: data race in interpreter_clear() vs take_gil() during finalization with daemon threads

Bug report

Bug description:

ThreadSanitizer detects a data race between the main thread clearing the interpreter during finalization and daemon threads attempting to acquire the GIL. The race occurs on _PyRuntime.ceval.eval_breaker at address that is accessed by both:

Main thread (write): interpreter_clear() → writes to eval_breaker
Daemon thread T7 (atomic write): take_gil() → _Py_unset_eval_breaker_bit() → atomic operation on the same address

This appears to be related to but distinct from #124878, as the race involves interpreter_clear rather than free_threadstate, and occurs during take_gil rather than drop_gil.

How to Reproduce

Build configuration:

CC=clang CXX=clang++ ./configure --with-thread-sanitizer --with-pydebug --enable-experimental-jit=yes --disable-optimizations --with-lto=full

make -j$(nproc)

Run command:

./python -X dev -X showrefcount bug.py

Test script (bug.py):

import threading
from functools import lru_cache

@lru_cache(None)
def fib(n):
    """Function that JIT will compile"""
    if n < 2:
        return n
    return fib(n-1) + fib(n-2)

def hammer_jit():
    """Call the function many times to force JIT compilation"""
    for i in range(1_000_000):
        fib(i)

# Create a race: all threads simultaneously force JIT compilation
threads = [threading.Thread(target=hammer_jit) for _ in range(16)]
for t in threads: t.start()
for t in threads: t.join()

TSAN Output

Details

➜  main git:(main) ✗ ./python -X dev -X showrefcount bug.py
^C^CException ignored in the internal traceback machinery:
^CTraceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1379, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1341, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1267, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1304, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1278, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1410, in find_spec
  File "<frozen importlib._bootstrap_external>", line 166, in _path_isfile
  File "<frozen importlib._bootstrap_external>", line 158, in _path_is_mode_type
  File "<frozen importlib._bootstrap_external>", line 152, in _path_stat
KeyboardInterrupt:
Traceback (most recent call last):
  File "/home/shamil/oss/cpython/main/bug.py", line 18, in <module>
    for t in threads: t.start()
  File "/home/shamil/oss/cpython/main/Lib/threading.py", line 1126, in join
^Cobject address  : 0x7ffff5d41c50
object refcount : 3
object type     : 0x555556087868
object type name: KeyboardInterrupt
object repr     : KeyboardInterrupt()
lost sys.stderr
^CException ignored on threading shutdown:
Traceback (most recent call last):
  File "/home/shamil/oss/cpython/main/Lib/threading.py", line 1577, in _shutdown
    _thread_shutdown()
    ~~~~~~~~~~~~~~~~^^
KeyboardInterrupt
==================
WARNING: ThreadSanitizer: data race (pid=22017)
  Write of size 8 at 0x55555611abc8 by main thread:
    #0 interpreter_clear /home/shamil/oss/cpython/main/Python/pystate.c:769:26 (python+0x6cd57a) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #1 _PyInterpreterState_Clear /home/shamil/oss/cpython/main/Python/pystate.c:904:5 (python+0x699f80) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #2 finalize_interp_clear /home/shamil/oss/cpython/main/Python/pylifecycle.c:1920:5 (python+0x699f80)
    #3 _Py_Finalize /home/shamil/oss/cpython/main/Python/pylifecycle.c:2337:5 (python+0x6965a1) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #4 _Py_Finalize /home/shamil/oss/cpython/main/Python/pylifecycle.c:2337:5 (python+0x6965a1) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #5 Py_FinalizeEx /home/shamil/oss/cpython/main/Python/pylifecycle.c:2378:12 (python+0x72569b) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #6 Py_RunMain /home/shamil/oss/cpython/main/Modules/main.c:774:9 (python+0x72569b)
    #7 pymain_main /home/shamil/oss/cpython/main/Modules/main.c:802:12 (python+0x728d18) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #8 pymain_main /home/shamil/oss/cpython/main/Modules/main.c:802:12 (python+0x728d18) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #9 Py_BytesMain /home/shamil/oss/cpython/main/Modules/main.c:826:12 (python+0x180145) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #10 main /home/shamil/oss/cpython/main/./Programs/python.c:15:12 (python+0x180145)
    #11 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x2a574) (BuildId: bc071219b774e887d2477b96fa792af74144ff37)            #12 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x2a574) (BuildId: bc071219b774e887d2477b96fa792af74144ff37)

  Previous atomic write of size 8 at 0x55555611abc8 by thread T7:
    #0 _Py_atomic_and_uintptr /home/shamil/oss/cpython/main/./Include/cpython/pyatomic_gcc.h:218:10 (python+0x5c6cbe) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #1 _Py_unset_eval_breaker_bit /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:350:5 (python+0x5c6cbe)
    #2 take_gil /home/shamil/oss/cpython/main/Python/ceval_gil.c:351:21 (python+0x5c6cbe)
    #3 _PyEval_AcquireLock /home/shamil/oss/cpython/main/Python/ceval_gil.c:588:5 (python+0x6d152e) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #4 _PyThreadState_Attach /home/shamil/oss/cpython/main/Python/pystate.c:2100:9 (python+0x6d152e)
    #5 _Py_HandlePending /home/shamil/oss/cpython/main/Python/ceval_gil.c:1411:9 (python+0x5c8754) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)             #6 _Py_HandlePending /home/shamil/oss/cpython/main/Python/ceval_gil.c:1411:9 (python+0x5c8754) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #7 check_periodics /home/shamil/oss/cpython/main/Python/ceval_macros.h:412:16 (python+0x53b92b) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #8 _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/Python/generated_cases.c.h:10296:31 (python+0x53b92b)
    #9 _PyEval_EvalFrame /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:121:16 (python+0x4fa5b7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #10 _PyEval_Vector /home/shamil/oss/cpython/main/Python/ceval.c:2001:12 (python+0x4fa5b7)
    #11 _PyFunction_Vectorcall /home/shamil/oss/cpython/main/Objects/call.c (python+0x285b04) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)                  #12 _PyObject_VectorcallTstate.3425 /home/shamil/oss/cpython/main/./Include/internal/pycore_call.h:169:11 (python+0x28c0f7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #13 method_vectorcall /home/shamil/oss/cpython/main/Objects/classobject.c:73:20 (python+0x28bc07) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)          #14 _PyObject_VectorcallTstate /home/shamil/oss/cpython/main/./Include/internal/pycore_call.h:169:11 (python+0x586544) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)                                                                                                                                                    #15 context_run /home/shamil/oss/cpython/main/Python/context.c:728:29 (python+0x586544)
    #16 _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/Python/generated_cases.c.h:3710:35 (python+0x5139a5) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #17 _PyEval_EvalFrame /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:121:16 (python+0x4fa5b7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #18 _PyEval_Vector /home/shamil/oss/cpython/main/Python/ceval.c:2001:12 (python+0x4fa5b7)
    #19 _PyFunction_Vectorcall /home/shamil/oss/cpython/main/Objects/call.c (python+0x285b04) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #20 _PyObject_VectorcallTstate.3425 /home/shamil/oss/cpython/main/./Include/internal/pycore_call.h:169:11 (python+0x28c0f7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #21 method_vectorcall /home/shamil/oss/cpython/main/Objects/classobject.c:73:20 (python+0x28bc07) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #22 _PyVectorcall_Call /home/shamil/oss/cpython/main/Objects/call.c:273:16 (python+0x2854fd) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #23 _PyObject_Call /home/shamil/oss/cpython/main/Objects/call.c:348:16 (python+0x285764) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #24 PyObject_Call /home/shamil/oss/cpython/main/Objects/call.c:373:12 (python+0x828572) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #25 thread_run /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:387:21 (python+0x828572)
    #26 pythread_wrapper /home/shamil/oss/cpython/main/Python/thread_pthread.h:234:5 (python+0x709130) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #27 pythread_wrapper /home/shamil/oss/cpython/main/Python/thread_pthread.h:234:5 (python+0x709130) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)

  Location is global '_PyRuntime' of size 359232 at 0x5555560c3408 (python+0xbc6bc8)
  Thread T7 'Thread-7 (hamme' (tid=22032, running) created by main thread at:                                                                                      #0 pthread_create <null> (python+0xfc21e) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #1 do_start_joinable_thread /home/shamil/oss/cpython/main/Python/thread_pthread.h:281:14 (python+0x709041) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #2 PyThread_start_joinable_thread /home/shamil/oss/cpython/main/Python/thread_pthread.h:323:9 (python+0x828011) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #3 ThreadHandle_start /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:474:9 (python+0x828011)                                                          #4 do_start_new_thread /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:1920:9 (python+0x828011)
    #5 thread_PyThread_start_joinable_thread /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:2043:14 (python+0x825e58) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #6 thread_PyThread_start_joinable_thread /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:2043:14 (python+0x825e58) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #7 cfunction_call /home/shamil/oss/cpython/main/Objects/methodobject.c:564:18 (python+0x354c1b) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #8 _PyObject_MakeTpCall /home/shamil/oss/cpython/main/Objects/call.c:242:18 (python+0x283b90) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #9 _PyObject_VectorcallTstate /home/shamil/oss/cpython/main/./Include/internal/pycore_call.h:167:16 (python+0x2838b4) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #10 PyObject_Vectorcall /home/shamil/oss/cpython/main/Objects/call.c:327:12 (python+0x51093d) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #11 _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/Python/generated_cases.c.h:3188:35 (python+0x51093d)
    #12 _PyEval_EvalFrame /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:121:16 (python+0x4fa5b7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #13 _PyEval_Vector /home/shamil/oss/cpython/main/Python/ceval.c:2001:12 (python+0x4fa5b7)
    #14 _PyEval_EvalFrame /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:121:16 (python+0x4fa5b7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #15 _PyEval_Vector /home/shamil/oss/cpython/main/Python/ceval.c:2001:12 (python+0x4fa5b7)
    #16 PyEval_EvalCode /home/shamil/oss/cpython/main/Python/ceval.c:884:21 (python+0x4f9e99) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #17 run_eval_code_obj /home/shamil/oss/cpython/main/Python/pythonrun.c:1365:12 (python+0x6d7f60) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #18 run_mod /home/shamil/oss/cpython/main/Python/pythonrun.c:1459:19 (python+0x6d7f60)
    #19 pyrun_file /home/shamil/oss/cpython/main/Python/pythonrun.c:1293:15 (python+0x6d6adb) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #20 _PyRun_SimpleFileObject /home/shamil/oss/cpython/main/Python/pythonrun.c:521:13 (python+0x6d6adb)
    #21 _PyRun_AnyFileObject /home/shamil/oss/cpython/main/Python/pythonrun.c:81:15 (python+0x6d5736) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #22 pymain_run_file_obj /home/shamil/oss/cpython/main/Modules/main.c:410:15 (python+0x727676) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #23 pymain_run_file /home/shamil/oss/cpython/main/Modules/main.c:429:15 (python+0x727676)
    #24 pymain_run_python /home/shamil/oss/cpython/main/Modules/main.c:691:21 (python+0x72634e) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #25 Py_RunMain /home/shamil/oss/cpython/main/Modules/main.c:772:5 (python+0x72634e)
    #26 pymain_main /home/shamil/oss/cpython/main/Modules/main.c:802:12 (python+0x728d18) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #27 Py_BytesMain /home/shamil/oss/cpython/main/Modules/main.c:826:12 (python+0x180145) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #28 main /home/shamil/oss/cpython/main/./Programs/python.c:15:12 (python+0x180145)
    #29 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x2a574) (BuildId: bc071219b774e887d2477b96fa792af74144ff37)
    #30 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x2a574) (BuildId: bc071219b774e887d2477b96fa792af74144ff37)

SUMMARY: ThreadSanitizer: data race /home/shamil/oss/cpython/main/Python/pystate.c:769:26 in interpreter_clear
==================
[4156 refs, 1966 blocks]

Analysis

The race occurs because:

Main thread starts finalization and calls interpreter_clear() which writes to _PyRuntime.ceval.eval_breaker
Meanwhile, daemon threads are still running and attempt to acquire the GIL via take_gil()
take_gil() calls _Py_unset_eval_breaker_bit() which performs atomic operations on the same eval_breaker field
These accesses are not properly synchronized during the finalization phase

Environment

CPython version: main branch (commit: [fbf0843])
Operating System: Linux | Ubuntu 25.10
Compiler: Ubuntu clang version 20.1.8 (0ubuntu4)
Build flags: --with-thread-sanitizer --with-pydebug --enable-experimental-jit=yes

Related Issues

race condition in threading when interpreter finalized while daemon thread runs (thread sanitizer identified) #124878 - Similar race during finalization but with drop_gil and free_threadstate
test__xxsubinterpreters is Occasionally Crashing #104341 - Background on interpreter deletion during finalization

CPython versions tested on:

3.15, CPython main branch

Operating systems tested on:

Linux

Linked PRs

We should just change the assignment at:

cpython/Python/pystate.c

Line 769 in fbf0843

tstate->eval_breaker = 0;

To an atomic operation and update the code comment.

@colesbury

@colesbury Can I start fixing this bug?

Yes, please

gh-140257: fix data race on eval_breaker during finalization (#140265)

pythongh-140257: fix data race on eval_breaker during finalization (p…

[3.14] gh-140257: fix data race on eval_breaker during finalization (G…

pythongh-140257: fix data race on eval_breaker during finalization (p…

Navigation Menu

Repository navigation

ThreadSanitizer: data race in interpreter_clear() vs take_gil() during finalization with daemon threads #140257

Bug report

Bug description:

How to Reproduce

TSAN Output

Analysis

Environment

Related Issues

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ThreadSanitizer: data race in interpreter_clear() vs take_gil() during finalization with daemon threads #140257

Description

Bug report

Bug description:

How to Reproduce

TSAN Output

Analysis

Environment

Related Issues

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

colesbury commented on Oct 17, 2025

ashm-dev commented on Oct 17, 2025

colesbury commented on Oct 17, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions