Repository navigation
[sanitizer] Stress workloads / fq http_api ThreadSanitizer / data race in free at Python interpreter shutdown (release-tsan) #50414
Description
Summary
ThreadSanitizer data race in free during Python interpreter finalization (_Py_Finalize / free_threadstate) racing with drop_gil in a worker thread, in the py3-packaged test/stress binaries (release-tsan).
Triage
- Layer:
test_infra— both racing stacks are entirely inside the embedded CPython runtime of the py3-packaged test/stress binaries: the finalizer (_Py_Finalize->_PyThreadState_DeleteList->free_threadstate->free) on the main thread vsdrop_gil/_PyEval_ReleaseLockin a worker thread (contrib/tools/python3,library/python/runtime_py3/main/main.c). No ydbd / ydb/core / ydb/library product frame appears. - Production impact:
no— the race is entirely inside the CPython interpreter at shutdown of the test/stress binaries, not in ydbd or any product library. - Config / feature flag:
no— not config-related. - Introduced by:
unknown— CPython runtime teardown path, not a specific YDB change. - Seen on previous releases:
previous_releases— same CPython free race reported earlier as [sanitizer] Stress workloads / fq http_api ThreadSanitizer / data race in free at Python interpreter shutdown (release-tsan) #49949 (stable-26-2-1) and [sanitizer] Stress statistics_workload ThreadSanitizer / data race in free at Python interpreter shutdown (release-tsan) #49774 (stable-26-3-1), both closed as duplicate/migrated; not_planned ancestor TSAN: Data race in free in CPython runtime — stress workload binaries #40952.
Environment
- Branch: 26.2.1.ent.2
- Commit: 616359c
- Build: release-tsan
Reports
- HTML report (try_1): https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/Regression-run_Small_and_Medium/32136006575/ya-26-2-1-ent-2-x86-64-tsan/try_1/ya-test.html
- Artifacts index: https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/Regression-run_Small_and_Medium/32136006575/ya-26-2-1-ent-2-x86-64-tsan/index.html
- Workflow run: https://github.com/ydb-platform/ydb/actions/runs/32136006575
- kafka log (try_1): https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/Regression-run_Small_and_Medium/32136006575/ya-26-2-1-ent-2-x86-64-tsan/try_1/artifacts/logs/ydb/tests/stress/kafka/tests/test-results/py3test/testing_out_stuff/test_kafka_streams.py.TestYdbTopicWorkload.test.log
- fq/http_api stderr (try_1): https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/Regression-run_Small_and_Medium/32136006575/ya-26-2-1-ent-2-x86-64-tsan/try_1/artifacts/logs/ydb/tests/fq/http_api/test-results/py3test/testing_out_stuff/stderr
- streaming log (try_1): https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/Regression-run_Small_and_Medium/32136006575/ya-26-2-1-ent-2-x86-64-tsan/try_1/artifacts/logs/ydb/tests/stress/streaming/tests/test-results/py3test/testing_out_stuff/test_workload.py.TestYdbWorkload.test.log
- transfer log (try_1): https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/Regression-run_Small_and_Medium/32136006575/ya-26-2-1-ent-2-x86-64-tsan/try_1/artifacts/logs/ydb/tests/stress/transfer/tests/test-results/py3test/testing_out_stuff/test_workload.py.TestYdbWorkload.test.column-local.log
Affected tests
- ydb/tests/stress/kafka/tests/test_kafka_streams.py.TestYdbTopicWorkload.test (try_1, try_2, try_3)
- ydb/tests/stress/streaming/tests/test_workload.py.TestYdbWorkload.test (try_1, try_2, try_3)
- ydb/tests/stress/transfer/tests/test_workload.py.TestYdbWorkload.test[column-local] (try_1, try_2, try_3)
- ydb/tests/stress/transfer/tests/test_workload.py.TestYdbWorkload.test[column-remote] (try_1, try_2, try_3)
- ydb/tests/stress/transfer/tests/test_workload.py.TestYdbWorkload.test[row-local] (try_1, try_2, try_3)
- ydb/tests/fq/http_api/py3test (try_1, try_2, try_3; reported 2 warnings)
Error
Type: ThreadSanitizer data race in free during CPython interpreter finalization (drop_gil vs free_threadstate). The signature is identical across all affected py3 binaries (kafka_streams_test / streaming / transfer / ydb-tests-fq-http_api); the kafka_streams_test occurrence is shown verbatim below.
WARNING: ThreadSanitizer: data race (pid=1247856)
Write of size 8 at 0x724800028818 by main thread:
#0 free /-S/contrib/libs/clang20-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:741:3 (kafka_streams_test+0x51cf4aa) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#1 _PyMem_RawFree /-S/contrib/tools/python3/Objects/obmalloc.c:90:5 (kafka_streams_test+0x5679b79) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#2 PyMem_RawFree /-S/contrib/tools/python3/Objects/obmalloc.c:1005:5 (kafka_streams_test+0x567b748) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#3 free_threadstate /-S/contrib/tools/python3/Python/pystate.c:1476:9 (kafka_streams_test+0x5ce78b2) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#4 _PyThreadState_DeleteList /-S/contrib/tools/python3/Python/pystate.c:1940:9 (kafka_streams_test+0x5ce78b2)
#5 _Py_Finalize /-S/contrib/tools/python3/Python/pylifecycle.c:2092:5 (kafka_streams_test+0x5cde713) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#6 Py_FinalizeEx /-S/contrib/tools/python3/Python/pylifecycle.c:2261:12 (kafka_streams_test+0x5cde583) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#7 pymain /-S/library/python/runtime_py3/main/main.c:232:9 (kafka_streams_test+0x77c738e) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#8 main /-S/library/python/runtime_py3/main/main.c:251:12 (kafka_streams_test+0x77c743d) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
Previous atomic read of size 8 at 0x724800028818 by thread T59:
#0 _Py_atomic_load_uintptr_relaxed /-S/contrib/tools/python3/Include/cpython/pyatomic_gcc.h:347:10 (kafka_streams_test+0x5758d5c) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#1 _Py_eval_breaker_bit_is_set /-S/contrib/tools/python3/Include/internal/pycore_ceval.h:292:19 (kafka_streams_test+0x5758d5c)
#2 drop_gil /-S/contrib/tools/python3/Python/ceval_gil.c:266:9 (kafka_streams_test+0x5758d5c)
#3 _PyEval_ReleaseLock /-S/contrib/tools/python3/Python/ceval_gil.c:603:5 (kafka_streams_test+0x5758f17) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#4 detach_thread /-S/contrib/tools/python3/Python/pystate.c:2151:5 (kafka_streams_test+0x5ce5cd9) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#5 _PyThreadState_Detach /-S/contrib/tools/python3/Python/pystate.c:2157:5 (kafka_streams_test+0x5ce5cd9)
#6 PyEval_SaveThread /-S/contrib/tools/python3/Python/ceval_gil.c:647:5 (kafka_streams_test+0x5759081) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#7 select_epoll_poll_impl /-S/contrib/tools/python3/Modules/selectmodule.c:1628:9 (kafka_streams_test+0x5cb14e0) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#8 select_epoll_poll /-S/contrib/tools/python3/Modules/clinic/selectmodule.c.h:1007:20 (kafka_streams_test+0x5cb14e0)
#9 _PyEval_EvalFrameDefault /-S/contrib/tools/python3/Python/generated_cases.c.h:1701:23 (kafka_streams_test+0x5749a2c) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#10 _PyEval_EvalFrame /-S/contrib/tools/python3/Include/internal/pycore_ceval.h:120:16 (kafka_streams_test+0x574490a) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#11 _PyEval_Vector /-S/contrib/tools/python3/Python/ceval.c:1820:12 (kafka_streams_test+0x574490a)
#12 _PyFunction_Vectorcall /-S/contrib/tools/python3/Objects/call.c (kafka_streams_test+0x55e590c) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#13 _PyObject_VectorcallTstate /-S/contrib/tools/python3/Include/internal/pycore_call.h:168:11 (kafka_streams_test+0x565d897) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#14 method_vectorcall /-S/contrib/tools/python3/Objects/classobject.c:71:20 (kafka_streams_test+0x565d897)
#15 _PyVectorcall_Call /-S/contrib/tools/python3/Objects/call.c:273:16 (kafka_streams_test+0x55e5443) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#16 _PyObject_Call /-S/contrib/tools/python3/Objects/call.c:348:16 (kafka_streams_test+0x55e563d) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#17 PyObject_Call /-S/contrib/tools/python3/Objects/call.c:373:12 (kafka_streams_test+0x55e5704) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#18 thread_run /-S/contrib/tools/python3/Modules/_threadmodule.c:343:21 (kafka_streams_test+0x5c0f443) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#19 pythread_wrapper /-S/contrib/tools/python3/Python/thread_pthread.h:242:5 (kafka_streams_test+0x5d01afa) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
Thread T59 (tid=1323193, running) created by main thread at:
#0 pthread_create /-S/contrib/libs/clang20-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1041:3 (kafka_streams_test+0x51d08aa) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#1 do_start_joinable_thread /-S/contrib/tools/python3/Python/thread_pthread.h:289:14 (kafka_streams_test+0x5d00ba3) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#2 PyThread_start_joinable_thread /-S/contrib/tools/python3/Python/thread_pthread.h:331:9 (kafka_streams_test+0x5d00a3a) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#3 ThreadHandle_start /-S/contrib/tools/python3/Modules/_threadmodule.c:428:9 (kafka_streams_test+0x5c0f07a) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#4 do_start_new_thread /-S/contrib/tools/python3/Modules/_threadmodule.c:1857:9 (kafka_streams_test+0x5c0f07a)
#5 thread_PyThread_start_joinable_thread /-S/contrib/tools/python3/Modules/_threadmodule.c:1980:14 (kafka_streams_test+0x5c0e5de) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#6 cfunction_call /-S/contrib/tools/python3/Objects/methodobject.c:539:18 (kafka_streams_test+0x564980d) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#7 _PyObject_MakeTpCall /-S/contrib/tools/python3/Objects/call.c:242:18 (kafka_streams_test+0x55e4cb6) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#8 _PyObject_VectorcallTstate /-S/contrib/tools/python3/Include/internal/pycore_call.h:166:16 (kafka_streams_test+0x55e558c) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#9 PyObject_Vectorcall /-S/contrib/tools/python3/Objects/call.c:327:12 (kafka_streams_test+0x55e558c)
#10 _PyEval_EvalFrameDefault /-S/contrib/tools/python3/Python/generated_cases.c.h:1509:19 (kafka_streams_test+0x5749241) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#11 _PyEval_EvalFrame /-S/contrib/tools/python3/Include/internal/pycore_ceval.h:120:16 (kafka_streams_test+0x574490a) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#12 _PyEval_Vector /-S/contrib/tools/python3/Python/ceval.c:1820:12 (kafka_streams_test+0x574490a)
#13 _PyFunction_Vectorcall /-S/contrib/tools/python3/Objects/call.c (kafka_streams_test+0x55e590c) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#14 _PyObject_VectorcallDictTstate /-S/contrib/tools/python3/Objects/call.c:146:15 (kafka_streams_test+0x55e4ac6) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#15 _PyObject_Call_Prepend /-S/contrib/tools/python3/Objects/call.c:504:24 (kafka_streams_test+0x55e5c27) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#16 slot_tp_init /-S/contrib/tools/python3/Objects/typeobject.c:9838:15 (kafka_streams_test+0x56a2f7c) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#17 type_call /-S/contrib/tools/python3/Objects/typeobject.c:2008:19 (kafka_streams_test+0x569a5b3) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#18 _PyObject_MakeTpCall /-S/contrib/tools/python3/Objects/call.c:242:18 (kafka_streams_test+0x55e4cb6) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#19 _PyObject_VectorcallTstate /-S/contrib/tools/python3/Include/internal/pycore_call.h:166:16 (kafka_streams_test+0x55e558c) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#20 PyObject_Vectorcall /-S/contrib/tools/python3/Objects/call.c:327:12 (kafka_streams_test+0x55e558c)
#21 _PyEval_EvalFrameDefault /-S/contrib/tools/python3/Python/generated_cases.c.h:1509:19 (kafka_streams_test+0x5749241) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#22 _PyEval_EvalFrame /-S/contrib/tools/python3/Include/internal/pycore_ceval.h:120:16 (kafka_streams_test+0x574477d) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#23 _PyEval_Vector /-S/contrib/tools/python3/Python/ceval.c:1820:12 (kafka_streams_test+0x574477d)
#24 PyEval_EvalCode /-S/contrib/tools/python3/Python/ceval.c:604:21 (kafka_streams_test+0x574477d)
#25 builtin_exec_impl /-S/contrib/tools/python3/Python/bltinmodule.c:1142:17 (kafka_streams_test+0x570fc3a) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#26 builtin_exec /-S/contrib/tools/python3/Python/clinic/bltinmodule.c.h:559:20 (kafka_streams_test+0x570fc3a)
#27 cfunction_vectorcall_FASTCALL_KEYWORDS /-S/contrib/tools/python3/Objects/methodobject.c:440:24 (kafka_streams_test+0x5648ff6) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#28 _PyObject_VectorcallTstate /-S/contrib/tools/python3/Include/internal/pycore_call.h:168:11 (kafka_streams_test+0x55e5515) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#29 PyObject_Vectorcall /-S/contrib/tools/python3/Objects/call.c:327:12 (kafka_streams_test+0x55e5515)
#30 _PyEval_EvalFrameDefault /-S/contrib/tools/python3/Python/generated_cases.c.h:813:23 (kafka_streams_test+0x57474c8) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#31 _PyEval_EvalFrame /-S/contrib/tools/python3/Include/internal/pycore_ceval.h:120:16 (kafka_streams_test+0x574490a) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#32 _PyEval_Vector /-S/contrib/tools/python3/Python/ceval.c:1820:12 (kafka_streams_test+0x574490a)
#33 _PyFunction_Vectorcall /-S/contrib/tools/python3/Objects/call.c (kafka_streams_test+0x55e590c) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#34 _PyVectorcall_Call /-S/contrib/tools/python3/Objects/call.c:273:16 (kafka_streams_test+0x55e5443) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#35 _PyObject_Call /-S/contrib/tools/python3/Objects/call.c:348:16 (kafka_streams_test+0x55e563d) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#36 PyObject_Call /-S/contrib/tools/python3/Objects/call.c:373:12 (kafka_streams_test+0x55e5704) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#37 RunModule /-S/library/python/runtime_py3/main/main.c:62:14 (kafka_streams_test+0x77c6f28) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
#38 pymain /-S/library/python/runtime_py3/main/main.c:212:15 (kafka_streams_test+0x77c6f28)
#39 main /-S/library/python/runtime_py3/main/main.c:251:12 (kafka_streams_test+0x77c743d) (BuildId: d7406fbde644780426ec024e63a5a8c36ba84722)
SUMMARY: ThreadSanitizer: data race /-S/contrib/libs/clang20-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:741:3 in free
Activity
ydbot commentedon Aug 18, 2026
Further work and discussion for this issue continue in: YDBBUGS-712
robot-vibe-db commentedon Aug 18, 2026
Analysis
Root Cause
This is a known upstream CPython bug (python/cpython#124878). During interpreter finalization (
_Py_Finalize), the main thread tears down thread states via_PyThreadState_DeleteList→free_threadstate→PyMem_RawFree→free, while daemon/worker threads (here: epoll-polling threads inselect_epoll_poll_impl) are still running and accessing theirPyThreadStatethroughdrop_gil→_Py_eval_breaker_bit_is_set. The result is a use-after-free race on thePyThreadStatememory at address0x724800028818.The race signature is identical across all affected binaries (
kafka_streams_test,streaming,transfer,ydb-tests-fq-http_api) — all are py3-packaged test binaries using the embedded CPython 3.13.15 runtime (contrib/tools/python3). No YDB product frames (ydb/core,ydb/library) appear in any stack.Relationship to Prior Issues
This is a recurrence of the exact same race tracked in:
not_planned) — samefreevsdrop_gilCPython teardown race onstable-26-2-1/stable-26-3-126.2.1.1226.3.1.8Existing Suppression
The bundled CPython at
contrib/tools/python3/tsan.suppalready contains:and the
ya.makedeclaresSUPPRESSIONS(lsan.supp tsan.supp). This suppression should match the main-thread stack (frame #4 is_PyThreadState_DeleteList). The fact that TSan still reports the race on26.2.1.ent.2suggests the suppression file may not be propagating to the test binaries on this branch, orTSAN_OPTIONSis being overridden.Upstream Fix Status
The definitive fix was merged upstream in python/cpython#130649 (March 2025). It introduces a reference count on
PyThreadState(initialized to 2: one for the interpreter list, one for the OS thread) so thatfree_threadstateonly frees memory when both the interpreter and the OS thread are done with it. However, this fix targets CPython 3.14+ and was not backported to the 3.13 branch. For 3.13, only a TSan suppression backport was applied upstream (python/cpython#130687).The bundled CPython 3.13.15 in this repo does NOT contain the refcount-based fix (no
_refcountfield or_Py_THREAD_SHUTTING_DOWNstate inpystate.c).Criticality: LOW
ydbdor product library code is involved.release-tsanregression CI.Possible Solutions
Verify/fix suppression propagation (recommended, lowest effort): Investigate why the existing
race:_PyThreadState_DeleteListsuppression incontrib/tools/python3/tsan.suppis not taking effect on the26.2.1.ent.2branch. Possible causes: the test binaries' dependency chain doesn't inherit theSUPPRESSIONSfrom the Python library'sya.make, orTSAN_OPTIONSis being overridden by the test runner. Adding explicitSUPPRESSIONSdirectives to the affected testya.makefiles or tolibrary/python/runtime_py3would be a quick fix.Broaden the suppression pattern: Add
race:free_threadstatetocontrib/tools/python3/tsan.supp. This targets the exact function performing the racy write and would catch the race even if the inlined call chain prevents_PyThreadState_DeleteListfrom appearing in TSan's frame matching.Cherry-pick the upstream CPython fix: Port python/cpython#130649 into the bundled
contrib/tools/python3. This would fully eliminate the race by adding reference counting toPyThreadState. This is the most correct solution but carries risk since it modifies the embedded CPython runtime.Ensure graceful thread shutdown in the test harness: Modify
library/python/runtime_py3/main/main.cto explicitly request all non-daemon Python threads to stop (e.g., viathreading._shutdown()or setting a shutdown flag) and join them before callingPy_FinalizeEx(). This would prevent the race from occurring in the first place, but may require changes to the test workloads themselves if they spawn threads that don't check for shutdown.Recommendation
Given the low criticality and recurring nature, the most pragmatic path is option 1 or 2: ensure the existing TSan suppression reaches the test binaries (or add
race:free_threadstateas an additional suppression). This eliminates the CI noise without touching the CPython runtime. For a long-term fix, option 3 (cherry-picking the upstream refcount fix) would be ideal when the bundled CPython is next updated.robot-vibe-db commentedon Aug 18, 2026
Full analysis log
Analysis performed by claude, claude-opus-4-6.