Repository navigation
Segfault in faulthandler signal handler with threads #116008
Description
Activity
Thinking about it some more, I suspect that this is a race condition where a thread is executing Python code while the faulthandler signal handler code is running. The main thread handling the signal is looking at the stack frames of a thread while that thread is actively executing. The stack frames are being changed as the traceback executes.
I've just been investigating a similar issue myself - here's a short reproducer:
import faulthandler import signal faulthandler.enable() faulthandler.register(signal.SIGUSR1) import threading while True: threading.Thread(target=lambda: None).start()Whilst the above is running, spam it with SIGUSR1 calls, eg:
while [ -d . ]; do killall -USR1 python3; doneand it will instantly segfault (have tried Python 3.10.12, 3.11.8 and 3.12.2).
It seems to me that faulthandler is flawed by design - it doesn't hold the GIL, so state may get changed (eg, stack frames or thread states freed) whilst it is producing the the traceback.
A workaround is to avoid using
faulthandler.register(...)and instead catch the signal yourself and callfaulthandler.dump_traceback, eg:import signal import traceback import faulthandler def handler(signum, frame): faulthandler.dump_traceback(all_threads=True) signal.signal(signal.SIGUSR1, handler) import threading while True: threading.Thread(target=lambda: None).start()Then the GIL will be held when the traceback is output, and it no longer crashes.
The crash I am seeing does not involve rapid starting and stopping of threads. In my case, I have a relatively small number of threads that are busy making sequences of nested calls, meaning that the call stack is changing quite rapidly.
I'm guessing that the issue is caused by a thread exiting whilst the traceback is being generated - my silly reproducer obviously does that a lot, but your traceback segfaults at the
set_resultinconcurrent.futures, which might have been shortly followed by the thread exiting (eg, if the executor goes out of scope and gets collected) - does that happen in your code?I was able to crash Python 3.10 with a single additional thread running a while loop, but 3.11+ have so far required thread creation/destruction thrash to trigger a segfault (with the faulthandler+SIGUSR1 combo).
I get very similar backtraces to you in gdb (using the above reproducer on Python 3.11.8):
First crash:
Thread 1 "python" received signal SIGSEGV, Segmentation fault. dump_traceback (fd=2, tstate=<optimized out>, write_header=<optimized out>) at Python/traceback.c:1340 1340 frame = frame->previous; (gdb) bt #0 dump_traceback (fd=2, tstate=<optimized out>, write_header=<optimized out>) at Python/traceback.c:1340 #1 0x00007ffff7abc753 in _Py_DumpTracebackThreads (fd=fd@entry=2, interp=<optimized out>, interp@entry=0x0, current_tstate=0x7ffff7d8c278 <_PyRuntime+166328>) at Python/traceback.c:1446 #2 0x00007ffff7acebf5 in faulthandler_dump_traceback (interp=<optimized out>, all_threads=<optimized out>, fd=<optimized out>) at ./Modules/faulthandler.c:251 #3 faulthandler_dump_traceback (interp=<optimized out>, all_threads=<optimized out>, fd=<optimized out>) at ./Modules/faulthandler.c:230 #4 faulthandler_user (signum=10) at ./Modules/faulthandler.c:860 #5 <signal handler called> #6 0x00007ffff7491115 in __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555584290) at ./nptl/futex-internal.c:57 ...snipped... (gdb) p frame $1 = (_PyInterpreterFrame *) 0x7ffff7fa8090 (gdb) p frame->previous Cannot access memory at address 0x7ffff7fa80c0Second crash:
Thread 1 "python" received signal SIGSEGV, Segmentation fault. PyCode_Addr2Line (co=0x0, addrq=-184) at Objects/codeobject.c:769 769 return co->co_firstlineno; (gdb) bt #0 PyCode_Addr2Line (co=0x0, addrq=-184) at Objects/codeobject.c:769 #1 0x00007ffff7a7c8dd in _PyInterpreterFrame_GetLine (frame=frame@entry=0x7ffff7fa8190) at Python/frame.c:169 #2 0x00007ffff7abc4e7 in dump_frame (frame=0x7ffff7fa8190, fd=2) at Python/traceback.c:1274 #3 dump_traceback (fd=2, tstate=<optimized out>, write_header=<optimized out>) at Python/traceback.c:1339 #4 0x00007ffff7abc753 in _Py_DumpTracebackThreads (fd=fd@entry=2, interp=<optimized out>, interp@entry=0x0, current_tstate=0x7ffff7d8c278 <_PyRuntime+166328>) at Python/traceback.c:1446 #5 0x00007ffff7acebf5 in faulthandler_dump_traceback (interp=<optimized out>, all_threads=<optimized out>, fd=<optimized out>) at ./Modules/faulthandler.c:251 #6 faulthandler_dump_traceback (interp=<optimized out>, all_threads=<optimized out>, fd=<optimized out>) at ./Modules/faulthandler.c:230 #7 faulthandler_user (signum=10) at ./Modules/faulthandler.c:860 #8 <signal handler called> #9 0x00007ffff7491115 in __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555565820) at ./nptl/futex-internal.c:57 ...snipped... (gdb) p co $1 = (PyCodeObject *) 0x0The threads in my system are definitely not exiting. They are all long-lived.
Here's a reproducer that only needs one long-lived thread:
import faulthandler import signal faulthandler.register(signal.SIGUSR1) import threading import time def rec(x): a = 1 b = 2 c = 3 if x>0: return rec(x-1) def loop(): while True: rec(123) threading.Thread(target=loop).start() time.sleep(1000)
Again, run this whilst spamming it with USR1s from another terminal with:
while [ -d . ]; do killall -USR1 python3; doneIt crashes instantly on Python 3.11.8 and 3.12.2 (Linux, x86_64 on Ryzen 6800U).
It needs those three assignments in the loop, and sufficiently deep recursion, to trigger the bug (maybe this is pushing the stack frame over some size which changes the allocation/free behaviour, or bypassing some optimisations).
#110052 is possibly the same issue - although as you note (and the above reproducer indicates) the issue in this case seems to be races involving stack frame manipulation rather than thread creation/destruction.
It seems to me that faulthandler is flawed by design - it doesn't hold the GIL, so state may get changed (eg, stack frames or thread states freed) whilst it is producing the the traceback.
Correct, it's not thread-safe by design. faulthandler "fault handler" is only a "best effort" debugging tool, trying to provide some information about a crash. Sometimes it works, sometimes it crashs since Python internals are no longer consistent.
Maybe for non-fatal signals such as SIGUSR1, it can try to acquire/release the GIL, but I'm not sure if it's doable.
Correct, it's not thread-safe by design. faulthandler "fault handler" is only a "best effort" debugging tool, trying to provide some information about a crash. Sometimes it works, sometimes it crashs since Python internals are no longer consistent.
In that case, I think there should be a prominent warning in the documentation for faulthandler.register, that it must not be used in multi-threaded code.
Maybe for non-fatal signals such as SIGUSR1, it can try to acquire/release the GIL, but I'm not sure if it's doable.
That would seem like a sensible approach if it is possible.
Maybe for non-fatal signals such as SIGUSR1, it can try to acquire/release the GIL, but I'm not sure if it's doable.
That would seem like a sensible approach if it is possible.
While as far as I understand
PyGILState_Ensure()is not exactly signal-safe, acquiring the GIL inall_threadtraceback dump infaulthandler_dump_traceback()for non-fatal async signals does work and fixes jonny5532's single thread reproducer above. How much of a practical difference is there in callingPyGILState_Ensure()from a random thread without a py threadstate at a random time vs. calling it from an async signal handler (apart from the possibly separate stack)?diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c index ba7970d6656..67853c0a2e5 100644 --- a/Modules/faulthandler.c +++ b/Modules/faulthandler.c @@ -177,7 +177,8 @@ get_thread_state(void) static void faulthandler_dump_traceback(int fd, int all_threads, - PyInterpreterState *interp) + PyInterpreterState *interp, + int async) { static volatile int reentrant = 0; @@ -197,7 +198,16 @@ faulthandler_dump_traceback(int fd, int all_threads, PyThreadState *tstate = PyGILState_GetThisThreadState(); if (all_threads == 1) { + PyGILState_STATE gilstate; + if (async) { + gilstate = PyGILState_Ensure(); + } + (void)_Py_DumpTracebackThreads(fd, NULL, tstate); + + if (async) { + PyGILState_Release(gilstate); + } } else { if (all_threads == FT_IGNORE_ALL_THREADS) { @@ -349,7 +359,7 @@ faulthandler_fatal_error(int signum) } faulthandler_dump_traceback(fd, deduce_all_threads(), - fatal_error.interp); + fatal_error.interp, 0); _Py_DumpExtensionModules(fd, fatal_error.interp); @@ -424,7 +434,7 @@ faulthandler_exc_handler(struct _EXCEPTION_POINTERS *exc_info) } faulthandler_dump_traceback(fd, deduce_all_threads(), - fatal_error.interp); + fatal_error.interp, 0); /* call the next exception handler */ return EXCEPTION_CONTINUE_SEARCH; @@ -841,7 +851,7 @@ faulthandler_user(int signum) if (!user->enabled) return; - faulthandler_dump_traceback(user->fd, user->all_threads, user->interp); + faulthandler_dump_traceback(user->fd, user->all_threads, user->interp, 1); #ifdef HAVE_SIGACTION if (user->chain) {
- changed the title
[-]Segfault in faulthandler signal handler[/-][+]Segfault in faulthandler signal handler with threads[/+]on Mar 28, 2025 @vstinner Should we close this after #140921 or try to implement "Maybe for non-fatal signals such as SIGUSR1, it can try to acquire/release the GIL, but I'm not sure if it's doable." (#116008 (comment))?
3 remaining items
before a segfault
It is about non-fatal signals. Segfaults will be handled as before. If I get you right. IIUC
signalmodule redirects signals to main thread and handlers will run in particular points.Wouldn't that defeat the point of faulthandler? If we do it like that, the traceback would be dumped at the point where the signal was handled (in the main thread on a bytecode boundary, like you said), not where it was received, so it wouldn't provide any useful debugging information.
Reacted by Sergey MiryanovOugh! It is really good point! Thanks!
Afterward, I believe we have to update docs.
- added a commit that references this issue
on Nov 26, 2025 I wrote PR #141988 which makes the reproducer #116008 (comment) way more reliable. My change introduces more checks to detect if a Python thread state has been freed. Example:
Thread 0x00007ffac26d36c0 (most recent call first): <freed thread state> <freed thread state>
Sadly, it's still quick to crash faulthandler with the reproducer #116008 (comment).
At least, faulthandler is able to detect freed/invalid frames in more cases since commit a84181c. Example:
Thread 0x00007feb029ff6c0 [Thread-1 (loop)] (most recent call first): File "/home/vstinner/python/main/y.py", line 13 in rec <invalid frame>Reacted by Sergey Miryanov- added 3 commits that reference this issue
on Nov 27, 2025 This issue could be closed, right, @vstinner ?
It's still possible to crash Python with both reproducers, but recent changes made faulthandler way more reliable. I consider that it's enough to close the issue.
If you want a reliable code, you can use:
def handler(signum, frame): faulthandler.dump_traceback(all_threads=True) signal.signal(signal.SIGUSR1, handler)
instead of
faulthandler.register(). The behavior is different thanfaulthandler.register():- The "current thread" is different.
- If Python is blocked (ex: deadlock), the traceback may never be displayed.
Reacted by Mikhail Efimov- added a commit that references this issue
on Aug 21, 2026
Crash report
What happened?
We use faulthandler.register to dump stacks on SIGUSR1.
Sending the signal to a running multi-threaded process, it segfaulted. It successfully dumped several thread stacks, but segfaulted part way through:
The last line shows that it wrote "File", then segfaulted. (That was caught by a parent watchdog process that restarted the process. The timestamp and rest of the message is from the watchdog process.)
Looking at the core file:
co_filename is clearly invalid here.
Most of the time dumping the stack works fine, but doing it repeatedly I can reproduce this when the process is busy. A second time, it failed in a slightly different place, accessing co_name:
CPython versions tested on:
3.11
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.11.4 (main, Jun 19 2023, 17:32:00) [GCC 11.3.1 20221121 (Red Hat 11.3.1-4.3.0.1)]
Linked PRs