Repository navigation
Heisenbug that kills process via SIGALRM #133687
Description
Activity
- changed the title
[-]Heisenbug that kills process via SIGALARM[/-][+]Heisenbug that kills process via SIGALRM[/+]on May 8, 2025 I haven't looked at the reproducer yet, but this is understandable if unexpected behavior. Exceptions raised during destructors (i.e.,
__del__methods) get printed as unraisable exceptions and then swallowed:https://docs.python.org/3/library/sys.html#sys.unraisablehook
That's not new behavior and it's pretty much a fundamental limitation of raising exception asynchronously, such as with
signal.signal(). Most likely, the timing of things slightly changed in 3.13 such that you routinely get "unlucky" as to what's executing when the SIGALRM triggers.Interesting, so are you saying that raising an exception is always undefined behavior in Python, always has been?
The thing that makes me quite curious is that I do believe that in the scenario, this does look like it is well-defined behavior. I'm setting
signal.alarm()inside thetryblock, then cancel it before leaving the try block. AnySIGALRMshould call the installed signal handler and the raised exception should be possible to stack-unwind (there is only a single main thread there). In this instance, it seems to be the case that the signal is delivered while outside thetry...exceptblock, which should never be possible.Can you try to elaborate why this would be happening?
I would be fine with the exception being omitted (e.g., during
__del__, as you described) and being silently discarded. But somehow my program is not always able to catch the raised exception, which is indeed a problem.Thanks for taking the time to read through my report! Cheers.
Interesting, so are you saying that raising an exception is always undefined behavior in Python, always has been?
No, exceptions are defined behaviour.There are signals involved here, and those result in asynchronous code execution in C. Integration of signal handlers in the rest of the interpreter is non-trivial.
Anyway:
- I can reproduce the issue on macOS (in about 25 seconds)
- The proces is killed by SIGALRM reliably when using python3.13t (generally on the first iteration of the loop)
- I cannot reproduce the issue with Python 3.14b1 (both regular and free-threading)
First of all, thanks @ronaldoussoren for trying it out and confirming that there is indeed an issue. I agree that signal handling is exceptionally tricky and I'll be the first to admit that stack unwinding in a signal handler is just nasty. That said, if it's not UB, then it's an interpreter bug.
I did some more digging and while
alarm(2)allows only for second granularity,setitimer(2)does not -- and also has a helpful Python wrapper. With that I could boil down a minimal reproducer that triggers fairly quickly:#!/usr/bin/env python3 import signal def _input_timeout(prompt): class TimeExpiredException(Exception): pass def raise_exception(signum, frame): raise TimeExpiredException() signal.signal(signal.SIGALRM, raise_exception) try: signal.setitimer(0, 0.01) result = input(prompt) signal.setitimer(0, 0) except TimeExpiredException: result = None return result cnt = 0 while True: cnt += 1 _input_timeout(str(cnt)) print()Also note that while I said I could not reproduce on Python 3.12, I absolutely can now! Case in point:
Linux sz 6.11.0-25-generic #25-Ubuntu SMP PREEMPT_DYNAMIC Fri Apr 11 23:29:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux AMD Ryzen 7 2700X Eight-Core Processor Python 3.12.7 (main, Feb 4 2025, 14:46:03) [GCC 14.2.0] on linux $ ./heisenbug.py 16 17 18Exception ignored in tp_clear of: <class 'cell'> Traceback (most recent call last): File "/home/joe/./heisenbug.py", line 8, in raise_exception raise TimeExpiredException() _input_timeout.<locals>.TimeExpiredException: Traceback (most recent call last): File "/home/joe/./heisenbug.py", line 22, in <module> _input_timeout(str(cnt)) File "/home/joe/./heisenbug.py", line 13, in _input_timeout result = input(prompt) ^^^^^^^^^^^^^ KeyboardInterruptand
Linux reliant 6.14.0-15-generic #15-Ubuntu SMP PREEMPT_DYNAMIC Sun Apr 6 15:05:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux AMD Ryzen Threadripper PRO 3945WX 12-Cores Python 3.13.3 (main, Apr 8 2025, 19:55:40) [GCC 14.2.0] on linux $ ./heisenbug.py [...] 79 80 81 82 83Exception ignored in tp_clear of dict: Traceback (most recent call last): File "/home/joe/./heisenbug.py", line 8, in raise_exception raise TimeExpiredException() _input_timeout.<locals>.TimeExpiredException: Traceback (most recent call last): File "/home/joe/./heisenbug.py", line 22, in <module> _input_timeout(str(cnt)) ~~~~~~~~~~~~~~^^^^^^^^^^ File "/home/joe/./heisenbug.py", line 13, in _input_timeout result = input(prompt) KeyboardInterruptand
Linux julia 6.8.0-60-generic #63-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 19:04:15 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Intel(R) Core(TM) i7-6800K CPU @ 3.40GHz Python 3.12.3 (main, Feb 4 2025, 14:48:35) [GCC 13.3.0] on linux 29 30 31 32Exception ignored in tp_clear of: <class 'type'> Traceback (most recent call last): File "/home/julia/./heisenbug.py", line 8, in raise_exception raise TimeExpiredException() _input_timeout.<locals>.TimeExpiredException: Traceback (most recent call last): File "/home/julia/./heisenbug.py", line 22, in <module> _input_timeout(str(cnt)) File "/home/julia/./heisenbug.py", line 13, in _input_timeout result = input(prompt) ^^^^^^^^^^^^^ KeyboardInterruptNote that I cannot produce the issue on an ARM64 virutal machine (did not trigger up until a counter value of 40000, when I aborted).
So I did a lot of building and testing. Using this script:
#!/usr/bin/env python3 import signal def _input_timeout(prompt: str): class TimeExpiredException(Exception): pass def raise_exception(signum, frame): raise TimeExpiredException() signal.signal(signal.SIGALRM, raise_exception) try: signal.setitimer(0, 0.0001) result = input(prompt) signal.setitimer(0, 0) except TimeExpiredException: result = None return result cnt = 0 while True: cnt += 1 _input_timeout(str(cnt)) print() if cnt == 50000: breakI'm able to discern that a version is affected (return code != 0) or probably not affected (return code == 0). Note that since this is a nondeterministic issue, I can only ever verify presence of the issue, but not falsify it. This makes my analysis a bit messy.
First, I checked the 3.12, 3.13, 3.14 branches and all versions that have tags for the issue:
Ref v3.11.0a2 checked: good Ref v3.11.0a3 checked: good Ref v3.11.0a4 checked: good Ref v3.11.0a5 checked: good Ref v3.11.0a6 checked: good Ref v3.11.0a7 checked: good Ref v3.11.0b1 checked: good Ref v3.12.0a1 checked: good Ref v3.12.0a2 checked: good Ref v3.12.0a3 checked: good Ref v3.12.0a4 checked: good Ref v3.12.0a5 checked: good Ref v3.12.0a6 checked: good Ref v3.12.0a7 checked: good Ref v3.12.0b1 checked: good Ref v3.12.0b2 checked: good Ref v3.12.0b3 checked: good Ref v3.12.0b4 checked: good Ref v3.12.0rc1 checked: good Ref v3.12.0rc2 checked: good Ref v3.12.0rc3 checked: good Ref v3.12.0 checked: good Ref v3.12.1 checked: good Ref v3.12.2 checked: bad Ref v3.12.3 checked: bad Ref v3.12.4 checked: bad Ref v3.12.5 checked: bad Ref v3.12.6 checked: bad Ref v3.12.7 checked: bad Ref v3.12.8 checked: bad Ref v3.12.9 checked: bad Ref v3.12.10 checked: bad Ref v3.13.0a1 checked: skip Ref v3.13.0a2 checked: skip Ref v3.13.0a3 checked: skip Ref v3.13.0a4 checked: skip Ref v3.13.0a5 checked: skip Ref v3.13.0a6 checked: good Ref v3.13.0b1 checked: good Ref v3.13.0b2 checked: good Ref v3.13.0b3 checked: good Ref v3.13.0b4 checked: good Ref v3.13.0rc1 checked: good Ref v3.13.0rc2 checked: good Ref v3.13.0rc3 checked: bad Ref v3.13.0 checked: bad Ref v3.13.1 checked: bad Ref v3.13.2 checked: bad Ref v3.13.3 checked: bad Ref v3.14.0a1 checked: good Ref v3.14.0a2 checked: good Ref v3.14.0a3 checked: good Ref v3.14.0a4 checked: good Ref v3.14.0a5 checked: good Ref v3.14.0a6 checked: good Ref v3.14.0a7 checked: good Ref v3.14.0b1 checked: goodI built each version with
./configure --enable-optimizations --disable-test-modules. So a GIL should always be present.Then I bisected in each branch that has the issue occuring. For 3.13, the faulty commit is plausible:
# status: waiting for both good and bad commits # good: [ec610069637d56101896803a70d418a89afe0b4b] Python 3.13.0rc2 git bisect good ec610069637d56101896803a70d418a89afe0b4b # status: waiting for bad commit, 1 good commit known # bad: [fae84c70fbd9ac391ec0d3826cc114a0036a71e1] Python 3.13.0rc3 git bisect bad fae84c70fbd9ac391ec0d3826cc114a0036a71e1 # good: [c2aaf507801e34c574a10ea5f079a8260143ef9a] [3.13] gh-123560: Correct docs for "empty" format type for floats (GH-123561) (#124596) git bisect good c2aaf507801e34c574a10ea5f079a8260143ef9a # good: [c7f9332ee51f612899d817a927f706216287dd70] [3.13] gh-124378: Update test_ttk for Tcl/Tk 8.6.15 (GH-124542) (#124544) git bisect good c7f9332ee51f612899d817a927f706216287dd70 # good: [793cb77f55ac157cf3f32da863f39c408b29d78c] [3.13] gh-123942: add missing test for docstring-handling code in ast_opt.c (GH-123943) (#123955) git bisect good 793cb77f55ac157cf3f32da863f39c408b29d78c # good: [ce0eaa67032389d64fd2d7fb510e34372082eef8] [3.13] gh-124722: Fix leak in `test_detach_materialized_dict_no_memory` (GH-124769) (#124777) git bisect good ce0eaa67032389d64fd2d7fb510e34372082eef8 # good: [4f3e96f7c25c20850a7512e127045ff82d8b57f7] [3.13] gh-124613, regrtest: Detect JIT in build info (GH-124793) (#124797) git bisect good 4f3e96f7c25c20850a7512e127045ff82d8b57f7 # bad: [e0eb44ad49926dd131dc639f5506c6769e45b4eb] [3.13] GH-124567: Revert the Incremental GC in 3.13 (#124770) git bisect bad e0eb44ad49926dd131dc639f5506c6769e45b4eb # good: [bc1fae89af9df3888fab670f83b7aed8afe5a9f5] [3.13] Doc: Run HTML and non-HTML daily builds separately (GH-124493) (#124790) git bisect good bc1fae89af9df3888fab670f83b7aed8afe5a9f5 # first bad commit: [e0eb44ad49926dd131dc639f5506c6769e45b4eb] [3.13] GH-124567: Revert the Incremental GC in 3.13 (#124770)For 3.12, the faulty commit is much less plausible and has me scratching my head:
# status: waiting for both good and bad commits # good: [2305ca51448552542b2414186252123a8dc87db7] Python 3.12.1 git bisect good 2305ca51448552542b2414186252123a8dc87db7 # status: waiting for bad commit, 1 good commit known # bad: [6abddd9f6afdddc09031989e0deb25e301ecf315] Python 3.12.2 git bisect bad 6abddd9f6afdddc09031989e0deb25e301ecf315 # bad: [94b1d1fa38ada8cf7d196184a04a195c152eed75] [3.12] gh-114021: Pin various sphinxcontrib extensions to older versions (#114022) git bisect bad 94b1d1fa38ada8cf7d196184a04a195c152eed75 # good: [4882d508be46eff5ae2d55d82f0aab169a59a0ed] [3.12] gh-74573: document that ndbm can silently corrupt databases on macOS (GH-113354) (#113431) git bisect good 4882d508be46eff5ae2d55d82f0aab169a59a0ed # bad: [d2cdc6717a4a03f5fc7a7b3a5d3e55973b1ff9d5] [3.12] gh-101100: Fix Sphinx warnings for removed dead batteries (GH-113669) (#113681) git bisect bad d2cdc6717a4a03f5fc7a7b3a5d3e55973b1ff9d5 # bad: [6177a852c4b5dd82f40cef4340083a33cf16a5fd] [3.12] gh-101100: Fix Sphinx warnings in `library/cmd.rst` (GH-113502) (#113511) git bisect bad 6177a852c4b5dd82f40cef4340083a33cf16a5fd # bad: [bdad5c367f60d4939d6945f78d94608e4521009f] [3.12] gh-76785: Make test.test_interpreters executable (GH-112982) (GH-113470) git bisect bad bdad5c367f60d4939d6945f78d94608e4521009f # bad: [269cb342ad487c7c244659bd25ed6d4d7bb95876] [3.12] gh-113028: Correctly memoize str in pickle when escapes added (GH-113436) (GH-113448) git bisect bad 269cb342ad487c7c244659bd25ed6d4d7bb95876 # good: [8f526d3c9a078982937b402e3519c3bcf721c4ad] [3.12] gh-109989: Fix test_c_locale_coercion when PYTHONIOENCODING is set (GH-113378) (#113398) git bisect good 8f526d3c9a078982937b402e3519c3bcf721c4ad # bad: [15ea4a429245ddb9d919757308777521f61c681c] [3.12] gh-111784: Fix two segfaults in the elementtree module (GH-113405) (GH-113446) git bisect bad 15ea4a429245ddb9d919757308777521f61c681c # bad: [7f2ebc84e8066e2c51feb6ef6403360922bbc0aa] [3.12] gh-112559: Avoid unnecessary conversion attempts to enum_klass in signal.py (GH-113040) (#113443) git bisect bad 7f2ebc84e8066e2c51feb6ef6403360922bbc0aa # first bad commit: [7f2ebc84e8066e2c51feb6ef6403360922bbc0aa] [3.12] gh-112559: Avoid unnecessary conversion attempts to enum_klass in signal.py (GH-113040) (#113443)Hope this helps a bit. Here are links to the faulty commits:
Also note that because the faulty commit in 3.12 seems unlikely (most of the commit is testing code, the only change I can see is the type conversion to IntEnum), I did this bisection multiple times and always got the same results. I'd happily do more testing if this is useful.
Interesting, so are you saying that raising an exception is always undefined behavior in Python, always has been?
Not undefined in the C sense, but in practice very difficult to use safely. The exception can be raised from ~anywhere, and code that is not specifically and very carefully written with that expectation will break. Standard library & garbage collector code is not written like this (and realistically cannot be), so reliably using a signal handler that raises involves, amongst other things, not calling into the standard library or allowing the garbage collector to run.
The thing that makes me quite curious is that I do believe that in the scenario, this does look like it is well-defined behavior. I'm setting
signal.alarm()inside thetryblock, then cancel it before leaving the try block. AnySIGALRMshould call the installed signal handler and the raised exception should be possible to stack-unwind (there is only a single main thread there). In this instance, it seems to be the case that the signal is delivered while outside thetry...exceptblock, which should never be possible.Can you try to elaborate why this would be happening?
I am not the person you asked, but I'll try to explain (and hopefully if I get anything wrong someone more knowledgeable will correct me)...
In your example the signal will be delivered inside that
try...exceptblock, but theinputcall descends into interpreter internals and the standard library and will do a bunch of things, including possibly running the garbage collector, which is the source of the "ignored" error you are seeing. The garbage collector is callingtp_clearon some deleted garbage, then checking for an error. In this case, it finds one! The error was caused by the signal, not directly by the code it called, but it has no way to know that, and couldn't do anything useful with the information even if it did.GC has no way to gracefully handle such errors, so it prints that "ignored" message as a last ditch attempt to provide the user some indication as to why everything is going/about to go wrong. It cannot propagate the exception out and unwind the stack, since the garbage object which it thinks is throwing from
tp_clearcould have been allocated by some other totally unrelated code.
Bug report
Bug description:
First of all, I know exactly how this bug is going to sound: batshit insane. I implore you to keep reading.
I've had a timer application for ages that uses
signal.alarm()concurrently withinput()so that it has a rudimentary way of displaying things in the background while still allowing user interaction on the CLI. The gist of it is this:This program has mysteriously stopped working with Python 3.13.3 with an exceptionally strange error mode: my signal handler is called from somewhere I apparently do not expect (
HelpFormatter), causing the whole process to terminate. This always happens after exactly 29 seconds when I run my process as user and after exactly 25 seconds when I run as root. It happens only when there is no__pycache__present, i.e., when the Python bytecode is generated on startup.I tried creating a minimal reproducer, which was super difficult. Even when I remove dead code from the Python script, the bug stops triggering. Some code removal was fine, but for example if I remove the wrapper around
argparse, the bug disappears.As I said. This sounds batshit crazy.
When the mystery signal is caught, it gives one final indication where it comes from:
Note the message:
Exception ignored in tp_clear of HelpFormatter:(this was running as root, you see that it took 25 seconds.)Just so you believe me that it always triggers after the same time, here are three consecutive runs as root (I swear this is not copy paste, timings are accurate to millisecond level!):
And here some runs as user:
I'm running on 6.14.0-15-generic #15-Ubuntu. My CPU is an AMD Ryzen Threadripper PRO 3945WX 12-Cores. I'm using the Ubuntu-provided Python package (Ubuntu calls it 3.13.3-1):
I cannot reproduce this issue on Debian with Python 3.11/ARM nor on another Ubuntu with Python 3.12/x86_64.
I cannot attach files (the reproducer) to this bug, so here it is as base64:
You can simply unpack it by doing:
And pasting the above base64 blob in. Remember you have to clear
__pycache__to trigger the bug. I'm including the bytecode in case it contains something interesting.Please someone tell me this is reproducible and I'm not completely losing my marbles.
CPython versions tested on:
3.13
Operating systems tested on:
Linux