Repository navigation
JIT: Segfault in _Py_LazyJitTrampoline with ASan/UBSan enabled #139834
Description
Bug report
Bug description:
When CPython is built on the main branch with the experimental JIT, AddressSanitizer (ASan), and UndefinedBehaviorSanitizer (UBSan) enabled, it produces a segmentation fault.
The crash is easily reproducible and appears to be triggered by operations that use the inspect module, such as Tab autocompletion in the REPL.
The full crash log shows that before the final SEGV in _Py_LazyJitTrampoline, UBSan reports several runtime errors related to misaligned memory stores and loads within Python/jit.c. This suggests that incorrect memory access is the root cause of the crash.
The final error reported by AddressSanitizer is:
==...==ERROR: AddressSanitizer: SEGV on unknown address ... in _Py_LazyJitTrampoline
This indicates a read from an invalid memory address, likely due to a corrupted pointer from the earlier misaligned access.
Steps to Reproduce
-
Build CPython from the
mainbranch using the following configuration script:#!/bin/bash set -e BUILD_DIR="build/debug" echo "--- Configuring and building in $BUILD_DIR ---" mkdir -p "$BUILD_DIR" cd "$BUILD_DIR" CC=clang CXX=clang++ ../../configure --with-pydebug \ --with-address-sanitizer \ --with-undefined-behavior-sanitizer \ --enable-experimental-jit make -j$(nproc) echo "--- Build complete in $BUILD_DIR. Executable: $PWD/python ---" cd ../..
-
Start the REPL using the newly built executable:
./build/debug/python
-
Trigger the crash: In the Python REPL, type
import ins(without hitting Enter) and then press the Tab key for autocompletion. -
The REPL will immediately crash with a segmentation fault.
(Note: Running the test suite with python -m test also fails with similar errors.)
Full Crash Log (with UBSan and ASan)
Direct interpreter crash (with clang 18)
Python 3.15.0a0 (heads/main:8b9606a2c57, Oct 9 2025, 12:17:36) [Clang 18.1.8 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
Python/jit.c:228:5: runtime error: store to misaligned address 0x7f699e5aa5d4 for type 'uint64_t' (aka 'unsigned long'), which requires 8 byte alignment
0x7f699e5aa5d4: note: pointer points here
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/jit.c:228:5
Python/jit.c:396:24: runtime error: load of misaligned address 0x7f699e5aa5d4 for type 'uint64_t' (aka 'unsigned long'), which requires 8 byte alignment
0x7f699e5aa5d4: note: pointer points here
00 00 00 00 98 08 47 59 20 56 00 00 f8 8a db 58 20 56 00 00 ae bc 22 9e 69 7f 00 00 00 00 00 00
^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/jit.c:396:24
Python/jit.c:220:5: runtime error: store to misaligned address 0x7f699e5aa0d5 for type 'uint32_t' (aka 'unsigned int'), which requires 4 byte alignment
0x7f699e5aa0d5: note: pointer points here
89 e6 ff 15 00 00 00 00 4c 8b 7b 40 4d 85 ff 0f 84 c0 01 00 00 48 c7 43 40 00 00 00 00 85 c0 0f
^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/jit.c:220:5
AddressSanitizer:DEADLYSIGNAL
=================================================================
==18131==ERROR: AddressSanitizer: SEGV on unknown address 0x7f699da01ff8 (pc 0x56205944cf37 bp 0x7ffea7292d10 sp 0x7ffea7290720 T0)
==18131==The signal is caused by a READ memory access.
#0 0x56205944cf37 in _Py_LazyJitTrampoline /[...]/cpython/Python/jit.c:624:12
#1 0x5620591269bd in _PyEval_EvalFrameDefault /[...]/cpython/Python/generated_cases.c.h:7671:25
#2 0x5620590e1791 in _PyEval_Vector /[...]/cpython/Python/ceval.c:2001:12
#3 0x5620595d9cdf in _PyObject_VectorcallTstate /[...]/cpython/./Include/internal/pycore_call.h:169:11
#4 0x5620595d9cdf in _PyObject_CallNoArgs /[...]/cpython/./Include/internal/pycore_call.h:185:12
#5 0x5620595d9cdf in pymain_run_interactive_hook /[...]/cpython/Modules/main.c:512:24
#6 0x5620595d9459 in pymain_run_stdin /[...]/cpython/Modules/main.c:550:13
#7 0x5620595d7215 in pymain_run_python /[...]/cpython/Modules/main.c:694:21
#8 0x5620595d7215 in Py_RunMain /[...]/cpython/Modules/main.c:772:5
#9 0x5620595d8168 in pymain_main /[...]/cpython/Modules/main.c:802:12
#10 0x5620595d83db in Py_BytesMain /[...]/cpython/Modules/main.c:826:12
#11 0x7f699e21e24c in __libc_start_main (/lib64/libc.so.6+0x3524c) (BuildId: 74f77bf013a66413c77197c121955e029c32d259)
#12 0x5620587d3759 in _start /home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:120
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /[...]/cpython/Python/jit.c:624:12 in _Py_LazyJitTrampoline
==18131==ABORTINGClick to expand the full crash log
➜ main git:(gh-139269) ./build/debug_main/python
Python 3.15.0a0 (heads/main-dirty:d2deb8fdef, Oct 9 2025, 12:40:31) [Clang 20.1.2 (0ubuntu1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ins
../../Python/jit.c:228:5: runtime error: store to misaligned address 0x7b33f8e275ab for type 'uint64_t' (aka 'unsigned long'), which requires 8 byte alignment
0x7b33f8e275ab: note: pointer points here
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../../Python/jit.c:228:5
../../Python/jit.c:396:24: runtime error: load of misaligned address 0x7b33f8e275ab for type 'uint64_t' (aka 'unsigned long'), which requires 8 byte alignment
0x7b33f8e275ab: note: pointer points here
00 00 00 00 48 b0 75 7d bd 55 00 00 58 11 0a 7d
bd 55 00 00 d0 bd c3 f8 33 7b 00 00 00 00 00 00
^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../../Python/jit.c:396:24
../../Python/jit.c:220:5: runtime error: store to misaligned address 0x7b33f8e270ca for type 'uint32_t' (aka 'unsigned int'), which requires 4 byte alignment
0x7b33f8e270ca: note: pointer points here
89 de ff 15 00 00 00 00 4d 8b 6c 24 40 4d 85 ed
0f 84 a1 01 00 00 49 c7 44 24 40 00 00 00 00 85
^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../../Python/jit.c:220:5
AddressSanitizer:DEADLYSIGNAL
=================================================================
==212839==ERROR: AddressSanitizer: SEGV on unknown address 0x7b33f8e21ff8 (pc 0x55bd7d737ab7 bp 0x7ffcb5aacf70 sp 0x7ffcb5aaa980 T0)
==212839==The signal is caused by a READ memory access.
#0 0x55bd7d737ab7 in _Py_LazyJitTrampoline /home/shamil/oss/cpython/main/build/debug_main/../../Python/jit.c:624:12
#1 0x55bd7d40e524 in _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/build/debug_main/../../Python/generated_cases.c.h:7671:25
#2 0x55bd7cf8f963 in _PyEval_EvalFrame /home/shamil/oss/cpython/main/build/debug_main/../../Include/internal/pycore_ceval.h:121:16
#3 0x55bd7cf8f963 in gen_send_ex2 /home/shamil/oss/cpython/main/build/debug_main/../../Objects/genobject.c:259:24
#4 0x55bd7cf87de0 in gen_iternext /home/shamil/oss/cpython/main/build/debug_main/../../Objects/genobject.c:634:9
#5 0x55bd7d495670 in _PyForIter_VirtualIteratorNext /home/shamil/oss/cpython/main/build/debug_main/../../Python/ceval.c:3587:24
#6 0x55bd7d40ba64 in _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/build/debug_main/../../Python/generated_cases.c.h:5649:36
#7 0x55bd7cf8f963 in _PyEval_EvalFrame /home/shamil/oss/cpython/main/build/debug_main/../../Include/internal/pycore_ceval.h:121:16
#8 0x55bd7cf8f963 in gen_send_ex2 /home/shamil/oss/cpython/main/build/debug_main/../../Objects/genobject.c:259:24
#9 0x55bd7cf87de0 in gen_iternext /home/shamil/oss/cpython/main/build/debug_main/../../Objects/genobject.c:634:9
#10 0x55bd7cfe844a in list_extend_iter_lock_held /home/shamil/oss/cpython/main/build/debug_main/../../Objects/listobject.c:1263:26
#11 0x55bd7cfde7f5 in _list_extend /home/shamil/oss/cpython/main/build/debug_main/../../Objects/listobject.c:1452:15
#12 0x55bd7cff4b1d in list___init___impl /home/shamil/oss/cpython/main/build/debug_main/../../Objects/listobject.c:3496:13
#13 0x55bd7cfe4cf9 in list_vectorcall /home/shamil/oss/cpython/main/build/debug_main/../../Objects/listobject.c:3520:13
#14 0x55bd7cf04e8e in _PyObject_VectorcallTstate /home/shamil/oss/cpython/main/build/debug_main/../../Include/internal/pycore_call.h:169:11
#15 0x55bd7d41e0c7 in _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/build/debug_main/../../Python/generated_cases.c.h:1620:35
#16 0x55bd7d3c99a1 in _PyEval_Vector /home/shamil/oss/cpython/main/build/debug_main/../../Python/ceval.c:2001:12
#17 0x55bd7cf08361 in _PyObject_VectorcallTstate /home/shamil/oss/cpython/main/build/debug_main/../../Include/internal/pycore_call.h:169:11
#18 0x55bd7cf08361 in PyObject_CallOneArg /home/shamil/oss/cpython/main/build/debug_main/../../Objects/call.c:395:12
#19 0x55bd7cf4b585 in property_descr_get /home/shamil/oss/cpython/main/build/debug_main/../../Objects/descrobject.c:1696:12
#20 0x55bd7d0aafb0 in _PyObject_GenericGetAttrWithDict /home/shamil/oss/cpython/main/build/debug_main/../../Objects/object.c:1837:19
#21 0x55bd7d0a8e86 in PyObject_GetAttr /home/shamil/oss/cpython/main/build/debug_main/../../Objects/object.c:1313:18
#22 0x55bd7d41277c in _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/build/debug_main/../../Python/generated_cases.c.h:7865:40
#23 0x55bd7d3c99a1 in _PyEval_Vector /home/shamil/oss/cpython/main/build/debug_main/../../Python/ceval.c:2001:12
#24 0x55bd7cf15183 in _PyObject_VectorcallTstate /home/shamil/oss/cpython/main/build/debug_main/../../Include/internal/pycore_call.h:169:11
#25 0x55bd7cf114fd in method_vectorcall /home/shamil/oss/cpython/main/build/debug_main/../../Objects/classobject.c:95:18
#26 0x55bd7cf07a4b in _PyVectorcall_Call /home/shamil/oss/cpython/main/build/debug_main/../../Objects/call.c:273:16
#27 0x55bd7d3fb088 in _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/build/debug_main/../../Python/generated_cases.c.h:2616:32
#28 0x55bd7d3c99a1 in _PyEval_Vector /home/shamil/oss/cpython/main/build/debug_main/../../Python/ceval.c:2001:12
#29 0x55bd7cf078fb in _PyVectorcall_Call /home/shamil/oss/cpython/main/build/debug_main/../../Objects/call.c:285:24
#30 0x55bd7d8d85ac in pymain_start_pyrepl /home/shamil/oss/cpython/main/build/debug_main/../../Modules/main.c:310:26
#31 0x55bd7d8d771b in pymain_run_stdin /home/shamil/oss/cpython/main/build/debug_main/../../Modules/main.c:570:12
#32 0x55bd7d8d53a1 in pymain_run_python /home/shamil/oss/cpython/main/build/debug_main/../../Modules/main.c:694:21
#33 0x55bd7d8d53a1 in Py_RunMain /home/shamil/oss/cpython/main/build/debug_main/../../Modules/main.c:772:5
#34 0x55bd7d8d62c1 in pymain_main /home/shamil/oss/cpython/main/build/debug_main/../../Modules/main.c:802:12
#35 0x55bd7d8d652b in Py_BytesMain /home/shamil/oss/cpython/main/build/debug_main/../../Modules/main.c:826:12
#36 0x7b33f8c2a577 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#37 0x7b33f8c2a63a in __libc_start_main csu/../csu/libc-start.c:360:3
#38 0x55bd7ca937d4 in _start (/home/shamil/oss/cpython/main/build/debug_main/python+0x96b7d4) (BuildId: e4e153f9517a81a1b32dbd4f646b3bf2a95714f3)
==212839==Register values:
rax = 0x00007b33f8e22000 rbx = 0x000079c3f7fe5be8 rcx = 0x0000000000000000 rdx = 0x0000000000000005
rdi = 0x00007b33f8e22000 rsi = 0x0000000000001000 rbp = 0x00007ffcb5aacf70 rsp = 0x00007ffcb5aaa980
r8 = 0x00000f667f1c4400 r9 = 0x00007b33f8e2201e r10 = 0x00000f667f1c4403 r11 = 0x0000000000000202
r12 = 0x000055bd7e72e1a0 r13 = 0x000079c3f7fe5c80 r14 = 0x00000ab7afc6975c r15 = 0x000055bd7e34bae0
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/shamil/oss/cpython/main/build/debug_main/../../Python/jit.c:624:12 in _Py_LazyJitTrampoline
==212839==ABORTING
% ➜ main git:(gh-139269)CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Activity
picnixz commentedon Oct 9, 2025
ashm-dev commentedon Oct 9, 2025
picnixz commentedon Oct 9, 2025
ashm-dev commentedon Oct 9, 2025
picnixz commentedon Oct 9, 2025
Ok, so it's even more funny but my interpreter directly crashes :D I don't know if it's because I have something in my history or whatever, but with LLVM 18 (and not the official 19), my REPL crashes without me writing anything.
picnixz commentedon Oct 9, 2025
The misaligned reads/writes are in the
patch_*functions that #139271 should have fixed. Do they still apear after this patch?ashm-dev commentedon Oct 9, 2025
No, after this fix, the tests with this error do not crash and my test scripts work without problems.
picnixz commentedon Oct 9, 2025
That wasn't exactly what I asked, sorry. I wanted to check if 139271 fixed the 3 first errors in the tracebacks, and yes it does. Now, we have an issue with trampolines, but the problem is that I cannot reproduce it consistently.
cc @markshannon
picnixz commentedon Oct 9, 2025
To summarize:
ashm-dev commentedon Oct 9, 2025
Yeah, makes sense — I’ve also seen it outside of inspect, e.g. with unittest.mock, so it’s probably the same lazy trampoline issue.
ashm-dev commentedon Oct 9, 2025
@picnixz You might try this one — it hits a segfault for me 100% of the time:
picnixz commentedon Oct 9, 2025
That's likely because we enable the trampolines for this snippet. IIRC, this can be verified by using
PYTHON_LLTRACE=3and see that jitification happens. I don't think it'll help tracking down the issue more precisely though.ashm-dev commentedon Oct 9, 2025
Here is the full log with
PYTHON_LLTRACE=3inspect
unittest.mock
picnixz commentedon Oct 9, 2025
I don't think this specific trace will be useful (but I may be wrong) considering its size. I just wanted to tell that you can see whether the JIT is indeed used or not. Not all code will be JIT-ified; it depends on whether it's needed or not (and you can check if the JIT is effectively in effect by using PYTHON_LLTRACE). I suggest you try to do find a reproducer that has smaller JIT traces if possible.
ashm-dev commentedon Oct 9, 2025
@picnixz
You're right, the previous trace was too large. I've created a better, minimal reproducer that triggers the same segmentation fault.
This version uses a
while Trueloop instead of aforloop. This avoids the overhead of the iterator opcodes (FOR_ITER_RANGE, etc.) and produces a much cleaner and shorter JIT trace, which should hopefully make debugging easier.efimov-mikhail commentedon Nov 17, 2025
Can you reproduce this crash on current main branch, @ashm-dev ?
ashm-dev commentedon Nov 17, 2025
@efimov-mikhail
I’d be happy to, but first I need to convince main to actually build on my machine – right now the build is failing.
Once I win that battle, I’ll re-check whether the crash is still reproducible on the current main branch and report back here.
I’m attaching my build.log with the errors in case I’m missing something obvious.
build.log
efimov-mikhail commentedon Nov 17, 2025
Ok, I got it. It's interesting that you can't build, but it's very hard (but not impossible) to reproduce build crash on my machine.