Repository navigation
Main segfaults importing _pyrepl with low value for JUMP_BACKWARD_INITIAL_VALUE #137218
Description
Activity
This is only on JIT builds, correct? Curious to see the new unpatched reproducer.
This is only on JIT builds, correct?
AFAICT, after double checking, yes.
I suspect that the bug isn't the initial value itself, but rather the code we try to compile when the value is that low. It could be that some specialization or inline cache has a nonsense value that we're not handling well, or something.
Here's the output from running
PYTHON_LLTRACE=4 ./python -m _pyrepl:Is part of this file missing, or was the output truncated by the crash itself? The last optimized trace is incomplete (it should be length 167, but only 132 elements are printed).
Or maybe we need to add a
flushcall somewhere?I believe the crash truncates it, yes.
I'm suspicious about such straightforward code segfaulting, hoping this isn't an operator error on my part.
Here's something that triggers the right crash for me, unpatched debug JIT build, please confirm whether it repros:
key = "abc" s = 0 while True: key[s] == "a"
And here's something that triggers another crash, not sure what configurations are affected (will investigate tomorrow), but also crashes an unpatched debug JIT build:
def p(keys): s = 0 r = [] while s < len(keys): ret = "" while not ret and s < len(keys): ret = keys[s] s += 1 r.extend([s]) return r while True: p("abc")
Traceback for 2nd crash:
Program received signal SIGSEGV, Segmentation fault. 0x00005555556da88a in PyObject_IsTrue (v=0x555555c88d38 <_PyRuntime+96376>) at Objects/object.c:2059 2059 else if (Py_TYPE(v)->tp_as_number != NULL && #0 0x00005555556da88a in PyObject_IsTrue (v=0x555555c88d38 <_PyRuntime+96376>) at Objects/object.c:2059 #1 0x00005555557d8a5b in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7e2a088, throwflag=0) at Python/generated_cases.c.h:11581 #2 0x00005555557db135 in _PyEval_EvalFrame (tstate=tstate@entry=0x555555cc2230 <_PyRuntime+331120>, frame=frame@entry=0x7ffff7e2a020, throwflag=throwflag@entry=0) at ./Include/internal/pycore_ceval.h:119 #3 0x00005555557db306 in _PyEval_Vector (tstate=tstate@entry=0x555555cc2230 <_PyRuntime+331120>, func=func@entry=0x7ffff7a4a8d0, locals=locals@entry=0x7ffff7a583b0, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:1981 #4 0x00005555557db402 in PyEval_EvalCode (co=co@entry=0x7ffff785c260, globals=globals@entry=0x7ffff7a583b0, locals=locals@entry=0x7ffff7a583b0) at Python/ceval.c:872 #5 0x00005555558bcf17 in run_eval_code_obj (tstate=tstate@entry=0x555555cc2230 <_PyRuntime+331120>, co=co@entry=0x7ffff785c260, globals=globals@entry=0x7ffff7a583b0, locals=locals@entry=0x7ffff7a583b0) at Python/pythonrun.c:1365 #6 0x00005555558bd0bd in run_mod (mod=mod@entry=0x555555ee2968, filename=filename@entry=0x7ffff7aa31b0, globals=globals@entry=0x7ffff7a583b0, locals=locals@entry=0x7ffff7a583b0, flags=flags@entry=0x7fffffffd9b8, arena=arena@entry=0x7ffff7ac4760, interactive_src=0x0, generate_new_source=0) at Python/pythonrun.c:1436 #7 0x00005555558bd93d in pyrun_file (fp=fp@entry=0x555555e2e6d0, filename=filename@entry=0x7ffff7aa31b0, start=start@entry=257, globals=globals@entry=0x7ffff7a583b0, locals=locals@entry=0x7ffff7a583b0, closeit=closeit@entry=1, flags=0x7fffffffd9b8) at Python/pythonrun.c:1293 #8 0x00005555558bf1d1 in _PyRun_SimpleFileObject (fp=fp@entry=0x555555e2e6d0, filename=filename@entry=0x7ffff7aa31b0, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffd9b8) at Python/pythonrun.c:521 #9 0x00005555558bf3c7 in _PyRun_AnyFileObject (fp=fp@entry=0x555555e2e6d0, filename=filename@entry=0x7ffff7aa31b0, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffd9b8) at Python/pythonrun.c:81 #10 0x00005555558e872d in pymain_run_file_obj (program_name=program_name@entry=0x7ffff7aa3d80, filename=filename@entry=0x7ffff7aa31b0, skip_source_first_line=0) at Modules/main.c:410 #11 0x00005555558e8855 in pymain_run_file (config=config@entry=0x555555c8d2d8 <_PyRuntime+114200>) at Modules/main.c:429 #12 0x00005555558e939f in pymain_run_python (exitcode=exitcode@entry=0x7fffffffdb34) at Modules/main.c:691 #13 0x00005555558e9654 in Py_RunMain () at Modules/main.c:772Here's the output from running with
PYTHON_LLTRACE=4for the second crash:
lltrace_2nd_crash.txt
And here's the output fromPYTHON_OPT_DEBUG=4:
opt_debug_2nd_crash.txtYep! This gives me a segfault:
X = "X" while True: X[0] == ""
This gives me an assertion failure (
_POP_TOP_UNICODE.c:121: _Py_CODEUNIT *_JIT_ENTRY(_PyInterpreterFrame *, _PyStackRef *, PyThreadState *): Assertion `PyUnicode_CheckExact(PyStackRef_AsPyObjectBorrow(value))' failed.):X = "X" while True: X[0]
Your second reproducer looks a lot like my second one here, and I wouldn't be surprised if both of these are related. Something about string indexing is broken.
Hm, weird. After a
make cleanand rebuild the issue goes away.Maybe our
makerule for the JIT is broken. I think we need to say thatjit.odepends onjit_stencils*.h, not justjit_stencils.h. We now have a platform-specific header file (likejit_stencils-x86_64-pc-linux-gnu.h) and I'm not sure that the existing logic is picking that dependency up correctly.@devdanzin, does a clean rebuild work correctly for you?
Yep! This gives me a segfault:
Great, I was really worried it was my fault 😄
@devdanzin, does a clean rebuild work correctly for you?
It still segfaults after make distclean, but if I also bypass ccache it stops crashing. Then patching and re-enabling ccache also doesn't crash, which makes no sense to me but I hope makes to you.
Edit: I simply can't repro anymore, any idea about how to bring the brokenness back?
Edit 2: BTW, I see
make distcleandoesrm -f jit_stencils.h, should it also removejit_stencils*.h?So I'm comfortable calling this a bug in the build system, not in the JIT. Are you interested in fixing it? You can ping me for help/review. Should be a couple of fairly straightforward changes in the Makefile.
CC @savannahostrowski since she might have thoughts here.
I'd rather not tackle this one, but thanks for asking.
Are you able to get back into the broken state? I wasted too much time trying to yesterday 😅.
I think getting into the broken state is hard for the same reasons that getting out of the broken state is hard. ;)
- added a commit that references this issue
on Sep 22, 2025 Since the PR was merged, running
maketwice (on the same source) will compilePrograms/_freeze_moduleagain and re-freeze a bunch of modules. Would it be possible to avoid that?Hmm, let me investigate this a bit.
Thanks!
If you want me to look into it, just shout :)Reacted by Savannah Ostrowski- added a commit that references this issue
on Oct 5, 2025
Crash report
What happened?
Edit: I'm seeing this crash even without patching
JUMP_BACKWARD_INITIAL_VALUE. Will try to figure out a MRE for unpatched crash.If we
#define JUMP_BACKWARD_INITIAL_VALUE 702(or lower) inInclude/internal/pycore_backoff.h, a debug JIT build will segfault when trying to run to the new REPL. Can also be triggered by:Using some other modules like
randomorhttp.serverdoesn't crash.An even lower value like 344 will segfault earlier, during the build process.
This happens since #136307 landed in 377b787.
The diff below, which is part of that commit, is enough to cause the segfault together with defining
JUMP_BACKWARD_INITIAL_VALUEto 702 or lower:Backtrace:
Here's the output from running
PYTHON_LLTRACE=4 ./python -m _pyrepl:lltrace_crash.txt
And here for
PYTHON_OPT_DEBUG=4 ./python -m _pyrepl:opt_debug_crash.txt
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.15.0a0 (heads/main-dirty:d7e12a362a2, Jul 29 2025, 18:24:07) [GCC 13.3.0]
Linked PRs
makefor JIT stencils #137265Programs/_freeze_module#139241