Repository navigation
JIT: assertion failure in _PyObject_GC_UNTRACK #137007
Description
Activity
I cannot seem to reproduce this with a debug build of the JIT and with a patch applied, for the reproducer. The reproducer runs till completion for me.
Wait sorry, I didn't see that you need two packages in editable mode for this. Re-investigating.
I feel like that part is probably a red herring. Probably the import machinery is taking a different path to find
randomwhen that's the case, causing GC to happen or something.I feel like that part is probably a red herring. Probably the import machinery is taking a different path to find
randomwhen that's the case, causing GC to happen or something.I tried tweaking GC thresholds and don't get anything either. It's not a GC problem. I narrowed it down to the code path in the executor code deallocating the executor when it's not yet tracked by GC.
I was able to reproduce this by (cloning, then) installing pip and pypa/sampleproject as editable. However, it somehow led to a segfault this time:
Program received signal SIGSEGV, Segmentation fault. _PyObject_GC_UNTRACK (op=0x555555e448a0) at ./Include/internal/pycore_gc.h:279 279 _PyGCHead_SET_NEXT(prev, next); #0 _PyObject_GC_UNTRACK (op=0x555555e448a0) at ./Include/internal/pycore_gc.h:279 #1 uop_dealloc (op=0x555555e448a0) at Python/optimizer.c:269 #2 0x00005555556c059d in _Py_Dealloc (op=0x555555e448a0) at Objects/object.c:3206 #3 0x00005555557c86de in Py_DECREF_MORTAL (op=<optimized out>) at ./Include/internal/pycore_object.h:482 #4 PyStackRef_CLOSE (ref=...) at ./Include/internal/pycore_stackref.h:701 #5 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:7656 #6 0x00005555557d2b3c in _PyEval_EvalFrame (tstate=0x555555bb9a48 <_PyRuntime+315048>, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119 #7 _PyEval_Vector (tstate=0x555555bb9a48 <_PyRuntime+315048>, func=0x7ffff7496fb0, locals=0x0, args=0x7fffffffd178, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1977 #8 0x000055555564905b in _PyObject_VectorcallTstate (tstate=0x555555bb9a48 <_PyRuntime+315048>, callable=0x7ffff7496fb0, args=0x7fffffffd178, nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:169 #9 PyObject_CallOneArg (func=func@entry=0x7ffff7496fb0, arg=arg@entry=0x7ffff7333b60) at Objects/call.c:395 #10 0x00005555556feed2 in call_unbound_noarg (unbound=<optimized out>, func=0x7ffff7496fb0, self=0x7ffff7333b60) at Objects/typeobject.c:3019 #11 maybe_call_special_no_args (self=0x7ffff7333b60, attr=<optimized out>, attr_is_none=<synthetic pointer>) at Objects/typeobject.c:3132 #12 slot_tp_hash (self=0x7ffff7333b60) at Objects/typeobject.c:10489 #13 0x00005555556e8e95 in _PyObject_HashFast (op=0x7ffff7333b60) at ./Include/internal/pycore_object.h:872 #14 _PySet_AddTakeRef (so=0x7ffff73bd2a0, key=0x7ffff7333b60) at Objects/setobject.c:234 #15 0x00007ffff71c0225 in ?? () #16 0x00007fffffffd590 in ?? () #17 0xfffffffffffffffe in ?? () #18 0x00007fffffffd590 in ?? () #19 0x00007ffff71c7017 in ?? () #20 0x00007ffff7fac0a0 in ?? () #21 0x00007ffff7fac160 in ?? () #22 0x0000555555c720c0 in ?? () #23 0x0000555555b70319 in _PyRuntime () #24 0x0000555555d11116 in ?? () #25 0x00005555557ca888 in _PyEval_EvalFrameDefault (tstate=<error reading variable: Cannot access memory at address 0xfffffffffffffd58>, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:5598 Backtrace stopped: previous frame inner to this frame (corrupt stack?)Can you reproduce by installing those two packages?
Here's a reproducer for the segfault in a JIT non-debug build without the need of installing packages, please use this diff:
index 454c8dde031..9e21c41421a 100644 --- a/Include/internal/pycore_backoff.h +++ b/Include/internal/pycore_backoff.h @@ -99,8 +99,8 @@ backoff_counter_triggers(_Py_BackoffCounter counter) // Must be larger than ADAPTIVE_COOLDOWN_VALUE, otherwise when JIT code is // invalidated we may construct a new trace before the bytecode has properly // re-specialized: -#define JUMP_BACKWARD_INITIAL_VALUE 4095 -#define JUMP_BACKWARD_INITIAL_BACKOFF 12 +#define JUMP_BACKWARD_INITIAL_VALUE 63 +#define JUMP_BACKWARD_INITIAL_BACKOFF 6 static inline _Py_BackoffCounter initial_jump_backoff_counter(void) { @@ -112,8 +112,8 @@ initial_jump_backoff_counter(void) * Must be larger than ADAPTIVE_COOLDOWN_VALUE, * otherwise when a side exit warms up we may construct * a new trace before the Tier 1 code has properly re-specialized. */ -#define SIDE_EXIT_INITIAL_VALUE 4095 -#define SIDE_EXIT_INITIAL_BACKOFF 12 +#define SIDE_EXIT_INITIAL_VALUE 63 +#define SIDE_EXIT_INITIAL_BACKOFF 6 static inline _Py_BackoffCounter initial_temperature_backoff_counter(void) diff --git a/Include/internal/pycore_optimizer.h b/Include/internal/pycore_optimizer.h index 8b7f12bf03d..fee929f122f 100644 --- a/Include/internal/pycore_optimizer.h +++ b/Include/internal/pycore_optimizer.h @@ -121,7 +121,7 @@ PyAPI_FUNC(void) _Py_Executors_InvalidateCold(PyInterpreterState *interp); // This is the length of the trace we project initially. #define UOP_MAX_TRACE_LENGTH 800 -#define TRACE_STACK_SIZE 5 +#define TRACE_STACK_SIZE 10 int _Py_uop_analyze_and_optimize(_PyInterpreterFrame *frame, _PyUOpInstruction *trace, int trace_len, int curr_stackentries, @@ -152,7 +152,7 @@ static inline uint16_t uop_get_error_target(const _PyUOpInstruction *inst) } // Holds locals, stack, locals, stack ... co_consts (in that order) -#define MAX_ABSTRACT_INTERP_SIZE 4096 +#define MAX_ABSTRACT_INTERP_SIZE 8192 #define TY_ARENA_SIZE (UOP_MAX_TRACE_LENGTH * 5) @@ -163,7 +163,7 @@ static inline uint16_t uop_get_error_target(const _PyUOpInstruction *inst) // progress (and inserting a new ENTER_EXECUTOR instruction). In practice, this // is the "maximum amount of polymorphism" that an isolated trace tree can // handle before rejoining the rest of the program. -#define MAX_CHAIN_DEPTH 4 +#define MAX_CHAIN_DEPTH 16 /* Symbols */ /* See explanation in optimizer_symbols.c */
And this reproducer:
import sys from importlib.machinery import all_suffixes as module_suffixes # Not sure why, but this unused import seems necessary from importlib.util import spec_from_file_location class _EditableFinder: @classmethod def _find_spec(cls, fullname, candidate_path): return spec_from_file_location(fullname, candidate_path.with_suffix("pyc")) class _EditableFinder2: @classmethod def _find_spec(cls, fullname, candidate_path): return spec_from_file_location(fullname, candidate_path.with_suffix("py")) sys.meta_path.append(_EditableFinder) sys.meta_path.append(_EditableFinder2) from random import randint class UnstableHash_unstable_hash_object_v4: hash_count = 0 def __hash__(self): self.hash_count += 1 new_hash = 5 if self.hash_count < 70 else randint(0, 1024) print(new_hash) return new_hash unstable_hash_object_v4 = UnstableHash_unstable_hash_object_v4() tuple_v6 = (90.1433,) def uop_harness_f1(): class ShapeA_la_7776: payload = 123 class ShapeB_la_7776: @property def payload(self): return 'property_payload' class ShapeC_la_7776: def payload(self): return id(self) class ShapeD_la_7776: __slots__ = ['payload'] def __init__(self): self.payload = 'slot_payload' shapes_la_7776 = [ShapeA_la_7776(), ShapeB_la_7776(), ShapeC_la_7776(), ShapeD_la_7776()] for i in range(500): obj = shapes_la_7776[i % len(shapes_la_7776)] try: payload_val = obj.payload if callable(payload_val): payload_val() payload_val() payload_val() payload_val() except Exception: pass {unstable_hash_object_v4, unstable_hash_object_v4} {unstable_hash_object_v4, unstable_hash_object_v4} {unstable_hash_object_v4, unstable_hash_object_v4} {unstable_hash_object_v4, unstable_hash_object_v4} for i_loop_1589 in {2, 1, 2, 3, 4, 5, 6, 8, 10, 9, 12, 10, 12, 13, 14, 15, 17, 17}: x, *y = tuple_v6 x, *y = tuple_v6 for i_f1 in range(300): try: uop_harness_f1() except Exception as e: break
Here's a reproducer for the segfault in a JIT non-debug build without the need of installing packages, please use this diff:
Hm, this isn't working for me. Non-debug build, patches applied, no venv.
Hm, this isn't working for me. Non-debug build, patches applied, no venv.
This isn't my make-it-reliably-reproducible day, is it? It doesn't work without a venv for me either, and it turns out I had lafleur installed when I though I didn't and removing it stops reproducing. What a mess. Sorry again. I will spend a lot more time making sure any reports are reproducible.
So here's how it's working for me:
(release_jit_cpython_venv) :~ $ pip3 list Package Version Editable project location ---------- ------- ----------------------------- lafleur 0.0.1 /home/danzin/projects/lafleur peppercorn 0.6 pip 25.1.1 (release_jit_cpython_venv) :~ $ ~/venvs/release_jit_cpython_venv/bin/python ~/crashers/jit/reduced_crash_retcode_child_15732_2.py 5 5 5 5 [... a bunch of numbers] 427 493 35 Segmentation fault (core dumped)I hope it works now.
Also, for the future, setting a random seed really helps when randomness is involved. ;)
Also, for the future, setting a random seed really helps when randomness is involved. ;)
There's a seed for the mutations (that is, test case generation is reproducible), it just never occurred to me to add it to the fuzzing script (that is, doh)🤦. Thanks, will adopt it :)
Are you able to reproduce it with latest instructions?
Here's a small reduction I got, that shows calling randint isn't necessary:
import gc gc.set_threshold(10) # Doesn't seem to work with 1 or most numbers, but 10 works from random import randint # Seems necessary to get GC to where we want class _EditableFinder: @classmethod def _find_spec(cls, fullname, candidate_path): pass class UnstableHash_unstable_hash_object_v4: def __hash__(self): print(5, end=" ", flush=True) # If we don't flush, no crash. If we do a bare print, it also crashes. return 5 unstable_hash_object_v4 = UnstableHash_unstable_hash_object_v4() tuple_v6 = (90.1433,) def uop_harness_f1(): class ShapeA_la_7776: payload = 123 class ShapeB_la_7776: @property def payload(self): return 'property_payload' class ShapeC_la_7776: def payload(self): return id(self) class ShapeD_la_7776: __slots__ = ['payload'] def __init__(self): self.payload = 'slot_payload' shapes_la_7776 = [ShapeA_la_7776(), ShapeB_la_7776(), ShapeC_la_7776(), ShapeD_la_7776()] for i in range(500): obj = shapes_la_7776[i % len(shapes_la_7776)] payload_val = obj.payload if callable(payload_val): payload_val() {unstable_hash_object_v4, unstable_hash_object_v4, unstable_hash_object_v4} for i_loop_1589 in {2, 1, 2, 3, 4, 5, 6, 8, 10, 9, 12, 10, 12, 13, 14, 15, 15, 17}: x, *y = tuple_v6 for i_f1 in range(100): uop_harness_f1()
Here's where I am:
from importlib.machinery import all_suffixes as module_suffixes from importlib.util import spec_from_file_location class C: @classmethod def f(): pass class D: def f(self): pass from random import randint from random import seed seed(5) # import gc # gc.set_threshold(1) class XXX: hash_count = 0 def __hash__(self): self.hash_count += 1 new_hash = None return randint(0, 1) xxx = XXX() tuple_v6 = (0,) def f(): class W: payload = None class X: @property def payload(self): return 'property_payload' class Y: def payload(self): return id(self) class Z: __slots__ = ['payload'] def __init__(self): self.payload = 'slot_payload' stuff = [W(), X(), Y(), Z()] for i in range(500): try: if callable(stuff[i % 4].payload): pass except: pass {xxx, xxx, xxx, xxx, xxx, xxx} for i_loop_1589 in {0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13}: x, *y = tuple_v6 for i_f1 in range(88): f()
I can no longer repro on main. I think #136411 fixed this.
I can still repro on main by changing the gc tuning to
gc.set_threshold(11)and raising number of iterations from 100 to 400. Does it abort for you too?import gc gc.set_threshold(11) # Doesn't seem to work with 1 or most numbers, but 10 works from random import randint # Seems necessary to get GC to where we want class _EditableFinder: @classmethod def _find_spec(cls, fullname, candidate_path): pass class UnstableHash_unstable_hash_object_v4: def __hash__(self): print(5, end=" ", flush=True) # If we don't flush, no crash. If we do a bare print, it also crashes. return 5 unstable_hash_object_v4 = UnstableHash_unstable_hash_object_v4() tuple_v6 = (90.1433,) def uop_harness_f1(): class ShapeA_la_7776: payload = 123 class ShapeB_la_7776: @property def payload(self): return 'property_payload' class ShapeC_la_7776: def payload(self): return id(self) class ShapeD_la_7776: __slots__ = ['payload'] def __init__(self): self.payload = 'slot_payload' shapes_la_7776 = [ShapeA_la_7776(), ShapeB_la_7776(), ShapeC_la_7776(), ShapeD_la_7776()] for i in range(500): obj = shapes_la_7776[i % len(shapes_la_7776)] payload_val = obj.payload if callable(payload_val): payload_val() {unstable_hash_object_v4, unstable_hash_object_v4, unstable_hash_object_v4} for i_loop_1589 in {2, 1, 2, 3, 4, 5, 6, 8, 10, 9, 12, 10, 12, 13, 14, 15, 15, 17}: x, *y = tuple_v6 for i_f1 in range(400): uop_harness_f1()
I'm unable to repro with the given code.
Hm, in another machine it only repros with
gc.set_threshold(14). Can you try a few different values for that threshold if it still doesn't abort for you?Sorry for the fickle MRE.
- added a parent issue
on Oct 11, 2025 - added a commit that references this issue
on Dec 10, 2025 Since this can't be reproduced anymore, I'm closing the isssue.
Crash report
What happened?
It's possible to non-deterministically abort a special debug JIT build by running the code below, running from a Python executable from a venv where you installed two packages in editable mode.
For this special build, apply this diff:
This MRE must be invoked like this:
Sorry about another long MRE, this code also seems stubbornly hard to reduce. MRE:
Backtrace:
Here are the outputs from running with PYTHON_OPT_DEBUG=4 and PYTHON_LLTRACE=4, since they are too long I've attached them as files:
opt_debug_15732.txt
lltrace_15732.txt
Found using lafleur.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.15.0a0 (heads/main-dirty:6bf1c0ab349, Jul 22 2025, 11:26:33) [GCC 14.2.0]
Linked PRs