Repository navigation
JIT: executor->vm_data.valid assertion failure in unlink_executor #136996
Description
Activity
I can't reproduce this. I installed https://github.com/pypa/sampleproject in editable mode on a JIT with the patch applied, and ran the reproducer in an infinite while loop for some time in bash. No crash.
Please try with https://github.com/devdanzin/lafleur/ installed as editable, with no other packages installed. Seems to do the trick here. Sorry for the bad reproducing steps.
OK, here's a reproducer that mimics the effects of having a package installed as editable. Please let me know whether it works.
import sys from importlib.machinery import all_suffixes as module_suffixes # Not sure why, but this unused import seems necessary from importlib.util import spec_from_file_location class _EditableFinder: @classmethod def _find_spec(cls, fullname, candidate_path): return spec_from_file_location(fullname, candidate_path.with_suffix("pyc")) sys.meta_path.append(_EditableFinder) from random import random class C_object_v1: def __getitem__(self, item): return 5 object_v1 = C_object_v1() int_v4 = -51452741571812374 float_v6 = 46.1 float_v7 = 77.26 class StatefulLen_stateful_len_object_v12: def __init__(self): self.len_count = 0 def __len__(self): _len = 0 if self.len_count < 70 else 99 self.len_count += 1 return _len stateful_len_object_v12 = StatefulLen_stateful_len_object_v12() def uop_harness_f1(): class StatefulIter_uop_harness_f1_6059: def __init__(self): self._iter_count = 0 self._iterable = {1, 2, 3} def __iter__(self): self._iter_count += 1 if self._iter_count <= 67: return iter((None,)) return iter(self._iterable) uop_harness_f1_6059 = StatefulIter_uop_harness_f1_6059() evil_iterable = StatefulIter_uop_harness_f1_6059() for _ in range(100): tuple(evil_iterable) object_v1 = globals()['object_v1'] class SwappedClass_2950: pass object_v1.__class__ = SwappedClass_2950 if object_v1: pass for i_loop_9595 in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] * 50: res_compareopint = int_v4 > int_v4 if random() < 0.01: object_v1.__class__.get_value = lambda *a, **kw: 'patched!' for i_f1 in range(500): try: uop_harness_f1() except Exception: break
Reproduced. Thanks!
I'm making some progress on reducing. Here's what I've got so far:
import gc class C1: pass O = C1() gc.set_threshold(1) for i in range(326): for j in []: pass class C2: pass O.__class__ = C2 if O: pass
I'm working now on bringing the JIT thresholds back up.
Reacted by devdanzin and Ken JinBTW the imports were just to get the GC in a ready-to-crash state.
gc.set_threshold(1)is usually my go-to for removing that factor.Reacted by devdanzinHigher JIT thresholds aren't reproducing. Highest I can get is cranking up the number of loops from 326 to 12868 to reproduce with thresholds of 127 instead of 63.
Thanks Brandt for the repro. Based on debugging this, I speculate the following is happening:
The code which crashes iscase _EXIT_TRACE: { .... if (exit->executor && !exit->executor->vm_data.valid) { exit->temperature = initial_temperature_backoff_counter(); _PyFrame_SetStackPointer(frame, stack_pointer); Py_CLEAR(exit->executor); // THIS CRASHES stack_pointer = _PyFrame_GetStackPointer(frame); }This causes
assert(executor->vm_data.valid);to fail. The assert is clearly wrong in this case as we only free the exit trace executor when the data is no longer valid.I think the correct thing is just to remove the assertion. WDYT Brandt?
@devdanzin can you still repro this on main? Mark recently changed the
_EXIT_TRACEmechanism.I cannot reproduce on main anymore. I can still reproduce on 3.14. However, since the JIT is experimental, I'm reluctant to backport any changes to 3.14, especially when it deals with executor lifetimes.
The fix should be #136411
@devdanzin can you still repro this on main? Mark recently changed the
_EXIT_TRACEmechanism.It repros for me in main if we remove the gc tuning from Brandt's MRE:
class C1: pass O = C1() for i in range(326): for j in []: pass class C2: pass O.__class__ = C2 if O: pass
Does this abort for you too?
@devdanzin I can't reproduce the segfault on main.
What build flags are you using?
I cannot reproduce the segfault either, just the abort. On another machine, it's necessary to add the GC tuning back to get the abort:
import gc gc.set_threshold(1) class C1: pass O = C1() for i in range(326): for j in []: pass class C2: pass O.__class__ = C2 if O: pass
I'm using the simplest config flags to get a debug JIT build:
./configure --with-pydebug --enable-experimental-jit && make -j 14Reproduced without the GC tuning on my machine.
Reacted by devdanzinThe failing assertion is in
unlink_executor'sassert(executor->vm_data.valid);. I think the assertion is just wrong. I don't see how an executor that is being unlinked should be guaranteed to be valid. Here's what happens:case _START_EXECUTOR: { PyObject *executor = (PyObject *)CURRENT_OPERAND0(); #ifndef _Py_JIT assert(current_executor == (_PyExecutorObject*)executor); #endif assert(tstate->jit_exit == NULL || tstate->jit_exit->executor == current_executor); tstate->current_executor = (PyObject *)executor; if (!current_executor->vm_data.valid) { assert(tstate->jit_exit->executor == current_executor); assert(tstate->current_executor == executor); _PyFrame_SetStackPointer(frame, stack_pointer); _PyExecutor_ClearExit(tstate->jit_exit); <------------ this gets called stack_pointer = _PyFrame_GetStackPointer(frame); if (true) { UOP_STAT_INC(uopcode, miss); JUMP_TO_JUMP_TARGET(); } } break; }If you look above, the branch taken is in
!current_executor->vm_data.valid. That means the current executor is not valid, and we clear it. Yet when we clear it, we assert it's valid??? Seems like a buggy assertion. @markshannon WDYT?
Crash report
What happened?
Sorry about the messy and convoluted bug report, I hope something interesting might be gleaned from it.
It's possible to non-deterministically abort a special debug JIT build by running the code below, running from a Python executable from a venv where you installed a package in editable mode. This darn requirement for an install in editable mode stumped me for hours!
For this special build, apply this diff (couldn't get it to work in a normal build by changing loop lengths):
This MRE must be invoked like this:
Sorry about the long MRE, this code seems stubbornly hard to reduce. MRE:
Backtrace:
Here are the outputs from running with
PYTHON_OPT_DEBUG=4andPYTHON_LLTRACE=4, since they are too long I've attached them as files:opt_debug2.txt
lltrace2.txt
Found using lafleur.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.15.0a0 (heads/main-dirty:6bf1c0ab349, Jul 21 2025, 20:28:08) [GCC 14.2.0]