Repository navigation
itertools._grouper.__next__ has the same re-entrant use-after-free that was fixed in groupby.__next__ #146613
Description
Crash report
What happened?
AI Disclosure: this issue, including the reproducer code, has been drafted by an LLM.
Bug description
gh-143543 / commit a91b5c3 fixed a re-entrant use-after-free in groupby_next by snapshotting gbo->tgtkey and gbo->currkey with Py_INCREF before calling PyObject_RichCompareBool. However, the sibling function _grouper_next (the inner group iterator) has the exact same unprotected comparison at Modules/itertoolsmodule.c:681:
rcmp = PyObject_RichCompareBool(igo->tgtkey, gbo->currkey, Py_EQ);Neither igo->tgtkey nor gbo->currkey is held with a strong reference during the comparison. A user-defined __eq__ that re-enters the _grouper iterator can trigger groupby_step, which calls Py_XSETREF(gbo->currkey, newkey) — freeing the old currkey while PyObject_RichCompareBool still holds a dangling pointer to it as local variable w. When __eq__ returns NotImplemented, do_richcompare tries the reverse comparison w.__eq__(v) on the freed object.
Reproducer
Crashes with a segfault:
import itertools
grouper_iter = None
class Key:
__hash__ = None
def __init__(self, do_advance):
self.do_advance = do_advance
self.payload = bytearray(256)
def __eq__(self, other):
if self.do_advance:
self.do_advance = False
if grouper_iter is not None:
try:
next(grouper_iter)
except StopIteration:
pass
for _ in range(50):
bytearray(256)
return NotImplemented
return True
def keyfunc(element):
if element == 0:
return Key(do_advance=True)
return Key(do_advance=False)
g = itertools.groupby(range(4), keyfunc)
key, grouper_iter = next(g)
items = list(grouper_iter) # segfaultBacktrace
Program received signal SIGSEGV, Segmentation fault.
do_richcompare (tstate=0x555555d6cfd8 <_PyRuntime+405272>, v=0x20002542ba0, w=0x200025428a0, op=2) at Objects/object.c:1065
warning: Source file is more recent than executable.
1065 Py_DECREF(res);
#0 do_richcompare (tstate=0x555555d6cfd8 <_PyRuntime+405272>, v=0x20002542ba0, w=0x200025428a0, op=2) at Objects/object.c:1065
#1 PyObject_RichCompare (v=v@entry=0x20002542ba0, w=w@entry=0x200025428a0, op=op@entry=2) at Objects/object.c:1109
#2 0x000055555573deca in PyObject_RichCompareBool (v=0x20002542ba0, w=0x200025428a0, op=op@entry=2) at Objects/object.c:1131
#3 0x0000555555a32c03 in _grouper_next (op=0x20002151c80) at ./Modules/itertoolsmodule.c:681
#4 0x00005555556f2805 in list_extend_iter_lock_held (self=self@entry=0x200029bd4b0, iterable=iterable@entry=0x20002151c80) at Objects/listobject.c:1318
#5 0x00005555556ed858 in _list_extend (self=self@entry=0x200029bd4b0, iterable=iterable@entry=0x20002151c80) at Objects/listobject.c:1507
#6 0x00005555556f7ba8 in list___init___impl (self=self@entry=0x200029bd4b0, iterable=0x20002151c80) at Objects/listobject.c:3541
#7 0x00005555556f0c46 in list_vectorcall (type=type@entry=0x555555cd8ef8 <PyList_Type>, args=args@entry=0x7fffffffa6c8, nargsf=nargsf@entry=9223372036854775809, kwnames=kwnames@entry=0x0)
at Objects/listobject.c:3565
#8 0x000055555569f1de in _PyObject_VectorcallTstate (tstate=0x555555d6cfd8 <_PyRuntime+405272>, callable=0x555555cd8ef8 <PyList_Type>, args=0x7fffffffa6c8, nargsf=9223372036854775809,
kwnames=0x0) at ./Include/internal/pycore_call.h:136
#9 0x0000555555869fc1 in _Py_VectorCallInstrumentation_StackRefSteal (callable=..., arguments=0x555555da57d0, total_args=total_args@entry=1, kwnames=kwnames@entry=...,
call_instrumentation=false, frame=0x555555da5770, this_instr=0x200023e8f3a, tstate=0x555555d6cfd8 <_PyRuntime+405272>) at Python/ceval.c:770
#10 0x0000555555874b23 in _PyEval_EvalFrameDefault (tstate=0x555555d6cfd8 <_PyRuntime+405272>, frame=<optimized out>, throwflag=0) at Python/generated_cases.c.h:1842
#11 0x0000555555869779 in _PyEval_EvalFrame (tstate=0x555555d6cfd8 <_PyRuntime+405272>, frame=0x555555da5770, throwflag=0) at ./Include/internal/pycore_ceval.h:118
#12 _PyEval_Vector (tstate=tstate@entry=0x555555d6cfd8 <_PyRuntime+405272>, func=func@entry=0x200024c3e30, locals=locals@entry=0x2000270cbb0, args=args@entry=0x0,
argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:2133
#13 0x0000555555869292 in PyEval_EvalCode (co=co@entry=0x200023e8dd0, globals=globals@entry=0x2000270cbb0, locals=locals@entry=0x2000270cbb0) at Python/ceval.c:681
#14 0x0000555555977e7a in run_eval_code_obj (tstate=tstate@entry=0x555555d6cfd8 <_PyRuntime+405272>, co=co@entry=0x200023e8dd0, globals=globals@entry=0x2000270cbb0,
locals=locals@entry=0x2000270cbb0) at Python/pythonrun.c:1366
#15 0x0000555555977b39 in run_mod (mod=mod@entry=0x200027e6508, filename=filename@entry=0x20002235290, globals=globals@entry=0x2000270cbb0, locals=locals@entry=0x2000270cbb0,
flags=0x7fffffffb9f0, arena=arena@entry=0x200020508b0, interactive_src=0x0, generate_new_source=0) at Python/pythonrun.c:1469
Root cause
_grouper_next at line 681 passes igo->tgtkey and gbo->currkey to PyObject_RichCompareBool without incrementing their refcounts. PyObject_RichCompareBool → PyObject_RichCompare → do_richcompare does not INCREF the arguments. When __eq__ re-enters the _grouper iterator:
- Inner
_grouper_nextcallsgroupby_step groupby_stepcallsPy_XSETREF(gbo->currkey, newkey)— this decrefs the oldcurrkeyto refcount 0, freeing it__eq__returnsNotImplementeddo_richcomparetries the reverse:w.__eq__(v)wherewis the freedcurrkey— use-after-free
Suggested fix
Apply the same fix from groupby_next (commit a91b5c3) to _grouper_next:
// Before (vulnerable):
rcmp = PyObject_RichCompareBool(igo->tgtkey, gbo->currkey, Py_EQ);
// After (safe):
PyObject *tgtkey = igo->tgtkey;
PyObject *currkey = gbo->currkey;
Py_INCREF(tgtkey);
Py_INCREF(currkey);
int rcmp = PyObject_RichCompareBool(tgtkey, currkey, Py_EQ);
Py_DECREF(tgtkey);
Py_DECREF(currkey);Versions affected
Same as gh-143543 — all versions with itertools.groupby, regardless of GIL or JIT being enabled. The fix for groupby_next was applied to 3.13 and 3.14; this sibling function needs the same treatment, besides also hitting 3.15.
Found using cpython-review-toolkit.
CPython versions tested on:
CPython main branch, 3.15, 3.14
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.15.0a6+ free-threading build (heads/main:592e8f0865c, Mar 2 2026, 16:35:36) [Clang 21.1.2 (2ubuntu6)]
Activity
picnixz commentedon Mar 29, 2026
This is a dupe of #145678 I believe and we had issues with the reproducer... I'm going to close this one for now, but please check that the reproducer reproduces.
devdanzin commentedon Mar 29, 2026
Thank you, sorry for not checking first :(
I can confirm the reproducer here works across Python versions and build options on my machine and the one in #145678 doesn't.
picnixz commentedon Mar 29, 2026
Cool. So I'll keep yours open and close the other actually. That way, we can have a cleaner history.
pythongh-146613: Fix re-entrant use-after-free in itertools._grouper
TheSkyC commentedon Apr 1, 2026
Thanks for the excellent minimal reproducer! I've reproduced the crash on current
main.This is exactly the same re-entrant use-after-free that was fixed in
groupby.__next__(gh-143543 / a91b5c3), but it was missed in the sibling_grouper.__next__.I've prepared the patch (temporary
Py_INCREF/Py_DECREFprotection on the borrowed keys) and regression test intest_itertools.py.I'll open the PR shortly
Merge branch 'main' into pythongh-146613-itertools-uaf
gh-146613: Fix re-entrant use-after-free in `itertools._grouper` (#14…
pythongh-146613: Fix re-entrant use-after-free in `itertools._grouper` (
pythongh-146613: Fix re-entrant use-after-free in `itertools._grouper` (
[3.14] gh-146613: Fix re-entrant use-after-free in `itertools._groupe…
[3.13] gh-146613: Fix re-entrant use-after-free in `itertools._groupe…
pythongh-146613: Fix re-entrant use-after-free in `itertools._grouper` (