Single-threaded re-entrancy via a recursive generator causes an access violation (segfault) in pairwise_next

@rhettinger

Crash report

What happened?

The existing partial fix in main (the it = po->it; if (it == NULL) re-read guard after the first tp_iternext call) does not cover the second tp_iternext call, where it is still a borrowed reference. In the single-threaded case a re-entrant call can trigger Py_CLEAR(po->it) which drops the generator's refcount to zero and frees it, leaving the outer call's local it as a dangling pointer.

Minimal reproducer:

from itertools import pairwise

def g():
    yield next(it)

for _ in range(10000):
    try:
        it = pairwise(g())
        next(it)
    except ValueError:
        pass

Exception ignored in: <generator object g at 0x...>
ValueError: generator already executing
Windows fatal exception: access violation

This is also the root cause of spyder-ide/qtconsole#635.

Some options to address:

Py_INCREF(it) before the second tp_iternext(it) call + Py_DECREF after, or
a re-entrancy guard flag (like teedataobject.running) as suggested by @rhettinger.

Note: Py_BEGIN_CRITICAL_SECTION (added in PR #144489) does not protect against same-thread re-entrancy.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Output from running 'python -VV' on the command line:

No response

Linked PRs

gh-149557: Make itertools.pairwise safe under re-entry and concurrency #150589

@rhettinger

A reproducer for batched:

from itertools import batched, islice


def batched_gen():
    try:
        next(it)   # re-enters batched_next; inner call clears bo->it (frees it)
    except Exception:
        print("caught in batched_gen", flush=True)
        pass
    yield 1        # outer iternext(it) at i=0 returns 1 — loop continues
    yield 2        # outer iternext(it) at i=1: 'it' is freed → crash

print("Testing batched ...", flush=True)
for ii in range(10_000):
    try:
        it = batched(batched_gen(), 2)
        print('--')
        next(it)
        print('--')
    except (ValueError, StopIteration):
        print(ii)
        pass
print("done")

The islice is vulnerable as well.

Ping @rhettinger is code owner for itertools.

Let's make all of the itertools (except for repeat behave like generators and raise an exception if called while running.

pythongh-149557: Raise RuntimeError on re-entrant itertools.pairwise.…

Concurrent use of count is also common, because it is atomic. This is an idiomatic way to generate unique consequent integers.

Many of other generators, like product or combinations can also be used concurrently to distribute work between threads. If we make filterfalse, starmap or zip_longest non-reentrant, we should do this for filter, map and zip too. I afraid this can break user code.

If we implemented these iterators in Python, there would not be leaks or crashes. Therefore, it is possible to implement them without leaks and crashes.

pythongh-149557, pythongh-123471: Make itertools.pairwise safe under …

Navigation Menu

Repository navigation

Single-threaded re-entrancy via a recursive generator causes an access violation (segfault) in `pairwise_next` #149557

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Single-threaded re-entrancy via a recursive generator causes an access violation (segfault) in pairwise_next #149557

Description

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Activity

eendebakpt commented on May 8, 2026

rhettinger commented on May 19, 2026

serhiy-storchaka commented on May 29, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Single-threaded re-entrancy via a recursive generator causes an access violation (segfault) in `pairwise_next` #149557