Free-threading QSBR delayed free mem fails in high thread turnover environment

Crash report

What happened?

When cycling a lot of threads and modifying objects which use PyMem_FreeDelayed() in those threads the queues of memory blocks to free build up without ever being freed until the process runs out of memory, unless something else causes them to be freed. For example in the reproducer below commenting in the gc.collect() or the for loop will cause the delayed blocks to be freed correctly. Note that this is an extreme case.

Reproducer:

import gc
import threading


def fupmem(b, l):
    b.wait()
    l *= 2


def check(funcs, *args):
    barrier = threading.Barrier(len(funcs))
    thrds = []

    for func in funcs:
        thrd = threading.Thread(target=func, args=(barrier, *args))

        thrds.append(thrd)
        thrd.start()

    for thrd in thrds:
        thrd.join()


if __name__ == "__main__":
    count = 0

    while True:
        print(count := count + 1)

        check([fupmem] * 10, [None] * 256)

        # gc.collect()

        # for i in range(1000):
        #     l = []
        #     del l

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a5+ experimental free-threading build (heads/main:b3c18bfd828, Mar 3 2025, 09:13:46) [GCC 11.4.0]

Linked PRs

@colesbury

ping @colesbury

Thanks for the bug report. Typically, each thread periodically processes their queue of memory blocks to be freed. This happens every 254 calls to _PyMem_FreeDelayed(). The problems is that:

Each thread only makes one call to _PyMem_FreeDelayed(), so they don't call _PyMem_ProcessDelayed.
The thread then exits and "abandons" it's queue to the shared per-interpreter queue
No thread processes the shared per-interpreter queue, because that happens in _PyMem_ProcessDelayed

The fix is relatively straightforward: when the thread exits and merges its queue into the shared per-interpreter queue, it should also take the opportunity to process the shared queue.

pythongh-130794: Process interpreter QSBR queue in _PyMem_AbandonDela…

Merge branch 'main' into pythongh-130794-process-queue

gh-130794: Process interpreter QSBR queue in _PyMem_AbandonDelayed. (g…

[3.13] pythongh-130794: Process interpreter QSBR queue in _PyMem_Aban…

[3.13] gh-130794: Process interpreter QSBR queue in _PyMem_AbandonDel…

Closing as it was fixed by #130808 and backported.

Navigation Menu

Repository navigation

Free-threading QSBR delayed free mem fails in high thread turnover environment #130794

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Free-threading QSBR delayed free mem fails in high thread turnover environment #130794

Description

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Activity

tom-pytel commented on Mar 3, 2025

colesbury commented on Mar 3, 2025

kumaraditya303 commented on Mar 11, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions