Skip to content

functools.lru_cache critical section locks self but _LockHeld dict APIs assert self->cache is locked #148180

Open
Open
functools.lru_cache critical section locks self but _LockHeld dict APIs assert self->cache is locked#148180
Labels
extension-modulesC modules in the Modules dirtopic-free-threadingtype-crashA hard crash of the interpreter, possibly with a core dump

Description

@devdanzin
devdanzin
opened on Apr 6, 2026

Crash report

What happened?

On a debug free-threaded build, functools.lru_cache crashes with a critical section assertion failure when gc.get_objects() is called between cache uses.

bounded_lru_cache_wrapper (line 1498) acquires Py_BEGIN_CRITICAL_SECTION(self), locking the lru_cache object's mutex. It then calls bounded_lru_cache_get_lock_held (line 1325) which calls _PyDict_GetItemRef_KnownHash_LockHeld on self->cache. The _LockHeld variant asserts that the dict's per-object mutex is held (_Py_CRITICAL_SECTION_ASSERT_OBJECT_LOCKED(mp) at dictobject.c:1254), but only self's mutex is held — self and self->cache have different mutexes.

In 3.15 we have this abort message:

Assertion "(cs != ((void*)0) && cs->_cs_mutex == mutex)" failed: Critical section of object is not held

This was introduced in gh-131757 (PR #131758), which split the critical section to allow the cached function to execute concurrently. The same pattern was extended in gh-132641 (PR #133787) which added more _LockHeld calls following the same assumption.

On non-debug builds the assertion is compiled out and the code happens to work in practice (no actual contention in the reproducer), but the lock contract is violated.

Reproducer

Requires a debug free-threaded build (e.g., --with-pydebug --disable-gil).

from functools import lru_cache
import gc

@lru_cache(maxsize=32)
def get_pep(num):
    return type('', (), {})()

for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
    pep = get_pep(n)
    print(n, pep)

objs = gc.get_objects()
print(dir(get_pep))

for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
    pep = get_pep(n)
    print(n, pep)

Backtrace on 3.14 (trimmed):

python: ./Include/internal/pycore_critical_section.h:226: void _PyCriticalSection_AssertHeld(PyMutex *): Assertion `cs != NULL && cs->_cs_mutex == mutex' failed.
[...]
#5  _PyCriticalSection_AssertHeld (mutex=...) at pycore_critical_section.h:226
#6  _Py_dict_lookup (mp=...) at Objects/dictobject.c:1254
#7  _PyDict_GetItemRef_KnownHash_LockHeld (...) at Objects/dictobject.c:2420
#8  bounded_lru_cache_get_lock_held (...) at Modules/_functoolsmodule.c:1325
#9  bounded_lru_cache_wrapper (...) at Modules/_functoolsmodule.c:1499

Backtrace on 3.15:

./Include/internal/pycore_critical_section.h:249: _PyCriticalSection_AssertHeldObj: Assertion "(cs != ((void*)0) && cs->_cs_mutex == mutex)" failed: Critical section of object is not held
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7bffb670ccd0
object refcount : 2
object type     : 0x555556749840
object type name: dict
object repr     : {8: <functools._lru_list_elem object at 0x7bffb65701d0>, 290: <functools._lru_list_elem object at 0x7bffb6570240>, 308: <functools._lru_list_elem object at 0x7bffb65702b0>, 320: <functools._lru_list_elem object at 0x7bffb6570320>, 218: <functools._lru_list_elem object at 0x7bffb6570390>, 279: <functools._lru_list_elem object at 0x7bffb6570400>, 289: <functools._lru_list_elem object at 0x7bffb6570470>, 9991: <functools._lru_list_elem object at 0x7bffb65704e0>}

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Stack (most recent call first):
  File "/home/danzin/crashers/yifei_01.py", line 15 in <module>

Program received signal SIGABRT, Aborted.
Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
__pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=0) at ./nptl/pthread_kill.c:44
warning: 44     ./nptl/pthread_kill.c: No such file or directory
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=0) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (threadid=<optimized out>, signo=6) at ./nptl/pthread_kill.c:89
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:100
#3  0x00007ffff7c45e2e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c28888 in __GI_abort () at ./stdlib/abort.c:77
#5  0x000055555606dd54 in fatal_error_exit (status=status@entry=-1) at Python/pylifecycle.c:3389
#6  0x000055555606da1b in fatal_error (fd=fd@entry=2, header=header@entry=1, prefix=prefix@entry=0x555556405360 <__func__._PyObject_AssertFailed> "_PyObject_AssertFailed",
    msg=msg@entry=0x555556405360 <__func__._PyObject_AssertFailed> "_PyObject_AssertFailed", status=status@entry=-1) at Python/pylifecycle.c:3612
#7  0x0000555556069730 in _Py_FatalErrorFunc (func=0x555556405360 <__func__._PyObject_AssertFailed> "_PyObject_AssertFailed",
    msg=0x555556405360 <__func__._PyObject_AssertFailed> "_PyObject_AssertFailed") at Python/pylifecycle.c:3635
#8  0x00005555557fc43b in _PyObject_AssertFailed (obj=obj@entry=0x7bffb670ccd0, expr=<optimized out>, msg=<optimized out>, file=<optimized out>, line=line@entry=249,
    function=<optimized out>) at Objects/object.c:3259
#9  0x0000555555bb7711 in _PyCriticalSection_AssertHeldObj (op=0x7bffb670ccd0) at ./Include/internal/pycore_critical_section.h:247
#10 _Py_dict_lookup (mp=0x7bffb670ccd0, key=key@entry=0x55555693c4f8 <_PyRuntime+30776>, hash=hash@entry=8, value_addr=value_addr@entry=0x7fffffff9d00) at Objects/dictobject.c:1305
#11 0x0000555555bbbc61 in _PyDict_GetItemRef_KnownHash_LockHeld (op=0x196f39, key=0x196f39, key@entry=0x55555693c4f8 <_PyRuntime+30776>, hash=6, hash@entry=8,
    result=result@entry=0x7fffffff9e80) at Objects/dictobject.c:2480
#12 0x00005555562ed221 in bounded_lru_cache_get_lock_held (self=0x7bffb6826c90, args=0x7bffb69bb290, kwds=0x0, result=<optimized out>, key=<optimized out>, hash=<optimized out>)
    at ./Modules/_functoolsmodule.c:1416
#13 bounded_lru_cache_wrapper (self=0x7bffb6826c90, args=0x7bffb69bb290, kwds=0x0) at ./Modules/_functoolsmodule.c:1590
#14 0x0000555555ac6b5d in _PyObject_MakeTpCall (tstate=0x555556997d60 <_PyRuntime+405664>, callable=0x7bffb6826c90, args=0x7fffffffa048, nargs=1, keywords=0x0) at Objects/call.c:242
#15 0x0000555555ea9e3b in _Py_VectorCallInstrumentation_StackRefSteal (callable=..., arguments=arguments@entry=0x7e8ff6fe0290, total_args=total_args@entry=1, kwnames=kwnames@entry=...,
    call_instrumentation=false, frame=frame@entry=0x7e8ff6fe0220, this_instr=0x7bffb6405394, tstate=0x555556997d60 <_PyRuntime+405664>) at Python/ceval.c:775
#16 0x0000555555ee754d in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:1841
#17 0x0000555555ea8d04 in _PyEval_EvalFrame (tstate=0x555556997d60 <_PyRuntime+405664>, frame=0x7e8ff6fe0220, throwflag=0) at ./Include/internal/pycore_ceval.h:118
#18 _PyEval_Vector (tstate=tstate@entry=0x555556997d60 <_PyRuntime+405664>, func=func@entry=0x7bffb64c3e30, locals=locals@entry=0x7bffb670ca90, args=args@entry=0x0,
    argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:2179
#19 0x0000555555ea80db in PyEval_EvalCode (co=co@entry=0x7bffb6405210, globals=globals@entry=0x7bffb670ca90, locals=locals@entry=0x7bffb670ca90) at Python/ceval.c:686
#20 0x00005555560c6ca0 in run_eval_code_obj (tstate=tstate@entry=0x555556997d60 <_PyRuntime+405664>, co=co@entry=0x7bffb6405210, globals=0x7bffb670ca90, locals=locals@entry=0x7bffb670ca90)
    at Python/pythonrun.c:1369

Possible fix

Either lock both objects:

// Line 1498, 1511: lock both self and self->cache
Py_BEGIN_CRITICAL_SECTION2(self, self->cache);

Or use the regular (self-locking) dict APIs instead of _LockHeld variants for the get path.

The initial crash and reproducer were found by @zhuyifei1999. Root cause analysis and issue draft done with assistance from Claude Code.

CPython versions tested on:

3.15, CPython main branch, 3.14

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.15.0a7+ free-threading build (heads/main:4d0e8ee649c, Mar 29 2026, 23:20:29) [Clang 21.1.2 (2ubuntu6)]

Activity

  1. devdanzin
    added
    extension-modulesC modules in the Modules dir
    type-crashA hard crash of the interpreter, possibly with a core dump
    topic-free-threading
    on Apr 6, 2026

  2. colesbury commented on Apr 6, 2026

    @colesbury
    colesbury
    on Apr 6, 2026
    Contributor

    I don't think we should fuzz for gc.get_objects or gc.get_referrers bugs.

  3. devdanzin commented on Apr 6, 2026

    @devdanzin
    devdanzin
    on Apr 6, 2026
    MemberAuthor

    Yes, I avoid doing that.

    This was found when trying to implement free threading in a C extension, the code looked wrong, but the only way we found to trigger it is with gc.get_objects or gc.get_referrers.

    Should I close this, since the bug isn't triggerable from other paths?

  4. colesbury commented on Apr 6, 2026

    @colesbury
    colesbury
    on Apr 6, 2026
    Contributor

    Ah, okay. I jumped to that conclusion.

    I'm not sure what we can do about it, but let's leave this open for now. I'm open to suggestions on how we should handle this.

  5. zhuyifei1999 commented on Apr 7, 2026

    @zhuyifei1999
    zhuyifei1999
    on Apr 7, 2026

    What's your thoughts on the Py_BEGIN_CRITICAL_SECTION2(self, self->cache); workaround suggested by the LLM? Nothing else should be locking the cache so it should be painless I think?

    For context, I hit this during zhuyifei1999/guppy3#52 (comment). This is a heap-traversing memory debugging tool I maintain, that answers the question "what objects are there and why are they not being freed". Because I need to track objects in order to look at them more closely, the library obtains refcounts to nearly everything in the heap after a heap traversal, which includes this functools.lru_cache->cache dict.

    And because I had a refcounting bug in my library while porting to freethreading, I popped open a Py_DEBUG build to find what the refcounting bug was, only for Python to hit this bug and crash.

  6. robertsdotpm commented on Jun 28, 2026

    @robertsdotpm
    robertsdotpm
    on Jun 28, 2026

    I hit what looks like the same root cause from an ordinary, gc-free path, which may be relevant to the "should we close this since it's only triggerable via gc.get_objects/get_referrers?" question above.

    Concurrent use of a single functools.lru_cache on a free-threaded build can make the cache exceed its maxsize — no gc introspection involved, just normal lookups under eviction churn. The over-count is a real, persistent structural state (read quiescently after all threads join via cache_info().currsize, which is PyDict_GET_SIZE(self->cache)), not a transient counter skew.

    Reproducer (pure stdlib, no gc calls) -- on Mac / ARM64

    import functools, threading, time

def trial(nthreads=24, seconds=5, maxsize=128, keyspace=8192):
    @functools.lru_cache(maxsize=maxsize)
    def f(k):
        return k * k
    stop = False
    def hammer(seed):
        x = seed | 1                                  # per-thread PRNG, no shared state
        while not stop:
            x = (x * 1103515245 + 12345) & 0x7fffffff
            f(x % keyspace)                           # working set >> maxsize => constant eviction
    ts = [threading.Thread(target=hammer, args=(i,)) for i in range(nthreads)]
    for t in ts: t.start()
    time.sleep(seconds)
    stop = True
    for t in ts: t.join()
    return f.cache_info()                             # quiescent read; currsize == PyDict_GET_SIZE

for i in range(40):
    info = trial()
    print(i, info)
    if info.currsize > 128:
        print("OVER maxsize:", info)
        break

    On a free-threaded arm64 build this prints e.g. CacheInfo(hits=..., misses=..., maxsize=128, currsize=129) within a few trials.

    Platform / GIL matrix (Python 3.13.13, same machine)

    build result
    arm64, free-threaded (PYTHON_GIL=0) ~3/40 trials reach currsize=129 (maxsize=128)
    arm64, GIL re-enabled (PYTHON_GIL=1) 0/40
    x86-64, free-threaded 0/40 (possibly masked by stronger memory ordering; I haven't confirmed the race is absent vs. merely hidden)

    So it's a free-threading bug (the GIL-on control is clean), reproducible without any heap-introspection API.

    Where it surfaces

    In a stock build instrumented only with an fprintf after each lru_cache_append_link (no --with-pydebug, to avoid perturbing timing), the size first passes maxsize at the evict-path append in bounded_lru_cache_wrapper (Modules/_functoolsmodule.c, ~L1205 in 3.13.13); a "grow branch taken while already full" probe never fired. That's consistent with two wrapper invocations interleaving while self's critical section is suspended (e.g. across the PyObject_Call to the user function), making the size-check-then-insert non-atomic — i.e. the same "wrapper guards self, not self->cache" issue described here, but observable as a maxsize violation rather than only the _LockHeld assertion.

    I'm not certain whether this is identical to the assertion failure above or a sibling of it (and whether the Py_BEGIN_CRITICAL_SECTION2(self, self->cache) suggestion would close this particular TOCTOU) — you'd know far better than I would. Posting it mainly as evidence that the defect is reachable from normal concurrent use with a user-visible consequence.

  7. robertsdotpm
    added a commit that references this issue on Jul 16, 2026

    big_100: skip p303 lru_cache over-grow test (upstream CPython gh-148180)

    1903957
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    extension-modulesC modules in the Modules dirtopic-free-threadingtype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions