Repository navigation
Segfault when cleaning up after a MemoryError with deeply nested objects #149146
Description
Activity
Bisect identified this as the change that introduced the segfault: @markshannon
147cd05 is the first bad commit
commit 147cd05
Author: Mark Shannon
Date: Wed Apr 17 11:08:05 2024 +0100GH-117760: Streamline the trashcan mechanism (GH-117763)Include/cpython/object.h | 47 ++++++++--------
Include/cpython/pystate.h | 7 +--
Include/internal/pycore_object.h | 1 -
Objects/object.c | 114 ++++++---------------------------------
Python/pystate.c | 2 +
5 files changed, 43 insertions(+), 128 deletions(-)
bisect found first bad commitHi — I'd like to take a look at this. Reproducing locally and reading through the tuple_dealloc path from the bisect. Will open a PR if I get somewhere.
@vstinner - Thank you! I've actually switched away from musl since this report, due to some memory issues, so that's good news.
My quick tests indicates this is still reproducible wthout musl:
echo 2 > /proc/sys/vm/overcommit_memory ~ # ~ # /pfm/py/bin/python3 /pfm/scratch/dist/gh149146.py [ 487.622390] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation [ 487.622514] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation [ 487.622595] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation [ 487.622646] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation [ 487.622759] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation [ 487.622840] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation [ 487.622878] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation [ 487.622936] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation [ 487.622987] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation [ 487.623022] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation Exception ignored in the internal traceback machinery: Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 1340, in _find_and_load File "<frozen importlib._bootstrap>", line 1290, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 1216, in _find_spec File "<frozen importlib._bootstrap_external>", line 1270, in find_spec File "<frozen importlib._bootstrap_external>", line 1244, in _get_spec File "<frozen importlib._bootstrap_external>", line 1364, in find_spec File "<frozen importlib._bootstrap_external>", line 1407, in _fill_cache OSError: [Errno 12] Cannot allocate memory: '/pfm/scratch/dist' Traceback (most recent call last): File "/pfm/scratch/dist/gh149146.py", line 3, in <module> object address : 0xfffff7c27700 object refcount : 3 object type : 0xaaaaab78d7b0 object type name: MemoryError object repr : MemoryError() lost sys.stderr Segmentation fault ~ # /pfm/py/bin/python3 --version Python 3.16.0a0 ~ # ldd /pfm/py/bin/python3 linux-vdso.so.1 (0x0000fffff7fff000) libm.so.6 => /usr/lib/libm.so.6 (0x0000fffff6e30000) libc.so.6 => /usr/lib/libc.so.6 (0x0000fffff6c60000) /lib/ld-linux-aarch64.so.1 => /usr/lib/ld-linux-aarch64.so.1 (0x0000fffff7fb0000)that was with:
commit a064b323f4350305e7486c7b1090cf12b19e7738 (grafted, HEAD -> main, origin/main) Author: Victor Stinner <vstinner@python.org> Date: Wed Jun 17 10:58:22 2026 +0200 gh-151496: Use process groups in test_dtrace (#151512) ...I'll run again with latest main to verify
Yea, it looks to still be there:
~ # echo 2 > /proc/sys/vm/overcommit_memory ~ # lldb /pfm/py/bin/python3 /pfm/scratch/gh149146.py [ 28.116418] random: crng init done (lldb) target create "/pfm/py/bin/python3" Current executable set to '/pfm/py/bin/python3' (aarch64). (lldb) settings set -- target.run-args "/pfm/scratch/gh149146.py" (lldb) run Process 90 launched: '/pfm/py/bin/python3' (aarch64) Exception ignored in the internal traceback machinery: Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 1340, in _find_and_load File "<frozen importlib._bootstrap>", line 1290, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 1216, in _find_spec File "<frozen importlib._bootstrap_external>", line 1270, in find_spec File "<frozen importlib._bootstrap_external>", line 1244, in _get_spec File "<frozen importlib._bootstrap_external>", line 1364, in find_spec File "<frozen importlib._bootstrap_external>", line 1407, in _fill_cache OSError: [Errno 12] Cannot allocate memory: '/pfm/scratch' Traceback (most recent call last): File "/pfm/scratch/gh149146.py", line 3, in <module> object address : 0xfffff7c27700 object refcount : 3 object type : 0xaaaaab78d870 object type name: MemoryError object repr : MemoryError() lost sys.stderr Process 90 stopped * thread #1, name = 'python3', stop reason = signal SIGSEGV: address not mapped to object (fault address=0xfffffffddff0) frame #0: 0x0000aaaaaafebc58 python3`_Py_Dealloc(op=0x0000aaaaabdeac90) at object.c:3291 3288 stack is shallower */ 3289 void 3290 _Py_Dealloc(PyObject *op) -> 3291 { 3292 PyTypeObject *type = Py_TYPE(op); 3293 unsigned long gc_flag = type->tp_flags & Py_TPFLAGS_HAVE_GC; 3294 destructor dealloc = type->tp_dealloc; ~ # /pfm/py/bin/python3 -VV Python 3.16.0a0 (heads/main-dirty:12add38, Jun 18 2026, 11:18:38) [Clang 22.1.6 ]Ok, I reproduced the issue on Fedora 44 (using glibc). I limited the virtual memory to ~1 GiB:
$ ulimit -v 1024000 $ gdb -args ./python b.py (gdb) rungdb traceback:
#0 0x00000000005717f6 in _Py_RecursionLimit_GetMargin () #1 0x0000000000577dd0 in _Py_Dealloc () #2 0x00000000005ae871 in Py_DECREF () #3 0x00000000005ae8a1 in Py_XDECREF () #4 0x00000000005aff6c in tuple_dealloc () #5 0x0000000000577ed5 in _Py_Dealloc () #6 0x00000000005ae871 in Py_DECREF () #7 0x00000000005ae8a1 in Py_XDECREF () #8 0x00000000005aff6c in tuple_dealloc () #9 0x0000000000577ed5 in _Py_Dealloc () #10 0x00000000005ae871 in Py_DECREF () #11 0x00000000005ae8a1 in Py_XDECREF () #12 0x00000000005aff6c in tuple_dealloc () #13 0x0000000000577ed5 in _Py_Dealloc () #14 0x00000000005ae871 in Py_DECREF () #15 0x00000000005ae8a1 in Py_XDECREF () #16 0x00000000005aff6c in tuple_dealloc () #17 0x0000000000577ed5 in _Py_Dealloc () #18 0x00000000005ae871 in Py_DECREF () #19 0x00000000005ae8a1 in Py_XDECREF () #20 0x00000000005aff6c in tuple_dealloc () #21 0x0000000000577ed5 in _Py_Dealloc () #22 0x00000000005ae871 in Py_DECREF () ...Stack of the process:
$ grep stack /proc/1285237/maps 7ffffffc8000-7ffffffff000 rw-p 00000000 00:00 0 [stack]That's only 220 kiB, it's quite small! Glibc uses a stack size of 8 MiB and the stack is allocated on demand. Here, the stack cannot be allocated since we exhausted all the memory.
There isn't much that Python can do in this case: when the memory is exhausted, Python fails to allocate more stack and so it does crash with a stack overflow.
Another reproducer which limits the process memory to 1 GiB to not crash the whole system:
import resource # deallocation the chain uses a lot of C stack memory CHAIN_LENGTH = 1024 * 1024 # limit process memory to not crash the whole machine max_memory = 1024 * 1024 * 1024 resource.setrlimit(resource.RLIMIT_AS, (max_memory, max_memory)) print("create long chain...") long_chain = None for _ in range(CHAIN_LENGTH): long_chain = (long_chain, None) print("fill the process memory...") memory = [] try: large = 1024 * 1024 while True: memory.append(b'x' * large) except MemoryError: pass small = 1024 while True: memory.append(b'x' * small) # at exit (on error), deallocating long_chain will use a lot of C stack memory # ... and does crash!
So the problem is that the system/process memory is exhausted, and so when the Linux kernel tries to extend the process stack memory, it fails with a segfault since there is no more available memory.
There is nothing Python can do in this case. There is no generic and safe mechanism to catch the segfault and report it as a regular Python exception. Catching SIGSEGV signal is dangerous.
I close the issue as "won't fix".
Reacted by Steve Stagg
Crash report
What happened?
When in memory-constrained situations, the following code generates a segfault on main:
Clearly, b grows to be an unbounded nested tuple, which eventually exhausts the memory.
A MemoryError is correctly thrown, but during unwind, python tries to clean up the object, and this triggers a stack overflow.
Example stack
The following dockerfile reliably reproduces it for me:
Dockerfile
Gives me:
ISTR that stack overflows from recursive objects were at least considered low priority, if not 'not worth fixing' before, but I can't recall where I read that, it was a while ago, so apologies if this isn't a useful report!
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.15.0a8+ (heads/main:d4eee16, Apr 29 2026, 11:03:26) [GCC 15.2.0]
Linked PRs