Repository navigation
Frame teardown can create frame objects #99729
Description
Activity
see also bloomberg/memray#260 cc @pablogsal
This looks like another frame lifetime issue. Here is the stack:
Thread 1 "python" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff7cc5740 (LWP 223005)] 0x00005555556cbc8b in PyFrame_GetCode (frame=<error reading variable: Cannot access memory at address 0x7ffff5e4c210>) at Objects/frameobject.c:1303 1303 Objects/frameobject.c: Directory not empty. (gdb) up #1 frame_getcode (f=<error reading variable: Cannot access memory at address 0x7ffff5e4c210>, closure=<optimized out>) at Objects/frameobject.c:97 97 in Objects/frameobject.c (gdb) #2 0x0000555555702309 in _PyObject_GenericGetAttrWithDict (Python Exception <class 'gdb.error'>: There is no member named f_frame. obj=, name='f_code', dict=0x0, suppress=0) at ./Include/object.h:133 133 ./Include/object.h: Directory not empty. (gdb) #3 0x0000555555701836 in PyObject_GetAttr (Python Exception <class 'gdb.error'>: There is no member named f_frame. v=v@entry=, name='f_code') at Objects/object.c:916 916 Objects/object.c: Directory not empty. (gdb) #4 0x00005555556440a1 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3466 3466 Python/ceval.c: Directory not empty. (gdb) #5 0x00005555556c1bd5 in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff4840330, tstate=0x555555adfa18 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 73 ./Include/internal/pycore_ceval.h: Directory not empty. (gdb) #6 gen_send_ex2 (closing=0, exc=0, presult=<synthetic pointer>, arg=0x0, gen=0x7ffff48402e0) at Objects/genobject.c:219 219 Objects/genobject.c: Directory not empty. (gdb) #7 gen_iternext (gen=0x7ffff48402e0) at Objects/genobject.c:594I can confirm that still happens with 3.11 current.
Thanks, I'll look at this today (Mark's out right now).
I've now eliminated pytest from my reproducer: graingert/segfault-repro@34b5962
Reacted by Pradyun Gedam, Brandt Bucher, Thomas Grainger and Matt WozniskiI've now eliminated pytest from my reproducer: graingert/segfault-repro@34b5962
Oh thank God. You have no idea how helpful this is.
@graingert, how should I run the new reproducer? I'm unable to get 3.11 head to crash with
pip install -r requirements.txt; python test_ssltransport.py.I see the segfault intermittently when running
pytest, though.In the meantime, I'm using
pytestto reproduce. I've gottentest_ssltransport.pydown to 270 lines, removed__init__.pyentirely, and gottenpyproject.tomldown to just:[tool.pytest.ini_options] filterwarnings = ["error"]
Let me know if you want me to open a PR with the simplified reproducers (or, even better, if you want me to just push them).
@graingert, how should I run the new reproducer? I'm unable to get 3.11 head to crash with
pip install -r requirements.txt; python test_ssltransport.py.You need to run it with
python -W error18 remaining items
I still don't undetstand why this doesn't crash:
import sys class DelRaises: def __del__(self): global sneaky sneaky = sys._getframe() def run(): _ = DelRaises() print("Running...") run() print("Crashing...")Unfortunately, I am currently dealing with a health issue so I could not find time to deal with this in detail :(
Sorry, to hear that. :(
I was able to figure this out yesterday, but was waiting to share in the meeting. I’ll post here in case you can’t come:
I was able to get a single-threaded version to crash, too. It’s just easier to do with multiple threads since the frame object has a stale pointer into the thread’s frame stack. In our threaded example, after the thread finishes, the frame stack ceases to exist, so things can go south pretty quickly. When single-threaded, we just have a pointer into an old part of the current chunk, so things appear valid for longer if you don’t try to do much.
I’m on my phone now, but if I remember correctly, doing something like
sneaky.f_back.f_localsafter therun()makes the single-threaded version crash.Basically, we should mark this frame as “incomplete” (or just unlink it, not sure yet which is easier) after checking for existing frame objects, but before clearing stuff. This restores the behavior of 3.10, which made it look like
__del__was called fromrun’s caller, not fromrunitself.I’ll have a PR up for review this week.
Reacted by Pablo Galindo Salgado, Thomas Grainger and Łukasz LangaMarking this as 3.11 release blocker
Reacted by Łukasz LangaPRs merged so I am closing the issue. Thanks a lot to everyone that participated in this bug, from identifying it to fixing it. You all rock 🤘
I've confirmed that the 3.11 patch fixes the full original reproducer, as well.
Crash report
using https://github.com/graingert/segfault-repro running
pytestyields a segfault in about 1 in 3 runsError messages
Enter any relevant error message caused by the crash, including a core dump if there is one.
Your environment
Python 3.11.0 (main, Oct 24 2022, 19:56:13) [GCC 11.2.0] on linux5.15.0-53-generic #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64 x86_64 x86_64 GNU/LinuxLinked PRs