Repository navigation
s390x RHEL7 got crash on the refleak test #102351
Description
Activity
- changed the title
[-]s390x RHEL7 got crash on the refleak test.[/-][+]s390x RHEL7 got crash on the refleak test[/+]on Mar 1, 2023 I will try to debug issue with QEMU..
Okay no way to setup through QEMU since I can not get the image of s390x RHEL image.
So I will try to contract with the machine manager.And also @vstinner share useful approach for this issue: https://pythondev.readthedocs.io/debug_tools.html#debug-crash-in-garbage-collection-visit-decref
I've reproduced it on a Mac and bisected to:
efc985a714b6f43c43ae629183f95618054422ae is the first bad commit commit efc985a714b6f43c43ae629183f95618054422ae Author: Erlend E. Aasland <erlend.aasland@protonmail.com> Date: Thu Feb 23 16:03:13 2023 +0100 gh-93649: Split exception tests from _testcapimodule.c (GH-102173) Automerge-Triggered-By: GH:erlend-aasland Lib/test/test_capi/test_exceptions.py | 145 +++++++++++++++++ Lib/test/test_capi/test_misc.py | 131 +-------------- Modules/Setup.stdlib.in | 2 +- Modules/_testcapi/exceptions.c | 277 ++++++++++++++++++++++++++++++++ Modules/_testcapi/parts.h | 1 + Modules/_testcapimodule.c | 290 +--------------------------------- PCbuild/_testcapi.vcxproj | 1 + PCbuild/_testcapi.vcxproj.filters | 3 + 8 files changed, 434 insertions(+), 416 deletions(-) create mode 100644 Lib/test/test_capi/test_exceptions.py create mode 100644 Modules/_testcapi/exceptions.cReacted by Erlend E. AaslandBut I think it may be a red herring (when I touch stuff the crash goes away). Maybe we just need to bump the magic number.
- added a commit that references this issue
on Mar 1, 2023 I suspect this PR needed the magic number bump: #101933
But I think it may be a red herring (when I touch stuff the crash goes away). Maybe we just need to bump the magic number.
Bumping magic number didn't solve this: #102359.
@iritkatriel, thanks for the heads-up, I'll have a look!
@iritkatriel: what's your repro on macOS? I can't reproduce it (yet) on my M1 running Ventura 13.1.
I was running
python -m test -j2 -R3:3 test_capirunning just a submodule of test_capi didn’t fail. I think something to do with module import/setup?
When I debug the recount: -2459565876494606883
It could be overflowed issue or an uninitialized issue.When I debug the recount: -2459565876494606883
Ah it's because already freed.
I think it might be the same issue as #102350.
I narrowed it down to that test, and if I update my branch with that last commit the crash is gone.Reacted by Donghee NaI narrowed it down to that test, and if I update my branch with that last commit the crash is gone.
Ah for my case, it's still happening with the latest commit.
Clean build?
Clean build?
Yes with the clean-build.
Please read my comment when I triggered the refleak test with the PR.
#102350 (comment)
This issue was created from the comment.Yes, I just got the crash again with main too. Sorry about the noise.
Reacted by Donghee NaI think the problem comes from test_watcher_raises_error in Lib/test/test_capi/test_watchers.py.
This test registers a dictionary watcher which raises an exception. That takes us to
_PyErr_WriteUnraisableMsg, wherePy_DECREF(hook_args);fails because the last item in hook_args (theobj, i.e., the dictionary) has refcnt 0.This only happens when I run it with
./python.exe -m test -v -j2 -R3:3 test_capi. If I remove the-j2it doesn't fail.I added
assert(Py_REFCNT((PyObject*)mp) > 0);at the beginning of_PyDict_NotifyEvent, and it failed (during make)which seem wrong(apparently this is fine, for PyDict_EVENT_DEALLOCATED events).Reacted by Donghee Na and Carl MeyerFurther debugging confirmed that the root cause of this issue is #102381. Fixing all code/func/dict dealloc callbacks to temporarily resurrect the object before calling the callback, and to not save references to the object in unraisablehook, makes this issue go away.
For posterity: the specific repro here requiring
-j2 -Rseems to be an artifact of those options causing a GC run to tend to occur while an error-raising func watcher is active, and that GC run deallocates a cycle of unrelated objects from thedatetime.pypure-Python fallbacks, including some functions.
link: https://buildbot.python.org/all/#/builders/217/builds/1082/steps/5/logs/stdio
Linked PRs