Skip to content

gh-122697: Fix free-threading memory leaks at shutdown #122703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
colesbury merged 10 commits into from
Aug 8, 2024
Merged

gh-122697: Fix free-threading memory leaks at shutdown #122703

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
c9d7132
gh-122697: Fix free-threading memory leaks at shutdown
colesbury Aug 5, 2024
04f9582
Add blurb
colesbury Aug 5, 2024
28d6768
Adjust comment
colesbury Aug 5, 2024
dcb709d
Merge branch 'main' into gh-122697-memory-leaks
colesbury Aug 6, 2024
7fc200b
Use interpreter shutdown instead of runtime shutdown
colesbury Aug 7, 2024
05db2d8
Update Misc/NEWS.d/next/Core_and_Builtins/2024-08-05-19-28-12.gh-issu…
colesbury Aug 7, 2024
44db4e0
Merge branch 'main' into gh-122697-memory-leaks
colesbury Aug 7, 2024
33cc40c
Add missing call to _PyType_FinalizeIdPool
colesbury Aug 7, 2024
312f860
Fix includes
colesbury Aug 7, 2024
70e25f5
Add missing Py_GIL_DISABLED ifdef guard
colesbury Aug 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Core_and_Builtins/2024-08-05-19-28-12.gh-issue-122697.17MvYl.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Fixed memory leaks at interpreter shutdown in the free-threaded build, and
also reporting of leaked memory blocks via ``-X showrefcount``.
Comment thread
colesbury marked this conversation as resolved.
Outdated
11 changes: 8 additions & 3 deletions Objects/obmalloc.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1109,9 +1109,12 @@ free_delayed(uintptr_t ptr)
#ifndef Py_GIL_DISABLED
free_work_item(ptr);
#else
if (_PyRuntime.stoptheworld.world_stopped) {
// Free immediately if the world is stopped, including during
// interpreter shutdown.
PyInterpreterState *interp = _PyInterpreterState_GET();
if (_PyInterpreterState_GetFinalizing(interp) != NULL ||
interp->stoptheworld.world_stopped)
{
// Free immediately during interpreter shutdown or if the world is

@kumaraditya303 kumaraditya303 Aug 7, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runtime shutdown would be better instead of interpreter shutdown?

@colesbury colesbury Aug 7, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this should be interpreter shutdown. That's why the refleak buildbots had failures.

// stopped.
free_work_item(ptr);
return;
}
Expand Down Expand Up @@ -1474,6 +1477,8 @@ _PyInterpreterState_FinalizeAllocatedBlocks(PyInterpreterState *interp)
{
#ifdef WITH_MIMALLOC
if (_PyMem_MimallocEnabled()) {
Py_ssize_t leaked = _PyInterpreterState_GetAllocatedBlocks(interp);
interp->runtime->obmalloc.interpreter_leaks += leaked;
return;
}
#endif
Expand Down
12 changes: 12 additions & 0 deletions Python/gc_free_threading.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ struct collection_state {
struct visitor_args base;
PyInterpreterState *interp;
GCState *gcstate;
_PyGC_Reason reason;
Py_ssize_t collected;
Py_ssize_t uncollectable;
Py_ssize_t long_lived_total;
Expand Down Expand Up @@ -571,6 +572,16 @@ scan_heap_visitor(const mi_heap_t *heap, const mi_heap_area_t *area,
worklist_push(&state->unreachable, op);
}
}
else if (state->reason == _Py_GC_REASON_SHUTDOWN &&
_PyObject_HasDeferredRefcount(op))
{
// Disable deferred refcounting for reachable objects as well during
// interpreter shutdown. This ensures that these objects are collected
// immediately when their last reference is removed.
disable_deferred_refcounting(op);
merge_refcount(op, 0);
state->long_lived_total++;
}
else {
// object is reachable, restore `ob_tid`; we're done with these objects
gc_restore_tid(op);
Expand Down Expand Up @@ -1221,6 +1232,7 @@ gc_collect_main(PyThreadState *tstate, int generation, _PyGC_Reason reason)
struct collection_state state = {
.interp = interp,
.gcstate = gcstate,
.reason = reason,
};

gc_collect_internal(interp, &state, generation);
Expand Down