Skip to content

Commit 5654983

Browse files
pseudo-rnd-thoughtsMark TowersMengjinYan
authored
[Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack protection to fiber stacks (ray-project#64772)
# Description On Python 3.14 + Linux, every async-actor task permanently leaks ~518 KiB of live malloc (the per-task `asyncio.Task`, `concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack `Packer`s with 256 KiB internal buffers). Closes ray-project#63290 ### Root cause **1. CPython 3.14 changed how it avoids stack overflow when freeing objects.** Freeing one object can recursively free many others (a dict frees its values, which free their contents, …), and each level is a nested C call. To keep that from overflowing the C stack, CPython has long had a safety mechanism (the "trashcan"): when it decides it's too deep, it doesn't free the object right away. Instead it parks the object on a per-thread *delete-later* list and drains the list once there's stack headroom again. Up to 3.13, "too deep" was a simple recursion counter. In 3.14 it's decided by comparing the actual machine **stack pointer** against the stack bounds CPython recorded for the thread when it attached (from pthreads, on Linux). **2. Ray async actors don't run task code on the thread's normal stack.** Each task executes on a small 256 KiB boost fiber stack allocated elsewhere in memory. The problem is that CPython still thinks the thread runs on its original pthread stack. So while a task runs on a fiber, every "am I near the stack limit?" check compares the fiber's stack pointer against the *pthread* stack's bounds. On Linux, fiber stacks happen to be allocated at lower addresses than the pthread stack, so CPython concludes the stack is hopelessly overflowed and parks **every** object freed during the task (including return-value serialization and end-of-task cleanup) on the delete-later list. That list is only ever drained by a later free on the same thread state at a healthy stack margin, which never happens here as the Ray thread only runs on fibers and Ray creates a fresh Python thread state per task and destroys it at task end. This means that CPython destroys a thread state **without draining its delete-later list** and the parked objects are orphaned permanently. That's the leak. Why the confusing symptoms: - `boost::make_fcontext` in the issue's flamegraphs just marks *where* the leaked allocations were made (on a fiber stack); the fiber stacks themselves are freed correctly. - macOS is unaffected only by luck: fiber stacks there land at *higher* addresses than the pthread stack, so the check passes. - 3.13 and earlier are unaffected because their trashcan uses the counter, not the stack pointer. ### Fix CPython 3.14.2 added an official API for exactly this situation: `PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets an embedder tell CPython "this thread is currently executing on *this* stack." We call it with the fiber's stack bounds: - at async-actor task entry in `task_execution_handler`, and - whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers share the thread state, so each must re-register its own stack). With the bounds correct, the near-limit check returns to normal behavior: objects are freed immediately, and the rare genuinely-deep free is parked and then properly drained. Implementation notes: the symbol is looked up via `dlsym`, so `_raylet` still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a more severe, since-fixed stack-check bug anyway, python/cpython#141944). No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are derived from the current stack pointer minus a conservative allowance for stack already used, so the protection errs toward triggering slightly early rather than missing an overflow. Side benefit: fibers gain real C-stack overflow protection (RecursionError) on 3.14, which they currently lack entirely (`boost::fibers::fixedsize_stack` has no guard pages). Also makes `FiberState::kStackSize` public so the anchoring uses the real fiber stack size. ## Related issue number Closes ray-project#63290. Supersedes ray-project#63284 (same diagnosis direction, but hand-rolled `_PyThreadStateImpl` offsets, a deliberate `gilstate_counter` leak that freezes non-main threads, and a crash premise that CPython 3.14.2 already fixed upstream). ## Checks - Verified with a locally built cp314 Linux (aarch64, python:3.14.6 docker) wheel: - refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks) - `__del__` deferral probe: dealloc during return serialization on the fiber **deferred → immediate** - live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3 KiB/task** across async call → dict/bytes, async generator, sync generator on async actor - reporter-shaped streaming workload (400 tasks, 10 concurrent sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped regions 0 → 0 - async-actor smoke: correctness (echo, state, async generators, recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s) - throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs baseline, same container image): fixed 5624–6076 tasks/s vs unpatched 4551–4825 tasks/s meaning no regression (the unpatched build is slower while leaking) - baseline (unpatched) wheel from the same tree reproduces the bug: +4.00 refs/task, fiber dealloc deferred=True note: fable did a majority of the heavy lifting in this investigation with prompting on what to check next and validate the solution --------- Signed-off-by: Mark Towers <mark@anyscale.com> Signed-off-by: myan <myan@anyscale.com> Co-authored-by: Mark Towers <mark@anyscale.com> Co-authored-by: myan <myan@anyscale.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>
1 parent c9228c4 commit 5654983

3 files changed

Lines changed: 190 additions & 4 deletions

File tree

‎python/ray/_raylet.pyx‎

Lines changed: 138 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -431,6 +431,107 @@ def wait_for_persisted_port(
431431
return result.value()
432432

433433

434+
cdef extern from *:
435+
"""
436+
#if PY_VERSION_HEX >= 0x030E0000 && !defined(MS_WINDOWS)
437+
#include <dlfcn.h>
438+
#ifndef RTLD_DEFAULT
439+
#define RTLD_DEFAULT ((void *)0)
440+
#endif
441+
#include "ray/core_worker/task_execution/fiber.h"
442+
443+
typedef int (*_RaySetStackProtectionFn)(PyThreadState *, void *, size_t);
444+
445+
/* CPython 3.14+ detects C-stack overflow by comparing the stack pointer
446+
* against the thread's recorded stack bounds, which it fills in once, from
447+
* pthreads, when a thread state first attaches. Async-actor tasks run on
448+
* 256 KiB boost fiber stacks pthreads knows nothing about; on Linux the fiber
449+
* stacks sit below the thread stack, so the checks see a negative margin,
450+
* conclude the stack is exhausted, and _Py_Dealloc defers every GC-object
451+
* deallocation onto a per-thread-state list that is never drained -- leaking
452+
* the task's whole object graph. Call this at async-actor task entry and again
453+
* after YieldCurrentFiber resumes, because concurrent fibers share one thread
454+
* state and each overwrites the bounds recorded by the last one to run.
455+
*
456+
* Callers must only invoke this while running on a fiber, which both call
457+
* sites enforce by testing for an actor task. An async actor's creation task
458+
* reaches run_async_func_or_coro_in_event_loop too -- for argument
459+
* deserialization and for __init__ via sync_to_async, where
460+
* boost::this_fiber::yield() is a no-op on a thread's main context -- and
461+
* re-anchoring there would replace the correct bounds on that thread's
462+
* long-lived PyThreadState with fiber-shaped ones that nothing restores.
463+
*
464+
* used_upper_bound reconstructs the fiber stack's top from a local's address:
465+
* top = &anchor + used_upper_bound, base = top - kStackSize. It should
466+
* upper-bound the fiber stack already consumed at the call site.
467+
*
468+
* - Over-estimating is safe but costs usable stack: the reported base sits
469+
* (used_upper_bound - actual) bytes above the real one, so Python raises
470+
* RecursionError that much earlier.
471+
* - Under-estimating is only unsafe beyond one soft margin. CPython places
472+
* its soft limit at base + 2 * _PyOS_STACK_MARGIN_BYTES (32 KiB on 64-bit),
473+
* so recursion can only run past the real end of the buffer -- which has no
474+
* guard page -- once actual usage exceeds used_upper_bound + 32 KiB.
475+
*
476+
* Hence the two values below: 32 KiB at task entry, which sits roughly six C++
477+
* frames from the fiber's entry point, and 96 KiB after YieldCurrentFiber
478+
* returns, which is reached through the interpreter and so carries additional
479+
* eval-loop and Cython frames. Both are round numbers chosen to be far above
480+
* anything those chains plausibly use, not measurements.
481+
*
482+
* TODO(#63290): record the fiber stack's real base in FiberState when the fiber
483+
* starts -- its body is the outermost frame, so the unknown shrinks to boost's
484+
* fixed entry trampoline -- and drop these estimates entirely.
485+
*
486+
* PyUnstable_ThreadState_SetStackProtection only exists on CPython >= 3.14.2;
487+
* on older 3.14 releases the dlsym below fails and this is a no-op. The leak
488+
* itself only reproduces where the fiber stacks land below the thread stack
489+
* (Linux), but the bounds are equally wrong elsewhere -- on macOS they make the
490+
* margin hugely positive, which suppresses the overflow check instead of the
491+
* deallocator -- so the re-anchoring is applied on every non-Windows platform.
492+
*/
493+
static int RayReanchorStackProtectionToCurrentFiberStack(size_t used_upper_bound) {
494+
static _RaySetStackProtectionFn set_stack_protection =
495+
(_RaySetStackProtectionFn)dlsym(
496+
RTLD_DEFAULT, "PyUnstable_ThreadState_SetStackProtection");
497+
if (set_stack_protection == NULL) {
498+
/* CPython 3.14.0 and 3.14.1 predate the API. Say so once: otherwise the
499+
* only symptom is unbounded memory growth in async actors, with nothing
500+
* in the logs to point at the cause. */
501+
static bool warned_missing_api = false;
502+
if (!warned_missing_api) {
503+
warned_missing_api = true;
504+
RAY_LOG(WARNING)
505+
<< "PyUnstable_ThreadState_SetStackProtection is not available in "
506+
<< "this interpreter (Python " << PY_VERSION << "). Async actor "
507+
<< "tasks will leak memory on Python 3.14; upgrade to CPython "
508+
<< "3.14.2 or later. See "
509+
<< "https://github.com/ray-project/ray/issues/63290";
510+
}
511+
return 0;
512+
}
513+
char anchor;
514+
uintptr_t top = (uintptr_t)&anchor + used_upper_bound;
515+
size_t stack_size = ray::core::FiberState::kStackSize;
516+
int rc = set_stack_protection(
517+
PyThreadState_Get(), (void *)(top - stack_size), stack_size);
518+
if (rc < 0) {
519+
/* Cannot happen for kStackSize (only fails for sizes below
520+
* _PyOS_MIN_STACK_SIZE), but never leak an exception. */
521+
PyErr_Clear();
522+
}
523+
return rc;
524+
}
525+
#else
526+
static int RayReanchorStackProtectionToCurrentFiberStack(size_t used_upper_bound) {
527+
(void)used_upper_bound;
528+
return 0;
529+
}
530+
#endif
531+
"""
532+
int RayReanchorStackProtectionToCurrentFiberStack(size_t used_upper_bound)
533+
534+
434535
cdef increase_recursion_limit():
435536
"""
436537
Ray does some weird things with asio fibers and asyncio to run asyncio actors.
@@ -1967,6 +2068,12 @@ cdef void execute_task(
19672068
TaskID task_id = core_worker.get_current_task_id()
19682069
uint64_t attempt_number = core_worker.get_current_task_attempt_number()
19692070

2071+
# Whether this is an actor *task*, as opposed to an actor creation task or a
2072+
# normal task. Async-actor tasks run on boost fiber stacks; the creation task
2073+
# of an async actor does not -- it runs on the main task-execution thread --
2074+
# so anything keyed on "am I on a fiber" must distinguish the two.
2075+
is_actor_task = <int>task_type == <int>TASK_TYPE_ACTOR_TASK
2076+
19702077
# Helper method used to exit current asyncio actor.
19712078
# This is called when a KeyboardInterrupt is received by the main thread.
19722079
# Upon receiving a KeyboardInterrupt signal, Ray will exit the current
@@ -2034,7 +2141,8 @@ cdef void execute_task(
20342141
else:
20352142
return core_worker.run_async_func_or_coro_in_event_loop(
20362143
async_function, function_descriptor,
2037-
name_of_concurrency_group_to_execute, task_id=task_id,
2144+
name_of_concurrency_group_to_execute,
2145+
is_actor_task=is_actor_task, task_id=task_id,
20382146
task_name=task_name, func_args=(actor, *arguments),
20392147
func_kwargs=kwarguments)
20402148

@@ -2061,7 +2169,8 @@ cdef void execute_task(
20612169
metadata_pairs, object_refs))
20622170
args = core_worker.run_async_func_or_coro_in_event_loop(
20632171
deserialize_args, function_descriptor,
2064-
name_of_concurrency_group_to_execute)
2172+
name_of_concurrency_group_to_execute,
2173+
is_actor_task=is_actor_task)
20652174
else:
20662175
# Defer task cancellation (SIGINT) until after the task argument
20672176
# deserialization context has been left.
@@ -2142,6 +2251,7 @@ cdef void execute_task(
21422251
execute_streaming_generator_async(context),
21432252
function_descriptor,
21442253
name_of_concurrency_group_to_execute,
2254+
is_actor_task=is_actor_task,
21452255
task_id=task_id,
21462256
task_name=task_name)
21472257
else:
@@ -2563,6 +2673,16 @@ cdef CRayStatus task_execution_handler(
25632673
int64_t num_objects_per_yield,
25642674
optional[c_string] c_tensor_transport) nogil:
25652675
with gil, disable_client_hook():
2676+
# Async actor tasks run on a boost fiber stack; re-anchor CPython's
2677+
# stack protection to it. The handler entry is close to the top of the
2678+
# fiber stack, so a small used-stack upper bound suffices. Actor
2679+
# creation tasks are excluded: they run on the regular task-execution
2680+
# pthread, not a fiber, even for async actors.
2681+
if (<int>task_type == <int>TASK_TYPE_ACTOR_TASK
2682+
and CCoreWorkerProcess.GetCoreWorker().GetWorkerContext()
2683+
.CurrentActorIsAsync()):
2684+
RayReanchorStackProtectionToCurrentFiberStack(32 * 1024)
2685+
25662686
# Initialize job_config if it hasn't already.
25672687
# Setup system paths configured in job_config.
25682688
maybe_initialize_job_config()
@@ -4834,6 +4954,7 @@ cdef class CoreWorker:
48344954
function_descriptor: FunctionDescriptor,
48354955
specified_cgname: str,
48364956
*,
4957+
is_actor_task: bool,
48374958
task_id: Optional[TaskID] = None,
48384959
task_name: Optional[str] = None,
48394960
func_args: Optional[Tuple] = None,
@@ -4903,6 +5024,21 @@ cdef class CoreWorker:
49035024
with nogil:
49045025
(CCoreWorkerProcess.GetCoreWorker()
49055026
.YieldCurrentFiber(event))
5027+
# Re-anchor this fiber's stack before running the rest of the task on it:
5028+
# while this fiber was parked, other fibers of the same concurrency group
5029+
# ran and overwrote the bounds on the thread state they all share. The
5030+
# bound is larger than at task entry because this call site is several C
5031+
# frames deeper.
5032+
#
5033+
# Guarded on the same condition as the hook at task entry. An async
5034+
# actor's creation task also reaches here -- for argument deserialization
5035+
# and for __init__ via sync_to_async -- but runs on the main
5036+
# task-execution thread rather than a fiber, where CPython's recorded
5037+
# bounds are already correct and must not be overwritten. The
5038+
# CurrentActorIsAsync() half of the entry-site condition is implied here:
5039+
# every caller of this method is already inside an is-asyncio branch.
5040+
if is_actor_task:
5041+
RayReanchorStackProtectionToCurrentFiberStack(96 * 1024)
49065042
try:
49075043
result = future.result()
49085044
except concurrent.futures.CancelledError:

‎python/ray/tests/test_asyncio.py‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
import sys
55
import threading
66
import time
7+
import weakref
78

89
import pytest
910

@@ -435,5 +436,54 @@ def hi_sync(self, task_id: str, *, specified_cgname: str):
435436
)
436437

437438

439+
def test_async_actor_finalizes_objects_dropped_on_fiber(ray_start_regular_shared):
440+
"""Objects dropped as an async-actor task returns must be finalized promptly.
441+
442+
Async-actor tasks execute on boost fiber stacks. CPython's C-stack overflow
443+
checks are keyed on the bounds it recorded for the *thread*, so a fiber stack
444+
can be mistaken for an exhausted one -- and on CPython 3.14 the deallocator
445+
then parks every object it frees on a per-thread-state list that is never
446+
drained, leaking the task's entire object graph.
447+
448+
A task's return value is serialized on the fiber, so the last reference the
449+
actor process holds to the payload below is released there. If deallocation
450+
is being deferred, the payload's finalizer never runs.
451+
452+
max_concurrency > 1 is required for coverage rather than realism: with a
453+
single fiber, anchoring once at task entry would suffice, so only interleaved
454+
fibers exercise the re-anchoring that happens after a fiber resumes.
455+
"""
456+
457+
class Payload:
458+
pass
459+
460+
@ray.remote(max_concurrency=2)
461+
class Probe:
462+
def __init__(self, unused):
463+
# Taking a constructor argument is deliberate: it makes the creation
464+
# task deserialize arguments, which is one of the paths that reaches
465+
# the fiber bookkeeping from a thread that is not running a fiber.
466+
self._finalizers = []
467+
468+
async def make_payload(self):
469+
payload = Payload()
470+
self._finalizers.append(weakref.finalize(payload, lambda: None))
471+
return payload
472+
473+
async def num_finalized(self):
474+
return sum(not f.alive for f in self._finalizers)
475+
476+
num_tasks = 50
477+
probe = Probe.remote("unused")
478+
ray.get([probe.make_payload.remote() for _ in range(num_tasks)])
479+
480+
num_finalized = ray.get(probe.num_finalized.remote())
481+
assert num_finalized == num_tasks, (
482+
f"{num_tasks - num_finalized} of {num_tasks} payloads returned by an async "
483+
"actor were never finalized in the actor process, so their deallocation is "
484+
"being deferred and never completed"
485+
)
486+
487+
438488
if __name__ == "__main__":
439489
sys.exit(pytest.main(["-sv", __file__]))

‎src/ray/core_worker/task_execution/fiber.h‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,8 @@ using FiberChannel = boost::fibers::unbuffered_channel<std::function<void()>>;
8989

9090
class FiberState {
9191
public:
92+
static constexpr size_t kStackSize = 1024 * 256;
93+
9294
static bool NeedDefaultExecutor(int32_t max_concurrency_in_default_group,
9395
bool has_other_concurrency_groups) {
9496
RAY_UNUSED(max_concurrency_in_default_group);
@@ -201,8 +203,6 @@ class FiberState {
201203
}
202204

203205
private:
204-
static constexpr size_t kStackSize = 1024 * 256;
205-
206206
// The fiber stack allocator.
207207
boost::fibers::fixedsize_stack allocator_;
208208
/// The fiber channel used to send task between the submitter thread

0 commit comments

Comments
 (0)