Repository navigation
Commit 5654983
[Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack protection to fiber stacks (ray-project#64772)
# Description
On Python 3.14 + Linux, every async-actor task permanently leaks ~518
KiB of live malloc (the per-task `asyncio.Task`,
`concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack
`Packer`s with 256 KiB internal buffers).
Closes ray-project#63290
### Root cause
**1. CPython 3.14 changed how it avoids stack overflow when freeing
objects.** Freeing one object can recursively free many others (a dict
frees its values, which free their contents, …), and each level is a
nested C call. To keep that from overflowing the C stack, CPython has
long had a safety mechanism (the "trashcan"): when it decides it's too
deep, it doesn't free the object right away. Instead it parks the object
on a per-thread *delete-later* list and drains the list once there's
stack headroom again. Up to 3.13, "too deep" was a simple recursion
counter. In 3.14 it's decided by comparing the actual machine **stack
pointer** against the stack bounds CPython recorded for the thread when
it attached (from pthreads, on Linux).
**2. Ray async actors don't run task code on the thread's normal
stack.** Each task executes on a small 256 KiB boost fiber stack
allocated elsewhere in memory. The problem is that CPython still thinks
the thread runs on its original pthread stack.
So while a task runs on a fiber, every "am I near the stack limit?"
check compares the fiber's stack pointer against the *pthread* stack's
bounds. On Linux, fiber stacks happen to be allocated at lower addresses
than the pthread stack, so CPython concludes the stack is hopelessly
overflowed and parks **every** object freed during the task (including
return-value serialization and end-of-task cleanup) on the delete-later
list.
That list is only ever drained by a later free on the same thread state
at a healthy stack margin, which never happens here as the Ray thread
only runs on fibers and Ray creates a fresh Python thread state per task
and destroys it at task end. This means that CPython destroys a thread
state **without draining its delete-later list** and the parked objects
are orphaned permanently. That's the leak.
Why the confusing symptoms:
- `boost::make_fcontext` in the issue's flamegraphs just marks *where*
the leaked allocations were made (on a fiber stack); the fiber stacks
themselves are freed correctly.
- macOS is unaffected only by luck: fiber stacks there land at *higher*
addresses than the pthread stack, so the check passes.
- 3.13 and earlier are unaffected because their trashcan uses the
counter, not the stack pointer.
### Fix
CPython 3.14.2 added an official API for exactly this situation:
`PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets
an embedder tell CPython "this thread is currently executing on *this*
stack." We call it with the fiber's stack bounds:
- at async-actor task entry in `task_execution_handler`, and
- whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers
share the thread state, so each must re-register its own stack).
With the bounds correct, the near-limit check returns to normal
behavior: objects are freed immediately, and the rare genuinely-deep
free is parked and then properly drained.
Implementation notes: the symbol is looked up via `dlsym`, so `_raylet`
still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a
more severe, since-fixed stack-check bug anyway, python/cpython#141944).
No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are
derived from the current stack pointer minus a conservative allowance
for stack already used, so the protection errs toward triggering
slightly early rather than missing an overflow. Side benefit: fibers
gain real C-stack overflow protection (RecursionError) on 3.14, which
they currently lack entirely (`boost::fibers::fixedsize_stack` has no
guard pages). Also makes `FiberState::kStackSize` public so the
anchoring uses the real fiber stack size.
## Related issue number
Closes ray-project#63290. Supersedes ray-project#63284 (same diagnosis direction, but
hand-rolled `_PyThreadStateImpl` offsets, a deliberate
`gilstate_counter` leak that freezes non-main threads, and a crash
premise that CPython 3.14.2 already fixed upstream).
## Checks
- Verified with a locally built cp314 Linux (aarch64, python:3.14.6
docker) wheel:
- refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks)
- `__del__` deferral probe: dealloc during return serialization on the
fiber **deferred → immediate**
- live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3
KiB/task** across async call → dict/bytes, async generator, sync
generator on async actor
- reporter-shaped streaming workload (400 tasks, 10 concurrent
sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped
regions 0 → 0
- async-actor smoke: correctness (echo, state, async generators,
recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s)
- throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs
baseline, same container image): fixed 5624–6076 tasks/s vs unpatched
4551–4825 tasks/s meaning no regression (the unpatched build is slower
while leaking)
- baseline (unpatched) wheel from the same tree reproduces the bug:
+4.00 refs/task, fiber dealloc deferred=True
note: fable did a majority of the heavy lifting in this investigation
with prompting on what to check next and validate the solution
---------
Signed-off-by: Mark Towers <mark@anyscale.com>
Signed-off-by: myan <myan@anyscale.com>
Co-authored-by: Mark Towers <mark@anyscale.com>
Co-authored-by: myan <myan@anyscale.com>
Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>1 parent c9228c4 commit 5654983
3 files changed
Lines changed: 190 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
431 | 431 | | |
432 | 432 | | |
433 | 433 | | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
434 | 535 | | |
435 | 536 | | |
436 | 537 | | |
| |||
1967 | 2068 | | |
1968 | 2069 | | |
1969 | 2070 | | |
| 2071 | + | |
| 2072 | + | |
| 2073 | + | |
| 2074 | + | |
| 2075 | + | |
| 2076 | + | |
1970 | 2077 | | |
1971 | 2078 | | |
1972 | 2079 | | |
| |||
2034 | 2141 | | |
2035 | 2142 | | |
2036 | 2143 | | |
2037 | | - | |
| 2144 | + | |
| 2145 | + | |
2038 | 2146 | | |
2039 | 2147 | | |
2040 | 2148 | | |
| |||
2061 | 2169 | | |
2062 | 2170 | | |
2063 | 2171 | | |
2064 | | - | |
| 2172 | + | |
| 2173 | + | |
2065 | 2174 | | |
2066 | 2175 | | |
2067 | 2176 | | |
| |||
2142 | 2251 | | |
2143 | 2252 | | |
2144 | 2253 | | |
| 2254 | + | |
2145 | 2255 | | |
2146 | 2256 | | |
2147 | 2257 | | |
| |||
2563 | 2673 | | |
2564 | 2674 | | |
2565 | 2675 | | |
| 2676 | + | |
| 2677 | + | |
| 2678 | + | |
| 2679 | + | |
| 2680 | + | |
| 2681 | + | |
| 2682 | + | |
| 2683 | + | |
| 2684 | + | |
| 2685 | + | |
2566 | 2686 | | |
2567 | 2687 | | |
2568 | 2688 | | |
| |||
4834 | 4954 | | |
4835 | 4955 | | |
4836 | 4956 | | |
| 4957 | + | |
4837 | 4958 | | |
4838 | 4959 | | |
4839 | 4960 | | |
| |||
4903 | 5024 | | |
4904 | 5025 | | |
4905 | 5026 | | |
| 5027 | + | |
| 5028 | + | |
| 5029 | + | |
| 5030 | + | |
| 5031 | + | |
| 5032 | + | |
| 5033 | + | |
| 5034 | + | |
| 5035 | + | |
| 5036 | + | |
| 5037 | + | |
| 5038 | + | |
| 5039 | + | |
| 5040 | + | |
| 5041 | + | |
4906 | 5042 | | |
4907 | 5043 | | |
4908 | 5044 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
435 | 436 | | |
436 | 437 | | |
437 | 438 | | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
438 | 488 | | |
439 | 489 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
| 92 | + | |
| 93 | + | |
92 | 94 | | |
93 | 95 | | |
94 | 96 | | |
| |||
201 | 203 | | |
202 | 204 | | |
203 | 205 | | |
204 | | - | |
205 | | - | |
206 | 206 | | |
207 | 207 | | |
208 | 208 | | |
| |||
0 commit comments