Skip to content

Survive the shared-memory arena running out of space, and report workers killed by a signal - #6313

Merged
ondrejmirtes merged 5 commits into
2.2.xfrom
arena-shm-exhaustion
Aug 31, 2026
Merged

ondrejmirtes merged 5 commits into
2.2.xfrom
arena-shm-exhaustion

Conversation

@ondrejmirtes

Copy link
Copy Markdown
Member

Forked parallel workers die silently under 2.2.10. Root cause, verified end to end:

The shared-memory arena can be handed memory it cannot commit. ArenaCache::create() does shm_open → ftruncate(fd, 256MB) → mmap(MAP_SHARED). POSIX shm is tmpfs: the truncate is sparse and pages commit on first touch. When the backing store is out of space, the touching process takes SIGBUS — instant death, no PHP error, no shutdown function, an empty capture file. Docker's default /dev/shm is 64 MB, so every containerised user is exposed; all reporters on the issue are in containers.

Measured on the reporter's repro (doctrine/orm + symfony/http-kernel, level 10, /dev/shm = 64 MB):

config arena after clear-result-cache
2.2.10 phar, as reported yes 3/6 runs die
2.2.10 extracted out of the phar yes 4/6 runs die
2.2.10, --shm-size=1g yes 0/4
2.2.10, PHPSTAN_ARENA=0 no 0/4
2.2.9 none created 0/3

/dev/shm peak during a run: 53 MB cold, 65536 K (100%) warm. A dying worker reports signaled=true signal=7 — SIGBUS.

Why 2.2.10 and not 2.2.9. 2.2.9 ran the main process without turbo (Turbo extension: enabled in worker processes), so ArenaCache::create() was the no-op PHP twin and no arena existed at all — 0 K of /dev/shm, measured. 2.2.10's TurboProcessRestarter re-execs the main process with the extension, which is what switched the arena on for these users for the first time.

Why a warm cache and not a cold one. publishRecord() bump-allocates and writes the whole record before trying to CAS the index slot, so a publisher that loses the race leaves its copy as permanently committed dead space; Cache::save() publishes with no prior check at all. Forked workers start from an identical image at the same instant, so 7-9 of them publish the same keys simultaneously. A warm disk cache is the worst case — every worker streams the same entries in lockstep.

Why it is undiagnosable. ForkedProcess reports $exitCode = null for a non-wifexited child and ParallelAnalyser's onExit treats null exactly like exit 0: quit the process, record nothing. The job it was holding is popped and lost, so a worker dying on the last jobs can end the run reporting success with files silently unanalysed. SpawnedProcess has the same hole.

We already knew the symptom: .github/workflows/phar.yml runs its docker legs with --shm-size=1g and the comment "docker's default 64M /dev/shm makes parallel analysis SIGBUS". The workaround went into our CI instead of into the arena.

This PR so far

Only the E2E coverage, so CI is red first:

  • turbo-arena-e2e-tests — builds the extension, analyses src with the runner's /dev/shm and again with it remounted at 32 MB, and requires the two outputs to be identical.
  • e2e/parallel-worker-signal — SIGKILLs one worker from a bootstrap file and requires the run to name the signal.

Still to come

  • cut the duplicate publishing so the arena's committed footprint stops multiplying by worker count
  • report a signalled worker as an internal error naming the signal, in both ForkedProcess and SpawnedProcess

Closes phpstan/phpstan#15131

🤖 Generated with Claude Code

https://claude.ai/code/session_01XMcokMDwXGSugGttoDhhJf

fetch-depth: 0

- name: "Install PHP"
uses: "shivammathur/setup-php@f3e473d116dcccaddc5834248c87452386958240" # 2.37.2
coverage: "none"
php-version: "8.5"

- uses: "ramsey/composer-install@65e4f84970763564f46a70b8a54b90d033b3bdda" # v4.0.0
- uses: "ramsey/composer-install@65e4f84970763564f46a70b8a54b90d033b3bdda" # v4.0.0

- name: "Install bashunit"
uses: "TypedDevs/bashunit@94933c088b0719e0c4cf5e3147dad67f713b970d" # 0.44.0
Comment on lines +964 to +970
- name: "Checkout"
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0
with:
# the extension bakes its version from `git log -- turbo-ext/src`, and
# TurboExtensionEnabler activates only the expected version - a shallow
# clone hands us a "dev" build, which would skip the whole test
fetch-depth: 0
@staabm

staabm commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Maybe report in the diagnose command when the arena storage is not "big enough"?

@ondrejmirtes

Copy link
Copy Markdown
Member Author

I don't think it's necessary, we'll take what we can from the memory

ondrejmirtes and others added 5 commits August 31, 2026 11:38
…gnal

Both legs are red.

`turbo-arena-e2e-tests` builds the extension and analyses `src` twice: once
with the runner's own /dev/shm, then again with it remounted at 32 MB. The
arena is a per-run POSIX shared-memory object - shm_open() plus an ftruncate()
to 256 MB that tmpfs honours sparsely, committing pages on first touch - so
when the backing store cannot commit a page, the worker touching it takes
SIGBUS and dies where it stands. Docker gives /dev/shm 64 MB by default, which
is how phpstan/phpstan#15131 reached users; 32 MB
makes it deterministic on a 4-core runner. The two runs must come out
identical: running out of shared memory may cost sharing, never results.

`e2e/parallel-worker-signal` kills one worker with SIGKILL from a bootstrap
file, the way a native crash kills it: no PHP error, no shutdown function,
nothing in the output the main process reads back. ParallelAnalyser currently
treats such a worker exactly like one that exited 0 - it quits it and records
nothing - so the whole run says only

    Some parallel worker jobs have not finished.

which is what left the issue undiagnosable. It has to name the signal instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMcokMDwXGSugGttoDhhJf
POSIX shm is a tmpfs file, so the ftruncate() to ARENA_SIZE_LIMIT succeeded
however little room that filesystem had - the file is sparse, and the shortfall
only surfaced when a process first touched a page tmpfs could not back, as
SIGBUS. The worker died on the spot: no PHP error, no shutdown function, an
empty capture file. Docker gives /dev/shm 64 MB by default, so this was the
normal container configuration.

Measured on the reproducer of phpstan/phpstan#15131 (doctrine/orm +
symfony/http-kernel at level 10, /dev/shm at its docker default), the arena
wants ~65 MB and 3 of 6 runs lost a worker to SIGBUS. Worker count barely moves
that number - 66.6 MB at 2 workers, 68.3 MB at 8 - so it is the payload that
does not fit, not the workers racing to publish it.

So the creator now asks the filesystem what it can give and takes that, minus a
reserve for whatever else lives there. Overflowing an arena is already handled:
the bump cursor passes the end, publishing stops and analysis continues
unshared. The size travels in the header the workers already validate, and the
object's own size tells an attaching worker how much to map.

Windows is unchanged: a pagefile-backed section is committed when it is
created, so it either backs the whole reservation or fails right there.

On the same reproducer, with the same 64 MB: 3 of 3 cycles clean, arena sized
to 56 MB, /dev/shm peaking at 56 MB.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMcokMDwXGSugGttoDhhJf
Callers probe the index before serializing a value, but building the record
takes long enough for another process to publish the same key meanwhile - and
by the time the CAS discovers that, the loser has already memcpy'd the whole
payload into the arena, committing pages the backing store never gets back.
Probing again right before the copy makes losing cost only the bump it already
took.

The narrowness of the window is why this is a tidy-up rather than the fix for
phpstan/phpstan#15131: duplicates account for ~2.5% of the arena's footprint
(66.6 MB at 2 workers, 68.3 MB at 8 on the reproducer there).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMcokMDwXGSugGttoDhhJf
…lently

A worker killed by a signal never gets to run any PHP: no error is printed, no
shutdown function runs, and the output the main process reads back is empty. It
also has no exit code, and `onExit` treated that `null` exactly like a clean
exit 0 - quit the process, record nothing. The whole run then said only

    Some parallel worker jobs have not finished.

and when the dead worker happened to be holding the last job, not even that:
the job was already popped, so the analysis reported success with those files
silently unanalysed. That is what made phpstan/phpstan#15131 - workers dying of
SIGBUS on an exhausted shared-memory arena - undiagnosable.

So the signal now travels alongside the exit code, from both mechanisms (the
wait status for a forked worker, react's term signal for a spawned one), and
becomes an internal error naming it. Signal numbers are platform-specific -
SIGBUS is 7 on Linux and 10 on macOS - so TerminationSignal reads the name off
the running platform's own constants. The same report goes to the fixer's
forked child, and an exit that is neither a code nor a signal no longer
swallows whatever the child did write (`pcntl_fork() failed.`, until now).

Covered by e2e/parallel-worker-signal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMcokMDwXGSugGttoDhhJf
@ondrejmirtes
ondrejmirtes merged commit 237575e into 2.2.x Aug 31, 2026
441 checks passed
@ondrejmirtes
ondrejmirtes deleted the arena-shm-exhaustion branch August 31, 2026 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2.2.10: forked parallel workers die with "Some parallel worker jobs have not finished"

3 participants