Repository navigation
Survive the shared-memory arena running out of space, and report workers killed by a signal - #6313
Merged
Merged
Conversation
| fetch-depth: 0 | ||
|
|
||
| - name: "Install PHP" | ||
| uses: "shivammathur/setup-php@f3e473d116dcccaddc5834248c87452386958240" # 2.37.2 |
| coverage: "none" | ||
| php-version: "8.5" | ||
|
|
||
| - uses: "ramsey/composer-install@65e4f84970763564f46a70b8a54b90d033b3bdda" # v4.0.0 |
| - uses: "ramsey/composer-install@65e4f84970763564f46a70b8a54b90d033b3bdda" # v4.0.0 | ||
|
|
||
| - name: "Install bashunit" | ||
| uses: "TypedDevs/bashunit@94933c088b0719e0c4cf5e3147dad67f713b970d" # 0.44.0 |
Comment on lines
+964
to
+970
| - name: "Checkout" | ||
| uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 | ||
| with: | ||
| # the extension bakes its version from `git log -- turbo-ext/src`, and | ||
| # TurboExtensionEnabler activates only the expected version - a shallow | ||
| # clone hands us a "dev" build, which would skip the whole test | ||
| fetch-depth: 0 |
Contributor
|
Maybe report in the diagnose command when the arena storage is not "big enough"? |
Member
Author
|
I don't think it's necessary, we'll take what we can from the memory |
…gnal Both legs are red. `turbo-arena-e2e-tests` builds the extension and analyses `src` twice: once with the runner's own /dev/shm, then again with it remounted at 32 MB. The arena is a per-run POSIX shared-memory object - shm_open() plus an ftruncate() to 256 MB that tmpfs honours sparsely, committing pages on first touch - so when the backing store cannot commit a page, the worker touching it takes SIGBUS and dies where it stands. Docker gives /dev/shm 64 MB by default, which is how phpstan/phpstan#15131 reached users; 32 MB makes it deterministic on a 4-core runner. The two runs must come out identical: running out of shared memory may cost sharing, never results. `e2e/parallel-worker-signal` kills one worker with SIGKILL from a bootstrap file, the way a native crash kills it: no PHP error, no shutdown function, nothing in the output the main process reads back. ParallelAnalyser currently treats such a worker exactly like one that exited 0 - it quits it and records nothing - so the whole run says only Some parallel worker jobs have not finished. which is what left the issue undiagnosable. It has to name the signal instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XMcokMDwXGSugGttoDhhJf
POSIX shm is a tmpfs file, so the ftruncate() to ARENA_SIZE_LIMIT succeeded however little room that filesystem had - the file is sparse, and the shortfall only surfaced when a process first touched a page tmpfs could not back, as SIGBUS. The worker died on the spot: no PHP error, no shutdown function, an empty capture file. Docker gives /dev/shm 64 MB by default, so this was the normal container configuration. Measured on the reproducer of phpstan/phpstan#15131 (doctrine/orm + symfony/http-kernel at level 10, /dev/shm at its docker default), the arena wants ~65 MB and 3 of 6 runs lost a worker to SIGBUS. Worker count barely moves that number - 66.6 MB at 2 workers, 68.3 MB at 8 - so it is the payload that does not fit, not the workers racing to publish it. So the creator now asks the filesystem what it can give and takes that, minus a reserve for whatever else lives there. Overflowing an arena is already handled: the bump cursor passes the end, publishing stops and analysis continues unshared. The size travels in the header the workers already validate, and the object's own size tells an attaching worker how much to map. Windows is unchanged: a pagefile-backed section is committed when it is created, so it either backs the whole reservation or fails right there. On the same reproducer, with the same 64 MB: 3 of 3 cycles clean, arena sized to 56 MB, /dev/shm peaking at 56 MB. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XMcokMDwXGSugGttoDhhJf
Callers probe the index before serializing a value, but building the record takes long enough for another process to publish the same key meanwhile - and by the time the CAS discovers that, the loser has already memcpy'd the whole payload into the arena, committing pages the backing store never gets back. Probing again right before the copy makes losing cost only the bump it already took. The narrowness of the window is why this is a tidy-up rather than the fix for phpstan/phpstan#15131: duplicates account for ~2.5% of the arena's footprint (66.6 MB at 2 workers, 68.3 MB at 8 on the reproducer there). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XMcokMDwXGSugGttoDhhJf
…lently
A worker killed by a signal never gets to run any PHP: no error is printed, no
shutdown function runs, and the output the main process reads back is empty. It
also has no exit code, and `onExit` treated that `null` exactly like a clean
exit 0 - quit the process, record nothing. The whole run then said only
Some parallel worker jobs have not finished.
and when the dead worker happened to be holding the last job, not even that:
the job was already popped, so the analysis reported success with those files
silently unanalysed. That is what made phpstan/phpstan#15131 - workers dying of
SIGBUS on an exhausted shared-memory arena - undiagnosable.
So the signal now travels alongside the exit code, from both mechanisms (the
wait status for a forked worker, react's term signal for a spawned one), and
becomes an internal error naming it. Signal numbers are platform-specific -
SIGBUS is 7 on Linux and 10 on macOS - so TerminationSignal reads the name off
the running platform's own constants. The same report goes to the fixer's
forked child, and an exit that is neither a code nor a signal no longer
swallows whatever the child did write (`pcntl_fork() failed.`, until now).
Covered by e2e/parallel-worker-signal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMcokMDwXGSugGttoDhhJf
ondrejmirtes
force-pushed
the
arena-shm-exhaustion
branch
from
August 31, 2026 09:41
f24ce7e to
237575e
Compare
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forked parallel workers die silently under 2.2.10. Root cause, verified end to end:
The shared-memory arena can be handed memory it cannot commit.
ArenaCache::create()doesshm_open→ftruncate(fd, 256MB)→mmap(MAP_SHARED). POSIX shm is tmpfs: the truncate is sparse and pages commit on first touch. When the backing store is out of space, the touching process takes SIGBUS — instant death, no PHP error, no shutdown function, an empty capture file. Docker's default/dev/shmis 64 MB, so every containerised user is exposed; all reporters on the issue are in containers.Measured on the reporter's repro (
doctrine/orm+symfony/http-kernel, level 10,/dev/shm= 64 MB):clear-result-cache--shm-size=1gPHPSTAN_ARENA=0/dev/shmpeak during a run: 53 MB cold, 65536 K (100%) warm. A dying worker reportssignaled=true signal=7— SIGBUS.Why 2.2.10 and not 2.2.9. 2.2.9 ran the main process without turbo (
Turbo extension: enabled in worker processes), soArenaCache::create()was the no-op PHP twin and no arena existed at all — 0 K of/dev/shm, measured. 2.2.10'sTurboProcessRestarterre-execs the main process with the extension, which is what switched the arena on for these users for the first time.Why a warm cache and not a cold one.
publishRecord()bump-allocates and writes the whole record before trying to CAS the index slot, so a publisher that loses the race leaves its copy as permanently committed dead space;Cache::save()publishes with no prior check at all. Forked workers start from an identical image at the same instant, so 7-9 of them publish the same keys simultaneously. A warm disk cache is the worst case — every worker streams the same entries in lockstep.Why it is undiagnosable.
ForkedProcessreports$exitCode = nullfor a non-wifexitedchild andParallelAnalyser'sonExittreatsnullexactly like exit 0: quit the process, record nothing. The job it was holding is popped and lost, so a worker dying on the last jobs can end the run reporting success with files silently unanalysed.SpawnedProcesshas the same hole.We already knew the symptom:
.github/workflows/phar.ymlruns its docker legs with--shm-size=1gand the comment "docker's default 64M /dev/shm makes parallel analysis SIGBUS". The workaround went into our CI instead of into the arena.This PR so far
Only the E2E coverage, so CI is red first:
turbo-arena-e2e-tests— builds the extension, analysessrcwith the runner's/dev/shmand again with it remounted at 32 MB, and requires the two outputs to be identical.e2e/parallel-worker-signal— SIGKILLs one worker from a bootstrap file and requires the run to name the signal.Still to come
ForkedProcessandSpawnedProcessCloses phpstan/phpstan#15131
🤖 Generated with Claude Code
https://claude.ai/code/session_01XMcokMDwXGSugGttoDhhJf