[workflows] gate-tests: robust temp-repo cleanup (ENOTEMPTY race) (#249) - #263

Refs #249

Governance: perf/coverage-gate.test.mjs is a perf/coverage-* path, so this is a [workflows] PR. It needs Helms's sign-off, plus Code Reviewer's governance check (gate clause 3). Nothing is loosened: every assertion, threshold, measurement scope and the gate itself (perf/coverage-gate.mjs) are unchanged, and gitleaks is untouched. It's a CI-reliability fix allowed under the #132 hold.

Merge: Tech Coordinator merges through the review gate; I won't merge this.

Summary

• Root cause. git commit forks git maintenance run --auto --quiet --detach. That process daemonizes (setsid) and can outlive the commit. When it runs late, it adds files under .git/objects (its maintenance.lock, or a commit-graph) after the test's rmSync has read the directory, so the final rmdir fails with ENOTEMPTY. This is the failure in [workflows] gate-tests: temp-repo cleanup races (ENOTEMPTY in rmdir) and fails unrelated PRs #249 (PR test(api): isolate the shared fixture clock; no wall-clock branches left (#226) #248, run 37771004042: ENOTEMPTY: directory not empty, rmdir '/tmp/gate-*/.git/objects'). I confirmed the fork and the daemon with GIT_TRACE and strace on git 2.56. CI runs git 2.55 and Node 22.
• Fix 1 (root cause), initRepo(): each temp repo sets maintenance.auto=false and gc.auto=0, so git commit never forks the daemon.
• Fix 2 (defence), removeRepo(dir): all 64 temp-repo cleanups (rmSync(dir, { recursive: true, force: true }) in finally) now call this helper. It retries the whole rmSync (a fresh walk of the tree) up to 5 times on ENOTEMPTY/EBUSY and rethrows anything else. rmSync's own maxRetries isn't enough by itself: on Node 22, lib/internal/fs/rimraf.js retries only the rmdir and never removes a file that appeared after it read the directory. The measurements below show this ("maxRetries only" is the shape the issue suggested).
• Test helpers only. The rmSync calls inside test bodies (removing fixture files) are unchanged, and no test, assertion or expected output changed.

Decisions

Measurements

The race needs the detached daemon to be scheduled late, which happens on a busy runner but not on an idle machine. Without help it didn't reproduce: 0 of 80 runs on Linux (Node 22, git 2.56), 0 of 40 with commit-graph writes forced on, and 0 on macOS. So the harness models the CI condition. It enables the commit-graph maintenance task globally (auto=-1, so the daemon always writes under .git/objects), and it wraps git's internal git-core/git so the detached maintenance run --auto starts 0–90 ms late. Every other git call goes to the real git. Each mode is 10 rounds × 8 parallel node --test perf/coverage-gate.test.mjs runs in a node:22-alpine container (--cpus 2, git 2.56.0, Node v22.23.2):

The "before" failures include the exact CI message, ENOTEMPTY: directory not empty, rmdir '/tmp/gate-X/.git/objects', and .../.git/objects/info/commit-graphs. Each layer is enough on its own, and together they cover both a daemon we don't fork and any other late writer.

RESULT mode=before runs=80 failed_runs=4 enotempty_runs=4
RESULT mode=after runs=80 failed_runs=0 enotempty_runs=0
RESULT mode=retryonly runs=80 failed_runs=0 enotempty_runs=0
RESULT mode=configonly runs=80 failed_runs=0 enotempty_runs=0

Local loop, as the issue asked (macOS, Node v24.18.0, git 2.56.0):

$ for i in $(seq 1 30); do node --test perf/coverage-gate.test.mjs || f=$((f+1)); done
local macOS sequential: 30 runs, 0 failed   (84/84 tests each)

The repro harness isn't committed: this PR owns only perf/coverage-gate.test.mjs, and a new perf/ file would be another governed path. Here it is in full:

# usage: MODE=before|after ROUNDS=n PAR=p sh loop.sh   (runs inside the container)
apk add -q git --repository=https://dl-cdn.alpinelinux.org/alpine/edge/main >/dev/null 2>&1
[ -n "$AMPLIFY" ] && git config --global maintenance.commit-graph.enabled true && git config --global maintenance.commit-graph.auto -1 && echo "amplified: detached maintenance always writes a commit-graph"
[ -n "$LATE" ] && cp /late-daemon.sh /usr/libexec/git-core/git && chmod +x /usr/libexec/git-core/git && echo "late daemon: detached maintenance delayed 0-90 ms"
mkdir -p /w && tar -xf /ctx.tar -C /w
cd /w && cp "$MODE.test.mjs" perf/coverage-gate.test.mjs
echo "git: $(git --version), node: $(node --version), mode: $MODE, rounds: $ROUNDS x $PAR parallel"
runs=0; fails=0; enot=0
r=1
while [ "$r" -le "$ROUNDS" ]; do
  p=1
  while [ "$p" -le "$PAR" ]; do node --test perf/coverage-gate.test.mjs > "/tmp/o.$p" 2>&1 & p=$((p+1)); done
  wait
  p=1
  while [ "$p" -le "$PAR" ]; do
    runs=$((runs+1))
    if grep -qE "^(ℹ|#) fail 0$" "/tmp/o.$p"; then :; else fails=$((fails+1)); fi
    if grep -q ENOTEMPTY "/tmp/o.$p"; then enot=$((enot+1)); grep "ENOTEMPTY\|^not ok" "/tmp/o.$p" | head -4; fi
    p=$((p+1))
  done
  r=$((r+1))
done
echo "RESULT mode=$MODE runs=$runs failed_runs=$fails enotempty_runs=$enot"

#!/bin/sh
# Stands in for /usr/libexec/git-core/git. Every call goes to the real git, except
# `git maintenance run --auto ... --detach` (which git commit forks): that one returns
# at once and runs the real maintenance 0-90 ms later in its own session. This models
# a detached daemon that the runner schedules late (#249).
case "$*" in
  "maintenance run --auto"*"--detach"*)
    ( sleep "0.0$((RANDOM % 10))"; exec /usr/bin/git maintenance run --auto --quiet ) </dev/null >/dev/null 2>&1 &
    exit 0 ;;
esac
exec /usr/bin/git "$@"

Run: docker create --name gate249-<mode> --cpus 2 -e AMPLIFY=1 -e LATE=1 -e MODE=<mode> -e ROUNDS=10 -e PAR=8 node:22-alpine sh /loop.sh, then docker cp loop.sh, late-daemon.sh and a tar of the worktree (with <mode>.test.mjs variants added) into the container, then docker start -a.

Screenshots (UI changes)

n/a (no UI change).

Coverage

Local, on this head. perf/coverage-* is a shared path, so CI runs every job. The CI summary link follows once it's green.

node perf/coverage-gate.mjs --dotnet TestResults/coverage --web web/coverage --base origin/main → "All coverage gates passed."

Tests

• Unit: perf/coverage-gate.test.mjs 84/84 (30× locally, plus the container runs above); .NET 494/494 (dotnet test -c Release, coverlet.MTP, Testcontainers); web Vitest 104/104, lint clean, build OK
• Integration (Testcontainers): covered by the .NET run above
• E2E (Playwright): n/a
• No new test. The regression check is the existing suite under the race harness: 4/80 runs fail on main and 0/80 with this change.
• gitleaks v8.30.1 (pinned digest) on origin/main..HEAD: 1 commit scanned, no leaks.

Checklist

• Small, single-purpose commits (Conventional Commits)
• No secrets, no real company or person names in the diff
• README §17 Status updated: n/a (no phase status change)
• CI green (build, tests, lint, e2e, budgets)
• Coverage ≥80% on new/changed code; main not dropped (local gate passes; CI pending)
• Helms's sign-off ([workflows])

🤖 Generated with Claude Code