Repository navigation
CI reports a test file as passing while silently dropping some of its tests, including failures #661
Description
tests/run.mts can report a test file as fully passing while silently discarding some of that file's tests, including failing ones. A real regression is on dev right now behind a green ci-aggregate.
Observed
At dev head d98e1dcc2, tests/unit/slack-mpim-dispatch.test.ts contains 6 tests. The two runners disagree:
npx tsx --experimental-test-module-mocks tests/run.mts tests/unit/slack-mpim-dispatch.test.ts
tests 2 pass 2 fail 0
npx tsx --experimental-test-module-mocks --test tests/unit/slack-mpim-dispatch.test.ts
tests 6 pass 4 fail 2
The four missing tests are not skipped — their bodies execute. The driver run prints their side effects ([slack:in] GMPIM: help four times) before reporting only two results. Their outcomes, including two assertion failures, are dropped.
CI uses the driver, so run 34322526725 on d98e1dcc2 reported success on all four shards, integration, gates and windows-unit while two tests were failing.
Shape that triggers it
The file declares two synchronous tests, then performs a chain of top-level await import(...) plus mock.module(...) setup, then declares four more tests. Deleting only the two early synchronous tests makes the remaining four report correctly:
tests 4 pass 2 fail 2
So the file's report is finalized once the early-registered tests drain, and everything registered after the top-level await chain is executed but uncounted. run() is called with isolation: 'process' and forceExit: !watch (tests/run.mts:89), which is the difference from tsx --test.
The trigger is narrower than "any top-level await". A two-line synthetic reproducer (one sync test, one await import, one failing test) reports correctly under both runners, and four other files with the same mix — tests/events.test.ts (87 before / 19 after), web-native-requests, manager-electron-desktop-contract, slack-table-content — report identically under both. The exact condition is not yet characterized; the heavier async top-level setup appears to matter.
Why it matters
94 test files declare a test before a top-level await import and more tests after it, so the at-risk population is large even though most are currently unaffected. More importantly the failure mode is silent and unbounded: a file can lose coverage at any time through an ordinary edit, and ci-aggregate cannot tell the difference.
Suggested direction
A cheap backstop is to make the count authoritative: have the driver compare each file's reported test count against an expected count, or fail a file whose process exits with tests still registering. Even a CI step that re-runs the suite under --test and diffs total counts would surface this class.
Concrete consequence
The dropped failures hide a real defect; see #651.
Found while auditing whether #651 could be closed. Both runners were executed locally at d98e1dcc2; no files were modified on dev.
Activity
fix(tests): stop the test driver from silently dropping late-register…
ci(test): give the shard step room for the driver's stall watchdog
lidge-jun commentedon Sep 9, 2026
Fixed on
dev.Cause.
tests/run.mtspassedforceExittorun(), which appends--test-force-exitto everyisolation:'process'child. A child with that flag exits as soon as its root test drains the tests it already knows about (lib/internal/test_runner/test.js), so anything a file registers after a top-level await that yields to the macrotask queue executes uncounted. A microtask-only await never loses anything, which is why the two-line reproducer in the report did not reproduce.Scale. Whole
root,unitscope, same commit: 12957 → 12981 tests on Node 22 (CI's version), 12916 → 12981 on Node 24.17.0. Three consecutive runs of the old driver on one commit lost 35, 4 and 4 tests —tests/events.test.tslost 31 in one run and none in the next. It is a race, so the number was never stable.Fix. #664 (
2be76dd23) removesforceExitand replaces the stall protection it was carrying with a per-file watchdog in the driver: a file whose child has started reporting and then goes quiet forJAW_TEST_FILE_STALL_MS(default 180s) is named — with its last event channel, event count and elapsed silence — fails the run and aborts the rest.tests/unit/test-runner-driver.test.tsruns the real driver against four fixtures and fails ifforceExitreturns.What it immediately caught. On its first CI run the watchdog reported
tests/unit/manager-design-routes.test.ts — last test:pass 180818ms ago after 25 event(s): a genuine hang the old driver had been hiding.fs.watch(root, { recursive: true })plusunref()does not release the loop on Linux, because node's non-native recursive watcher only unrefsStatWatcherentries and never the per-directory handles it keeps adding (nodejs/node#53350; fixed onmainonly, nodejs/node#65486). #666 (6d52e0245) switches that watcher topersistent: false.ci-aggregateis green ondevwith all four shards reporting the recovered tests.