Repository navigation
azure win10 vs2019 machines have been timing out in debugger tests #3014
Description
See nodejs/reliability#354 for reference, it seems the earliest CI run that exhibited this was https://ci.nodejs.org/job/node-test-binary-windows-js-suites/16283/, which is for a PR that hasn't landed rebased on top of a doc commit nodejs/node@dcc9589 so it might be infra issues.
Activity
joyeecheung commentedon Aug 22, 2022
The daily master build on the commit prior to the doc commit was also green on those machines: nodejs/node@5e57d24
joyeecheung commentedon Aug 22, 2022
I do not know how to log into those machines, but I suspect that there might be some stale Node.js processes running there that block the port 9229. That could cause these tests to timeout.
richardlau commentedon Aug 22, 2022
I couldn't see any stale Node.js processes still running on test-azure_msft-win10_vs2019-x64-1 or test-azure_msft-win10_vs2019-x64-2 but I've just scheduled a reboot of both machines.
joyeecheung commentedon Aug 22, 2022
The failures still showed up with nodejs/node#44342, so there must be something else that is causing the time out. Also I noticed that this only happes with test-azure_msft-win10_vs2019-x64-1 and -2, the tests don't fail on -3.
joyeecheung commentedon Aug 22, 2022
Looking at the build history, this seems to happen more with test-azure_msft-win10_vs2019-x64-1, and somewhat less frequently with -2. -3 and -4 seem to be immune to that.
richardlau commentedon Aug 22, 2022
FWIW -1 and -2 did get rebooted (#3014 (comment)) so whatever the problem is it has persisted, or recreated itself, across the reboot.
joyeecheung commentedon Aug 22, 2022
Another observation: it seems sometimes these failures are accompanied by a few random failures from other tests that involve timeouts, e.g. sequential/test-heap-prof, parallel/test-child-process-exec-timeout, see nodejs/reliability#354
tniessen commentedon Aug 23, 2022
It's been failing almost every time, making it difficult to land PRs:
test: raise sleep times in child process tests
joyeecheung commentedon Aug 29, 2022
Looks like the flakes are now gone, the only recent repros in the reliability issues came from resumed builds that weren't rebased. I'll close this issue.
tniessen commentedon Aug 29, 2022
Thank you for working on this @joyeecheung :)
test: raise sleep times in child process tests
test: raise sleep times in child process tests
test: raise sleep times in child process tests
test: raise sleep times in child process tests
test: raise sleep times in child process tests