Repository navigation
Running tests with --test-coverage-branches (same for lines|functions) doesn't emit test:fail when threshold isn't met #54812
Description
Version
22.8.0
Platform
Darwin N4V4PGFGPT 23.6.0 Darwin Kernel Version 23.6.0: Mon Jul 29 21:16:46 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T8112 arm64
Subsystem
No response
What steps will reproduce the bug?
I'm running Node with the following options:
--experimental-test-coverage --test-coverage-exclude='**/__tests__/**' --test-coverage-branches=95 --env-file=.env.test
and calling an executable that calls the run method as by doc.
If a test fails, a test:fail event is properly emitted.
If the coverage threshold is not met, I get a message from the test:diagnostic event like Error: 82.35% function coverage does not meet a threshold of 95%. (I get the same on the stdout). But no test:fail event is emitted, which makes it harder to detect it.
How often does it reproduce? Is there a required condition?
Always
What is the expected behavior? Why is that the expected behavior?
test:fail event should be emitted once coverage is not meet
What do you see instead?
no test:fail event emitted
Additional information
same thing apply when using --test-coverage-functions or --test-coverage-lines options
Activity
avivkeller commentedon Sep 6, 2024
I'm able to reproduce. I'll have a look at this later today.
FWIW the process does exit with code 1, however all the tests do pass.
CC @nodejs/test_runner
cjihrig commentedon Sep 6, 2024
A
test:failevent should not be emitted for coverage issues. There is atest:coverageevent that would be more appropriate.rozzilla commentedon Sep 6, 2024
Mmm, well, the CHANGELOG (not the docs, since they're not updated yet) says:
Considering that, it feels more like a
test:fail. Anyway, if there is a clear indication of coverage check failures somewhere, it'd be enough 👍🏼cjihrig commentedon Sep 6, 2024
test:failmeans a test failed. It's already possible to end up with a failing exit code independent of coverage. For example, if a test passes, but creates asetTimeout()or other async activity that generates an error. The test has already finished and reported itself as being successful. So the error gets surfaced through a diagnostic and the exit code is set to 1. That's basically what is happening in the code coverage case as well. I agree that it should be signaled in the coverage event though.avivkeller commentedon Sep 6, 2024
See #54813
test_runner: add 'test:summary' event
test_runner: add 'test:summary' event