Repository navigation
Jobs with flaky tests show as green #467
Description
See discussion in nodejs/node#8065 (comment)
Looks like jobs that have tests marked as flaky show as green instead of the expected yellow. At the same time the post of results shows that tests failed.
Activity
joaocgreis commentedon Aug 15, 2016
There are two issues here:
joaocgreis commentedon Aug 15, 2016
About 1:
Jobs need a post step to mark the job as unstable. Only the windows job had that, so I copied it to all others. Seems to have worked in all but osx and arm fanned: https://ci.nodejs.org/job/node-test-commit/4563/ . However, this seems to be a problem with
test.py, not Jenkins: note that tests are not marked as flaky in the console output (cc @jbergstroem @Trott )About 2:
Could be a bug in the code to do this in the AIX job. Didn't check carefully, but seems to be different from others (e.g. test-linux). EDIT: Checked carefully and made the job similar to others, might be fixed, but there are no AIX servers to test now.
Trott commentedon Aug 21, 2016
Possibly/likely related: Most jobs respect tests that are marked as flaky, but the Raspberry Pi arm-fanned job seemingly does not. This is quite a problem for the recent test-crypto-timing-safe-equal.
not-an-aardvark commentedon Aug 21, 2016
I think the issue is that the Raspberry Pi build is the only one that's failing; it might not be the only job that's not respecting flaky tests.(edit: never mind, see below)Trott commentedon Aug 21, 2016
@not-an-aardvark That's what I mean. (Or maybe I'm misunderstanding you?) On the other jobs, when that test fails, it turns them yellow rather than red. That's the result of that test being marked flaky. But on the Raspberry Pi, that test failing turns the job red.
not-an-aardvark commentedon Aug 21, 2016
Never mind, I hadn't realized some of the other tests turned yellow. I had thought the Raspberry Pi tests were the only ones that failed.
mhdawson commentedon Sep 6, 2016
@joaocgreis In terms of the jobs being green instead of yellow, that is now fixed so this issue may be able to be closed.
joaocgreis commentedon Sep 6, 2016
This issue still exists for
osxandarm-fanned. I suspect everything is OK on the Jenkins side and the failure is intest.py, but was not yet able to investigate further.joaocgreis commentedon Sep 19, 2016
The
FLAKY_TESTS_MODEvariable was not passed on those two jobs. Fixed now. Credit to @rvagg , thanks!