Temporarily disabling node-test-commit-arm-fanned #2227

Activity

1. mhdawson commented on Mar 18, 2020

   

   mhdawson

   on Mar 18, 2020

   Member

   More actions

   Off topic but @rvagg I'm wondering if you have a feel if PI4's are fast enough that we could avoid "fanned" and the need for so many machines?
2. rvagg commented on Mar 18, 2020

   

   rvagg

   on Mar 18, 2020

   MemberAuthor

   More actions

   @mhdawson not entirely sure but I think until Pi gets alternative storage it's always going to be a problem for workloads like ours. SD is just not a fast medium, but it's cheap, which is why they do it.

   I think I got a 4 to play with but never ran a Node build on it, maybe I should try.
3. mhdawson commented on Mar 19, 2020

   

   mhdawson

   on Mar 19, 2020

   Member

   More actions

   @rvagg is that true even with the nfs mounts you've been using?
4. rvagg commented on Mar 19, 2020

   

   rvagg

   on Mar 19, 2020

   MemberAuthor

   More actions

   @mhdawson nfs seems to speed things up a little but it's not a silver bullet since we have a fleet of machines all hammering the same network and same disk with small file reads and writes, and nfs isn't exactly an efficient protocol either.

   maybe the cpu and ram improvements in a pi4 would improve the situation though.
5. mhdawson commented on Mar 20, 2020

   

   mhdawson

   on Mar 20, 2020

   Member

   More actions

   @rvagg thanks, understood that if we still have large numbers of machines its still a problem. I don't have a PI 4 yet to experiment but I'll have to think about getting one.
6. rvagg commented on Apr 3, 2020

   

   rvagg

   on Apr 3, 2020

   MemberAuthor

   More actions

   Tended to the Pi cluster today and arm-fanned is now re-enabled with all machines running, connected, updated, cleaned and some runs for major release lines pushed through to re-prime caches and look for hiccups.

   Noteworthy items:

   • I've taken half of each type of Pi offline, they are marked as "RESERVE" with a note that they should be enabled when others are disabled. I'm doing this to save disk on the shared NFS host. One we move past Node 10 we'll ditch the Pi 1's and have lots more disk to play with. @Trott you're the most active with taking Pi's offline - feel free to enable a RESERVE Pi when you do to add more bandwidth. You can see the list easily by label: pi1-docker, pi2-docker & pi3-docker.
   • There was a bug in the arm-fanned config that was causing test-binary-arm and test-binary-arm-12+ to be run for Node 10, so Pi 2 & 3 tests were running twice. That should be fixed now.
   • addons/openssl-binding/test very flaky on ARM node#30786 is outstanding and will probably flare up again now ARMv7 is back in CI. There's some suggestions in there about how it might be addressed, additional perspectives or solutions (in PR form!) welcome.
   • We seem to have introduced a perma-fail test in the time that this job was offline known_issues/test-crypto-authenticated-stream times out on Pi 2 and 3 from master (Pi 1 not run for master). See below. Ping to @nodejs/crypto @nodejs/testing on this I guess.

   19:52:57 not ok 655 known_issues/test-crypto-authenticated-stream
   19:52:57   ---
   19:52:57   duration_ms: 240.118
   19:52:57   severity: fail
   19:52:57   exitcode: -15
   19:52:57   stack: |-
   19:52:57     timeout
   19:52:57   ...
   
7. rvagg

   closed this as completedon Apr 3, 2020
8. bnoordhuis commented on Apr 3, 2020

   

   bnoordhuis

   on Apr 3, 2020

   Member

   More actions

   known_issues/test-crypto-authenticated-stream times out on Pi 2 and 3

   Not really a surprise. I opened nodejs/node#32636 to disable it on ARM.