Repository navigation
AIX CI machines cannot access raw.githubusercontent.com #2330
Description
This is looking like some sort of infrastructure issue where the AIX CI machine doesn't appear to be able to download https://raw.githubusercontent.com/ URLs.
e.g.
Running AIX CI on v12.x to check if that works: https://ci.nodejs.org/job/node-test-commit-aix/30710/
(Note this build passed)
...
+ curl https://raw.githubusercontent.com/nodejs/build/master/jenkins/scripts/node-test-commit-pre.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0+ bash -xe
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
...
0 0 0 0 0 0 0 0 --:--:-- 0:01:14 --:--:-- 0
curl: (28) Failed to connect to raw.githubusercontent.com port 443: A remote host did not respond within the timeout period.
...
if [ -x /home/iojs/build/workspace/node-test-commit-aix/nodes/aix71-ppc64/./node ] && [ -e /home/iojs/build/workspace/node-test-commit-aix/nodes/aix71-ppc64/./node ]; then /home/iojs/build/workspace/node-test-commit-aix/nodes/aix71-ppc64/./node tools/doc/versions.js out/previous-doc-versions.json; elif [ -x `which node` ] && [ -e `which node` ] && [ `which node` ]; then `which node` tools/doc/versions.js out/previous-doc-versions.json; else echo "No available node, cannot run \"node tools/doc/versions.js out/previous-doc-versions.json\""; exit 1; fi;
Unable to retrieve https://raw.githubusercontent.com/nodejs/node/master/CHANGELOG.md. Falling back to /home/iojs/build/workspace/node-test-commit-aix/nodes/aix71-ppc64/CHANGELOG.md.
...
The tools/doc/versions.js tool falls back to the local copy of the CHANGELOG.md for non-release commits (which is why only the release commit build failed) to accommodate users with poor/firewalled Internet connections.
cc @nodejs/platform-aix
Originally posted by @richardlau in nodejs/node#33197 (comment)
Activity
sam-github commentedon May 26, 2020
Update: we have reached out internally to the infra providers to ask what's going on, it looks like some kind of off-machine firewall is banning the traffic. Waiting for a response, still.
sam-github commentedon May 26, 2020
We got a response, and unfortunately, its deliberate.
raw.githubusercontent.com is apparently a commonly blacklisted site (I assume because its used sometimes by bad actors as a CDN for malware). Avoiding using it in our CI would be quite onerous to us. The network specialists are in contact with GH to ask their opinion of this.
mhdawson commentedon May 28, 2020
@sam-github I believe this is now resolved right?
sam-github commentedon May 28, 2020
Builds are green now: https://ci.nodejs.org/job/node-test-commit-aix/30771/
The provider sec team resolved something with GH, and unblocked it.
targos commentedon Jun 2, 2020
This is still happening with today's security releases: https://ci.nodejs.org/job/node-test-commit-aix/30869/nodes=aix71-ppc64/console
sam-github commentedon Jun 2, 2020
I confirmed with a manual
curl https://raw.githubusercontent.com/nodejs/build/master/jenkins/scripts/node-test-commit-pre.sh, and sent an email to the infra provider (CCing @mhdawson @AshCripps @richardlau ).sam-github commentedon Jun 2, 2020
At this point, I think we have no choice but to go forward with the releases, and then release AIX afterwards. Agreed @mhdawson ?
BethGriggs commentedon Jun 2, 2020
Looks like the Release builds worked (assuming they don't depend on that domain) - https://ci-release.nodejs.org/job/iojs+release/nodes=aix71-ppc64/
targos commentedon Jun 2, 2020
@sam-github this doesn't impact the release build. Only
node-test-commitjobs run on release commits.richardlau commentedon Jun 2, 2020
Yes, the release build aren't affected because they don't access that domain.
Just to clarify being unable to contact raw.githubusercontent.com causes the following two issues for the test CI:
jenkins/scripts/node-test-commit-pre.shscript which handles things like rebasing. This doesn't appear to fail the build when it fails to download but does mean the script isn't executed (so if you were trying to rebase you may not be build/testing what you thought you requested).tools/doc/versions.js. For non-release commits this results in a warning and then fallback to the local changelog file and does not fail the build. For release commits the doc tool hard fails if it cannot grab the latest changelog (because the local changelog for older release lines has no references to the later release lines).One of the issues with this issue is that for non-release commits the builds are not failing so we don't know this has happened between releases unless we are scanning the job output of passing builds (i.e. it was fixed, but now appears to have come back but we didn't know until we tried to test a release).
sam-github commentedon Jun 2, 2020
OK, great that the release is not impacted, that is a relief.
Though I do wonder if the curls should be mandatory, so that we do notice when they break!
Of course, that would then maybe block the release.... sometimes available infrastructure is hard to deal with. I'm not sure what the right thing to do here is, longer term.
richardlau commentedon Jun 2, 2020
I'm not going to start messing around with the job config while the security release is being prepped, but for the first issue with the
node-test-commit-pre.shscript the current flow is:curlto get the latest version of the script and pipe it tobashto execute.post-build-status-updatejob is rungitto clone down https://github.com/nodejs/build.git (so it can run theselect-compiler.shscript).So we could move the
git cloneof the build repo to before running thepost-build-status-updatejob and then executenode-test-commit-pre.shfrom the locally cloned files.For the second issue options we can do include:
CHANGELOG.mdfrom GitHub (via the raw.githubusercontent.com URL) we could use one of the index files from nodejs.org (e.g.index.json) on the basis that (hopefully) nodejs.org is less likely to be blacklisted than raw.githubusercontent.com.NODE_TEST_NO_INTERNETenvironment variable (note despite the name of the env var it's not the tests that are failing here, it's the actual docs build) to affect release commit builds and set it in the AIX test job.richardlau commentedon Jun 3, 2020
I've made modifications to the node-test-commit-aix job so that the
git cloneof the build repo happens earlier and we use the checked out copy ofjenkins/scripts/node-test-commit-pre.shinstead of attempting to usecurlon the raw.githubusercontent.com URL. I've also added the following to set theNODE_TEST_NO_INTERNETenvironment variable if we cannot reach raw.githubusercontent.com:Test build on the v10.21.0 release commit passed but revealed that the
jenkins/scripts/node-test-commit-pre.shscript does anothercurl:https://ci.nodejs.org/job/node-test-commit-aix/30880/nodes=aix71-ppc64/consoleFull
There's a TODO comment in the script saying to run locally, so I'll look into addressing that.
build/jenkins/scripts/node-test-commit-pre.sh
Lines 42 to 43 in 9cf3680
richardlau commentedon Jun 5, 2020
I've drafted a change in #2342 but I'll need to clone some of the node-test-commit-* jobs and edit the copies to use that PR to test it hasn't regressed other platforms and does address this issue on AIX.
richardlau commentedon Jun 9, 2020
Landed #2342. Together with the job configuration changes we are no longer attempting to use raw.githubusercontent.com URLs except for one place where we test to see if we can access it (and only set the
NODE_TEST_NO_INTERNETenvironment variable to shortcut the part of the docs build that uses it if we cannot).