Repository navigation
Unable to build Nomad with Nodejs v16 LTS #11795
Description
Nomad version
1.1.6, 1.1.9 and 1.2.3
Operating system and Environment details
Alpine Linux v3.15 and Edge
Issue
Unable to build Nomad using Nodejs v16 - this version of Nodejs became LTS on 26th October as per https://nodejs.org/en/about/releases/
Appears to be a known Yuidoc issue: yui/yuidoc#453
Related Alpine issues:
https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/28610
https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/28611
Reproduction steps
Expected Result
Actual Result
error: An uncaught YUIDoc error has occurred, stack trace given below
error: -------------------------------------------------------------------------
-
error: TypeError [ERR_INVALID_ARG_TYPE]: The "options" argument must be of type object. Received an instance of Array
at ChildProcess.target.send (node:internal/child_process:713:7)
at Array.forEach (<anonymous>)
at dispatchQueuedRequests (/home/builder/package/src/nomad-1.1.6/ui/node_modules/ember-cli-test-loader/node_modules/workerpool/lib/WorkerHandler.js:174:21)
at ChildProcess.<anonymous> (/home/builder/package/src/nomad-1.1.6/ui/node_modules/ember-cli-test-loader/node_modules/workerpool/lib/WorkerHandler.js:129:7)
at ChildProcess.emit (node:events:390:28)
at ChildProcess.emit (node:domain:475:12)
at emit (node:internal/child_process:917:12)
at processTicksAndRejections (node:internal/process/task_queues:84:21)
error: --------------------------------------------------------------------------
error: Node.js version: v16.13.1
error: YUI version: 3.18.1
error: YUIDoc version: 0.10.2
error: Please file all tickets here: http://github.com/yui/yuidoc/issues
error: --------------------------------------------------------------------------
Activity
DerekStrickland commentedon Jan 7, 2022
Thanks for filing this issue! We'll take a look at it as soon as possible.
ChaiWithJai commentedon Jan 7, 2022
Hey, this is likely because we're using
ember-cli: 3.20. We need to upgrade 2 LTS's toember-cli: 3.28. This requires dependency upgrades that would need to be tackled in a major release1.3.For now, we can only support
Node v12. This issue has been on our radar and a candidate to tackle in1.3.dermotbradley commentedon Jan 7, 2022
@ChaiWithJai The Alpine current release (3.15.x) has Nomad 1.1.1 and NodeJS 16.13.1 packaged so we're stuck as Nomad 1.1.9 cannot be packaged for the recent security fixes due to this issue...
dermotbradley commentedon Jan 29, 2022
@ChaiWithJai
Any indication of timescales for a Nomad 1.3 release containing support for the current NodeJS LTS?
Alpine is still stuck with Nomad 1.1.1 in its current "stable" release (v3.15.0) and this Nomad version obviously does not have the security fixes from 1.1.2-1.1.9.
Should we remove Nomad completely from the current stable Alpine release? I see no other way to address the security implications of shipping a version with known CVEs.
We cannot switch NodeJs back to the previous LTS release as other packages in Alpine v3.15.0 may be relying on NodeJS 16.x.
tgross commentedon Jan 31, 2022
Hi @dermotbradley!
Generally speaking we don't give firm timelines for releases, because lots can happen between now and then. But our rough plan is on the scale of "a couple months" rather than "a couple days".
While I'm definitely sympathetic to this... it doesn't seem like security patches are really relevant to this issue, are they?
Here's where you may need to fill in for us a little bit in terms of Alpine processes around stale builds of packages, but my understanding is that the distro won't ship packages that it can't currently build. (Which is good!) So the problem isn't that you can't package newer versions of Nomad, but that the NodeJS package in Alpine stable has been bumped out from under us. So no versions of Nomad can be packaged until we bump the dependency. Even if there were no security patches or you were to backport them, you can't continue to package Nomad with our required version of NodeJS.
I see a few options here:
But I totally understand that either of those options might not work with Alpine's packaging policies! In which case, yes, you might need to remove Nomad from Alpine stable until we've released 1.3.0. That's unfortunate but we'll definitely make sure we get a newer NodeJS LTS by then (Node 12 is towards the end of maintenance anyways).
As an aside: Nomad has some CGO dependencies and we frankly don't test with musl at all. I'd love to hear more about how Alpine validates the package after the build, as that might help influence us to put the work in to do the same validation and that'll get Nomad into the hands of more Alpine users without them doing terrible things like installing glibc in Alpine. 😀
loqs commentedon Mar 8, 2022
I encountered a similar failure with vault. The change in nodejs causing the issue is 1. The fix for workerpool is 2. nomad 1.2.6 uses workerpool 2.3.3 3 the same as vault 1.9.4. I was able to get vault to build with nodejs 16 by vendoring a patched workerpool 2.3.3, I did not have to patch workerpool 3.1.2. @dermotbradley perhaps that can provide you with a solution until 1.3.0 is released.
dermotbradley commentedon Apr 21, 2022
@tgross Sorry for taking so long to respond again to this.
So Alpine does package more than one version of NodeJS - the current LTS ("Active LTS" as NodeJS call it: https://nodejs.org/en/about/releases/), which is v16, and 1 other newer non-LTS version, currently v18 as it just came out but was v17 up until a couple of days ago.
The underlying issue appears to be that Nomad (and also Vault) in general are lagging in the LTS version that they support - v12 goes out of support in 9 days time, whilst v14 is still in support for another year I see no reason to select it rather than v16 which has been the current LTS for almost 5 months now.
The same issue was raised by someone else with the Vault team some time ago: hashicorp/vault#8504
I see someone in that Vault issue ticket pointing out that RHEL/CentOS 9 also only provides NodeJS v16 so the underlying issue is not specific to Alpine.
Currently the Nomad and Vault packages for Alpine have been built without a UI to avoid this issue. However Consul does not appear to have the same issue - it seems that Consul is built for Alpine, complete with UI, without NodeJS being used.
It is not clear if Nomad v1.3.x will officially support NodeJS v16 - I have compiled v1.3.0-beta.1 successfully with that but have not yet had a chance to test it. Can you indicate what the status is regarding supported NodeJS version(s) for v1.3.x?
NodeJS v18 will be the next LTS in 6 months time so I suspect the same problem will reoccur then - v18 was released a couple of days ago and so the next 6 months should give ample time for projects to prepare for it becoming the new Active LTS in October. As Alpine v3.17 is due to be released approx 1 month afterwards then I expect that Alpine version to ship with NodeJS v18.
So I'm not sure whether we should keep the UI disabled or not going forward. I'd hate to re-enable it for Nomad 1.3.x and then have to disable it again after the next NodeJS LTS comes out.
One potential solution I think someone suggested is whether the Nomad source tarball could ship with the UI artifacts already present, so avoiding the need to use NodeJS when building from source.
Basically there is no validation apart from whether it builds on all the supported archs. In the specific case of Nomad it hasn't actually had an active maintainer since it first was packaged back in late 2019 - unfortunately it sometimes happens that someone creates a package, updates it a few times, and then stops maintaining it (others have updated the Nomad package during the past 2 years but not taken over active maintenance of it). I am intended to change this for the Alpine Nomad package.
You can see the 6 archs that it is currently packaged for here: https://pkgs.alpinelinux.org/packages?name=nomad&branch=edge
tgross commentedon Apr 21, 2022
Hi @dermotbradley! Just for some background, we're currently in the process of migrating our release infrastructure to a new HashiCorp-wide centrally managed release tooling. So there's some stuff in flux here as we migrate away from each product team having unique processes and you're getting caught up in that a bit. In the near future you should start seeing more consistency across products.
Also note that we support two previous major versions of Nomad at a time (so with Nomad 1.3.0's release we'll continue support for Nomad 1.2.x and Nomad 1.1.x). Although sometimes we have to upgrade our oldest supported Nomad's toolchain because of the lifecycle of the toolchains, we really try to avoid it if we can. So unless a NodeJS version goes entirely out of support and we need to ship a backported bug fix, we won't upgrade it in the older Nomad versions.
Agreed! Unfortunately we just didn't manage to get to the upgrade for 1.3.0-beta.1. There's always a bunch of decisions around priorities we have to make and this didn't quite get above the line.
We built Nomad 1.3.0 with NodeJS v14 and that'll most likely be the version we ship with for the entire 1.3.x cycle. Our test infrastructure doesn't currently test the UI with different versions of the toolchains (go or NodeJS), so we can't really speak to whether NodeJS v16 will actually work until we're ready to do the upgrade ourselves. We can only actively support the toolchains we build on but if you have patches to contribute for forwards compatibility that don't break the current builds, we're always more than happy to get those.
The source tarball under the GitHub release includes the bundled assets. You can find that at
./command/agent/bindata_assetfs.gowithin the tarball for a given release (ex [v1.3.0-beta.1]https://github.com/hashicorp/nomad/releases/tag/v1.3.0-beta.1)). I suspect some other package maintainers maybe won't care for that as a solution because of their distribution's policies. But if that works for you, great!