Skip to content

Use devtoolset-8 on Centos/RHEL for Node.js 14+ #2242

Closed
Closed
Use devtoolset-8 on Centos/RHEL for Node.js 14+#2242

Description

@richardlau
richardlau
opened on Mar 22, 2020

Requires access to the release machines. Install devtoolset-8 on the Centos/RHEL machines and use it to build releases for Node.js 14+.

Refs: #2168

Install devtoolset-8 on centos7:

  • release-osuosl-centos7-ppc64_le-1
  • test-osuosl-centos7-ppc64_le-1
  • test-osuosl-centos7-ppc64_le-2
  • release-digitalocean-centos7-x64-1 :
  • release-packetnet-centos7-arm64-1:
  • test-packetnet-centos7-arm64-1
  • test-packetnet-centos7-arm64-2
  • test-rackspace-centos7-x64-1
  • test-softlayer-centos7-x64-1:

Install on rhel7

  • release-ibm-rhel7-s390x-1
  • test-ibm-rhel7-s390x-1
  • test-ibm-rhel7-s390x-2
  • test-ibm-rhel7-s390x-3
  • test-ibm-rhel7-s390x-4

Use devtoolset-8 for 14.x: PR #2262

select-compiler.sh:

  • rhel7-s390x
  • centos7-ppc64_le
  • centos7-64
  • centos7-arm64

VersionSelectorScript.goovy

  • n/a rhel7-s390x
  • n/a centos7-ppc64_le: ubuntu1404 already excluded for gte 14.x
  • centos7-64
  • centos7-arm64

Jenkins config changes:

  • (n/a) centos7-ppc64_le
  • (n/a) centos7-s390x
  • centos7-64:
    • ci: machines labelled
    • ci: config selected for job
    • ci-release: machines labelled
    • ci-release: config selected for job
  • centos7-arm64
    • ci: machines labelled
    • ci: config selected for job
    • ci-release: machines labelled
    • ci-release: config selected for job

test builds:

Activity

  1. rvagg commented on Mar 23, 2020

    @rvagg
    rvagg
    on Mar 23, 2020
    Member

    do we have ansible changes ready for this to run?

  2. sam-github commented on Mar 23, 2020

    @sam-github
    sam-github
    on Mar 23, 2020
    Contributor

    devtoolset-6, the rpms weren't publically available, but I don't think we've looked yet into the situation for devtoolset-8

  3. sam-github commented on Apr 2, 2020

    @sam-github
    sam-github
    on Apr 2, 2020
    Contributor

    Some questions about this:

    1. Only release machines? So select-compiler.sh will do something different depending on test vs release?
    2. actual machines to be installed are all of following, or only some of them?
      • release-softlayer-centos6-x64-1
      • release-osuosl-centos7-ppc64_le-1
      • release-packetnet-centos7-arm64-1
      • release-ibm-rhel7-s390x-1

  4. sam-github commented on Apr 2, 2020

    @sam-github
    sam-github
    on Apr 2, 2020
    Contributor

    I need #2220 to even look into non-IBM machines (for IBM hosted platforms, I have special access from #1721)

  5. sam-github commented on Apr 2, 2020

    @sam-github
    sam-github
    on Apr 2, 2020
    Contributor

    Can't get it from (some of) the default repos yet, will have to look further afield:

    • test-softlayer-centos7-x64-1: devtoolset-8 is in yum repos 🎉
    • release-osuosl-centos7-ppc64_le-1: devtoolset-8 not available, and the devtoolset-6 repos have disappeared!

    https://cbs.centos.org/repos/sclo7-devtoolset-6-rh-release/ppc64le/os/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found

    • release-ibm-rhel7-s390x-1: devtoolset-8 not available

  6. richardlau commented on Apr 2, 2020

    @richardlau
    richardlau
    on Apr 2, 2020
    MemberAuthor
    * release-osuosl-centos7-ppc64_le-1: devtoolset-8 not available, and the devtoolset-6 repos have disappeared!

    https://cbs.centos.org/repos/sclo7-devtoolset-6-rh-release/ppc64le/os/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found

    Maybe https://cbs.centos.org/koji/buildinfo?buildID=26229?

  7. sam-github commented on Apr 3, 2020

    @sam-github
    sam-github
    on Apr 3, 2020
    Contributor

    Previous link pointed to an entire repo, which included all the rpms, I think the single rpm alone is not sufficient -- but I'll try tomorrow and see how it goes.

  8. rvagg commented on Apr 3, 2020

    @rvagg
    rvagg
    on Apr 3, 2020
    Member

    Only release machines?

    I think we could take the simple approach do this globally, for everything on CentOS. We'll presumably have non-CentOS machines running other versions of GCC. It means we'll have a gap in testing (old GCC on CentOS) but I don't think that's a bad compromise considering the complexity it'll induce if we try and be more exhaustive.

  9. sam-github commented on Apr 3, 2020

    @sam-github
    sam-github
    on Apr 3, 2020
    Contributor

    Installed on release-osuosl-centos7-ppc64_le-1

    [centos@release-osuosl-centos7-ppc64--le-1 tmp]$ ls /opt/rh/
devtoolset-6  devtoolset-8
[centos@release-osuosl-centos7-ppc64--le-1 tmp]$ ls /opt/rh/devtoolset-8
enable  root
[centos@release-osuosl-centos7-ppc64--le-1 tmp]$ . /opt/rh/devtoolset-8/enable 
[centos@release-osuosl-centos7-ppc64--le-1 tmp]$ which gcc
/opt/rh/devtoolset-8/root/usr/bin/gcc
[centos@release-osuosl-centos7-ppc64--le-1 tmp]$ gcc --version
gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

  10. sam-github commented on Apr 3, 2020

    @sam-github
    sam-github
    on Apr 3, 2020
    Contributor

    gpg keys were bad for test-softlayer-centos7-x64-1, fixed with advice from https://jc-lan.org/2019/12/14/error-when-updating-centos-7-gpg-keys-are-not-correct-for-this-package/

    Now it fails with "At least 32MB more space needed on the /boot filesystem."

  11. sam-github commented on Apr 3, 2020

    @sam-github
    sam-github
    on Apr 3, 2020
    Contributor

    test-rackspace-centos7-x64-1 doesn't accept git as a package name, once I pushed a fix for that onto #2262 , it installed devtoolset-8

  12. sam-github commented on Apr 3, 2020

    @sam-github
    sam-github
    on Apr 3, 2020
    Contributor

    Looks like changes that install devtoolset-6-libatomic-devel don't work on test-packetnet-centos7-arm64-2. Its not clear why, its not a clear package-not-found, or, perhaps it is? Its trying to get it from http://centosn7.centos.org/altarch/7.7.1908/sclo/aarch64/rh/devtoolset-6, which doesn't exist.

    But, that doesn't seem to be an issue, we seem to have been doing fine without it, so I moved it out of the centos7 common in #2262

    FTR:

    
[root@test-packetnet-centos7-arm64-2 yum.repos.d]# yum install devtoolset-6-libatomic-devel    
Loaded plugins: fastestmirror                                                                  
Loading mirror speeds from cached hostfile                                                     
 * base: packet04.centos.org                                                                   
 * centos-sclo-rh: centosn7.centos.org                                                         
 * epel: ewr.edge.kernel.org
 * extras: centosm7.centos.org
 * updates: centosg7.centos.org
Resolving Dependencies                         
--> Running transaction check
---> Package devtoolset-6-libatomic-devel.aarch64 0:6.3.1-3.1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved                          

===============================================================================================
 Package                           Arch         Version              Repository           Size
===============================================================================================
Installing:                                    
 devtoolset-6-libatomic-devel      aarch64      6.3.1-3.1.el7        centos-sclo-rh       15 k

Transaction Summary                            
===============================================================================================
Install  1 Package                             

Total download size: 15 k
Installed size: 110 k                          
Is this ok [y/d/N]: y                          
Downloading packages:                          
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
devtoolset-6-libatomic-devel-6 FAILED                                           
http://centosn7.centos.org/altarch/7.7.1908/sclo/aarch64/rh/devtoolset-6/devtoolset-6-libatomic-devel-6.3.1-3.1.el7.aarch64.rpm: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror. 
... a dozen other failing mirrors

    EDIT: after the fix, installed devtoolset-8 OK.

  13. sam-github commented on Apr 3, 2020

    @sam-github
    sam-github
    on Apr 3, 2020
    Contributor

    @rvagg I'm having trouble accessing an arm release machine, could you check the release key is authorized for it? I see:

    core/secrets (master % u=) % ssh release-packetnet-centos7-arm64-1 ls /opt/rh
root@147.75.104.218: Permission denied (publickey).
core/secrets (master % u=) % grep -A 5 release-packetnet-centos7-arm64-1 ~/.ssh/config 
Host release-packetnet-centos7-arm64-1 
  HostName 147.75.104.218
  IdentityFile ~/.ssh/nodejs_build_release
  User root

    EDIT: same on release-digitalocean-centos7-x64-1

  14. sam-github commented on Apr 3, 2020

    @sam-github
    sam-github
    on Apr 3, 2020
    Contributor

    release-packetnet-centos7-arm64-1 has two issues:

    1. Its jenkins secret is not in the inventory
    2. I can't ssh in
    task path: /home/sam/w/core/build/ansible/playbooks/jenkins/worker/create.yml:28
fatal: [release-packetnet-centos7-arm64-1]: FAILED! => {
    "msg": "The conditional check 'not secret' failed. The error was: error while evaluating conditional (not secret): 'secret' is undefined"
}

PLAY RECAP ************************************************************************************************************************************************************************************
release-packetnet-centos7-arm64-1 : ok=0    changed=0    unreachable=0    failed=1   

build/ansible (centos7-devtoolset-8 $ u=) % ssh release-packetnet-centos7-arm64-1
root@147.75.104.218: Permission denied (publickey).

  15. rvagg commented on Apr 6, 2020

    @rvagg
    rvagg
    on Apr 6, 2020
    Member

    @sam-github can you try with an explicit -i ~/.ssh/nodejs_build_release on that ssh? I'm getting in fine and the release pubkey is in .ssh/authorized_keys. You'll have to find the jenkins node secret from jenkins itself or on the machine itself to put it in secrets repo. When inventory.yml was originally set up with secrets, many of them weren't added in (because it was tedious) so we're all responsible for adding them in as we find them missing.

  16. 7 remaining items

  17. richardlau commented on Apr 8, 2020

    @richardlau
    richardlau
    on Apr 8, 2020
    MemberAuthor

    I think maybe we need a gcc -v; g++ -v before make in these builds. I can't tell for sure what version it's invoking after the devtoolset setup and ccache environment variable prepend. It looks correct but I'd have to grab a binary to be sure.

    I could change the configure script to log the versions when run with --verbose (which I think we do on the CI). The script already parses the compiler versions to check and warn if the detected compiler is not new enough.

    Edit: nodejs/node#32715

  18. richardlau commented on Apr 8, 2020

    @richardlau
    richardlau
    on Apr 8, 2020
    MemberAuthor

    With nodejs/node#32715:

    first ppc build: https://ci.nodejs.org/job/node-test-commit-plinux-sam-github/1/nodes=centos7-ppcle/

    https://ci.nodejs.org/job/node-test-commit-plinux-sam-github/2/nodes=centos7-ppcle/

    12:20:54 Detected C++ compiler (CXX=ccache ppc64le-redhat-linux-g++) version: 8.3.1
12:20:54 Detected C compiler (CC=ccache ppc64le-redhat-linux-gcc) version: 8.3.1

    first s390x build: https://ci.nodejs.org/job/node-test-commit-linuxone-sam-github/2/nodes=rhel7-s390x (well, second, first didn't have the select-compiler update)

    https://ci.nodejs.org/job/node-test-commit-linuxone-sam-github/3/nodes=rhel7-s390x/

    12:25:04 Detected C++ compiler (CXX=ccache s390x-redhat-linux-g++) version: 8.3.1
12:25:04 Detected C compiler (CC=ccache s390x-redhat-linux-gcc) version: 8.3.1

    first attempt at an x86 gcc8 build: https://ci.nodejs.org/job/node-test-commit-linux-sam-github/1/nodes=centos7-64-gcc8

    https://ci.nodejs.org/job/node-test-commit-linux-sam-github/2/nodes=centos7-64-gcc8/

    12:26:11 Detected C++ compiler (CXX=ccache g++) version: 8.2.1
12:26:11 Detected C compiler (CC=ccache gcc) version: 8.2.1

  19. rvagg commented on Apr 8, 2020

    @rvagg
    rvagg
    on Apr 8, 2020
    Member

    sweet, thanks @richardlau, lgtm then.

  20. sam-github commented on Apr 8, 2020

    @sam-github
    sam-github
    on Apr 8, 2020
    Contributor

    v12.x: node-test-commit-linux, all platforms: https://ci.nodejs.org/job/node-test-commit-linux-sam-github/3/

  21. sam-github commented on Apr 8, 2020

    @sam-github
    sam-github
    on Apr 8, 2020
    Contributor

    @rvagg https://ci.nodejs.org/job/node-test-commit-arm/30615/

    Is it intended that 14.x build against ubuntu1604-arm64?

    Its not excluded for any arm builds ATM, but it's EOL, I think it should have been excluded, or at least, should be excluded for anyType gte(14.x), and perhaps even 12.x. Note that its excluded from release right now by virtue of no machine having that label, so doesn't need .groovy exclusion.

    I am pushing a commit to exclude ubuntu14-04 onto #2262 , tell me if I got it wrong.

    cc: @nodejs/platform-arm @nodejs/build

  22. rvagg commented on Apr 9, 2020

    @rvagg
    rvagg
    on Apr 9, 2020
    Member

    No, we should probably remove 1604-arm64.

  23. sam-github
    added 2 commits that reference this issue on Apr 9, 2020

    ansible: install devtoolset-8 on centos7

    e0f5ca2

    ansible: install devtoolset-8 on centos7

    bec0ae2

  24. sam-github commented on Apr 9, 2020

    @sam-github
    sam-github
    on Apr 9, 2020
    Contributor

    #2262 landed, making ci/ci-release config changes.

  25. sam-github commented on Apr 9, 2020

    @sam-github
    sam-github
    on Apr 9, 2020
    Contributor

    Configured, kicked off test builds of 3 relevant jobs for master+12.x

  26. sam-github commented on Apr 9, 2020

    @sam-github
    sam-github
    on Apr 9, 2020
    Contributor

    kicked node-test-commit-arm off on master again, it was missing the bits of select-compiler.sh that are pasted into it, see #2279

  27. sam-github commented on Apr 9, 2020

    @sam-github
    sam-github
    on Apr 9, 2020
    Contributor

    arm and x86 for iojs+release required some extra bits in their config, rerunning.

  28. sam-github commented on Apr 9, 2020

    @sam-github
    sam-github
    on Apr 9, 2020
    Contributor

    https://ci-release.nodejs.org/job/iojs+release/5882/nodes=centos7-arm64-gcc8/consoleFull looks OK:

    12:50:12 gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)

    https://ci-release.nodejs.org/job/iojs+release/5882/nodes=centos7-ppcle-release-64/consoleFull looks OK:

    12:50:02 gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)

    https://ci-release.nodejs.org/job/iojs+release/5882/nodes=centos7-64-gcc8/consoleFull (x64) looks OK, but is still building:

    13:47:42 gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)

    https://ci-release.nodejs.org/job/iojs+release/5882/nodes=rhel7-s390x-release/consoleFull looks OK:

    12:50:01 gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)

  29. sam-github commented on Apr 9, 2020

    @sam-github
    sam-github
    on Apr 9, 2020
    Contributor

    I think this is done, but it could use a sanity check.

  30. richardlau
    closed this as completedon Apr 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions