Repository navigation
[BUG] npm ci gives timeout error on gitlab runner #7076
Description
Is there an existing issue for this?
- I have searched the existing issues
This issue exists in the latest npm version
- I am using the latest npm
Current Behavior
We are running using the node:18 docker image in our CI pipeline running on self hosted Gitlab runner. The runner is hosted on a VM running on GCP and our gitlab is running on GKE v.127.
We started having this error a week ago. This is not a runner issue since we are able to install packages using pip and mix deps.get for elixir and only npm fails. We are not using any proxy and internet is reachable from docker containers.
The runner is configured with the docker executor.
The exact error is:
$ npm ci
npm ERR! code ETIMEDOUT
npm ERR! syscall connect
npm ERR! errno ETIMEDOUT
npm ERR! network request to https://registry.npmjs.org/yargs-parser/-/yargs-parser-20.2.9.tgz failed, reason: connect ETIMEDOUT 104.16.24.34:443
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly. See: 'npm help config'
npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2023-12-12T04_38_02_576Z-debug-0.log
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
We were able to solve this issue by setting network_mode="host" in the gitlab runner's config but this breaks the container services communication which we need for the builds.
Expected Behavior
Running npm ci should fetch the packages from npm registry correctly.
Steps To Reproduce
- Setup Gitlab self hosted on GKE cluster v.127 and add a runner on a GCP VM.
- Using this Gitlab CI snippet:
.app-build: &app-build
stage: app-build
image: node:18
before_script:
- npm ci
script:
- npm run build- Produces the error
Environment
- npm: 10.2.3
- Node.js:
node:18docker image - v18.19.0 - OS Name: Debian 11
- npm config:
; node bin location = /usr/local/bin/node
; node version = v18.19.0
; npm local prefix = /builds/our/project
; npm version = 10.2.3
; cwd = /builds/our/project
; HOME = /root
; Run `npm config ls -l` to show all defaults.
Activity
zheng1716148634 commentedon Dec 12, 2023
yes，me too；I also experience timeout errors；I tried many times, but it didn't work；
Larsjep commentedon Dec 12, 2023
Hi,
This might be related to #7072 . We saw a similar problem on our internal proxy cache due to high number of connections opened by npm.
lxfu1 commentedon Dec 13, 2023
The same problem.
spookyintheam commentedon Dec 13, 2023
Hi @Larsjep, thanks for pointing out your issue. Just wanted to confirm how did you check how many connections
npm installwas opening?Larsjep commentedon Dec 13, 2023
I'm using wireshark and noticing when and how many SYN packages it sends.
spookyintheam commentedon Dec 13, 2023
I see. Using
node:20image seems to fix this issue for us but I only tested this on a single project. I do not know if this breaks any dependencies for us right now so will need to test more on other projects as well.I'd be interested in knowing what the root issue in here, from npm maintainers if they get around to this.
heath-freenome commentedon Dec 19, 2023
We were seeing the same issue on our
gitlabrunners and we were forced to pinned node to18.18.xto avoid it. This seems to be an issue with node10.x. Locally when I'm doing annpm ciI'm seeing some packages taking minutes to be installed (are there timeouts that keep retrying when on the command line?). There seems to be a big issue with the latest version ofnpmin terms of performance. NOTE: I tried my localnpm ciwith10.2.5and it takes 5+ minutes to install files. We have a local GAR based registry for some of our libraries. Could this be part of the problem?Now that I'm trying to upgrade our systems to
Node 20I'm afraid we'll start encountering these issues again, soon. Any word from the developers on when this can be fixed?spookyintheam commentedon Dec 19, 2023
We had this problem on using the
node:18docker image and moving tonode:20on some CIs seems to have fixed this we're also concerned with dependency issues in case we decide to move tonode:20completely.rosen-dimitrov commentedon Dec 20, 2023
Hi guys,
posting here as well for visibility. Looks like an issue with the new
@npmcli/agentpackage and the way it handles network connections.#7072 (comment)
michalpidanic commentedon Feb 11, 2024
Hello guys, has somebody solved this issue? Im having the same problem, using node:20, my npm version is 10.2.4 and im getting the same error. I tried to do
npm install -g npm@9in before_script however, it fails with same error 🙃 I really dont know how to solve this problem and it is already been few weeks. My runner and job has latest node images which seem to be shipped out with npm 10, so little bit of deadlock here 🙃dahei commentedon Feb 26, 2024
We face the same issue in our Gitlab pipelines and were forced to pin our node version to
18.18.1(which includes npm9.8.1).The difference also becomes obvious when running
npm installlocally. I tried 3 different versions on the same project.Result:
18.18.2 (npm 9.8.1) -> 38s
18.19.1 (npm 10.2.4) -> 48s
20.11.1 (npm 10.2.4) -> 2m13s (!)
In both installs with npm 10.x I could see long running requests for some packages as stated in the other comments. It's not always the same package, so it feels like a npm 10.x issue.
Within the Gitlab CI pipeline it becomes a serious blocker as the jobs take much longer and often fail.
melroy89 commentedon Mar 10, 2024
You should try to use:
npm ci --cache .npm --prefer-offline. And cache the.npmdirectory if possible in gitlab. But I digress.Despite all this, when using this npm ci command above with cache and prefer offline, my gitlab ci/cd is constantly failing at random on one or more jobs. It's very frustrating!
And I don't understand why npm doesn't give this issue more attention. As you notice a lot of developers are effected by this.
heath-freenome commentedon Apr 5, 2024
I'm still seeing issue with this on my gitlab pipelines as well, inside of Dockerfiles within the CI/CD pipelines. I am using
10.5.0:melroy89 commentedon Apr 5, 2024
Because it's not yet released. See: nodejs/node#52351
IceBjerg commentedon Apr 17, 2024
Hey everyone!
Just wanted to post here, maybe some of you might have an idea.
I am using node v16.20.2, and npm v8.19.4.
I somehow get this error, when executing npm install with the help of a gitlab runner.
I don't have any idea what's going on... The error message says basically nothing, I don't see status codes other than 200 in the verbose logs. Is this truly a network issue? I have increased the fetch retries to 5, however, I don't see any retries actually happening (or at least, not logged then).
What I see is that fetches slow down over time. Before it would crash, I see this kind of logs:
npm timing reifyNode:node_modules/jsdom Completed in 23901msWhy am I not seeing retries?
I also changed the
maxsocketsconfig to 5, still not seeing any result.My problem is strange, because my jobs not always fail! Even when increasing the
maxsocketsconfig to 50, I have the same, ~20% probability of the job failing due to timeout.Anyone? Anything?? Would be appreciated very much.
melroy89 commentedon Apr 17, 2024
I have the feeling I'm repeating myself. But this npm fix is not yet part of nodejs. See open PR until further notice: nodejs/node#52505
Then update to nodejs v22. V16 is end of life.
sdalonzo commentedon Apr 23, 2024
@IceBjerg try
--no-progress. we had an issue with long urls to a private npm registry and a memory leak in the progress bar, and it had the notable symptom of hanging onreifyfix pipeline
mehdicopter commentedon Jul 29, 2024
Hi folks, any news regarding this issue ?
I do have time outs. I am using runner within k8s and this is the command i perform:
melroy89 commentedon Jul 29, 2024
Are you behind a corporate firewall? You might need to setup a proxy or use a local artifactory server assuming you are within a business.
mehdicopter commentedon Jul 29, 2024
Nope I am using GKE, privates nodes, I have a Cloud Nat in order to access to Internet
milaninfy commentedon Jun 13, 2025
Closing. I believe this appears to be fixed on latest as some of the recent version contains fixes for overall improvements. If this is still an issue feel free to create new issue or reopen.