How To Speed Up Continuous Integration Build With New NPM CI And package-lock.json

While very controversial, the recent npm release 5.7.0 brought some amazing features which will have noticeable positive impact on your development workflow!

From official changelog…

The new npm ci command installs from your lock-file ONLY. If your package.json and your lock-file are out of sync then it will report an error.It works by throwing away your node_modules and recreating it from scratch. Beyond guaranteeing you that you’ll only get what is in your lock-file it’s also much faster (2x-10x!) than npm install

Generate and use package-lock.json file

I have hard time guessing what ratio of developer is using package-lock.json already. It wasn’t behaving intuitively when first introduced and I suppose many people kept on deleting the file to prevent unnecessary headache.

Current behavior is luckily much more in line with what is expected by most developers. Running npm install will generate package-lock.json file if it didn’t exist with the versions from current node_modules . Manual bumping of versions in the package.json will result in correct version bumps in already existing package-lock.json when using npm install.

On the other hand, newly introduced npm ci ignores package.json (only throws error if they are out of sync) and install dependencies as specified in package-lock.json .

How to enable faster install in Travis CI environment

Travis CI is great CI server with seamless integration with public Github repositories so it’s a very popular choice among the OSS projects.

Travis CI runs couple of build steps for every triggered build. One of them is the install step which runs npm install for all node projects by default. Luckily there is a simple way to override this default configuration.

We have to add install section and use npm ci instead.

As of 23. 2. 2018 npm v5.7.1 is still in pre-release so we will not get it by default in Travis CI environment and we have to install it manually. This step won’t be necessary in the future.

UPDATE: As of 26. 2. 2018 Travis CI still uses npm v5.6.0 by default but there have been other npm releases so we can use v6.1.0 instead of v5.7.1 as originally specified in the article

The updated .travis.yml will look something like this…

Build time improvements

This configuration resulted in almost 3 times faster installation of dependencies saving more than 80 seconds for small to mid sized Angular CLI project.

Almost 3 times faster installation of dependencies achieved by small change in configuration is just amazing!

Similarly, fresh local install of checked out project now takes ~70 instead ~170 seconds!

Resulting builds are faster and the dependencies are guaranteed to be installed in a consistent way across different environments from local dev machines to CI servers.

Follow me on Twitter to get notified about the newest blog posts and interesting frontend stuff

What about Yarn ?

Running yarn install takes ~90 seconds on the same machine which is 20 seconds compared to running npm ci .

Why is it faster?

From what I understand the difference in performance between npm installand npm ci is that npm install has to perform more checks and resolution of module versions compared to npm ci.

Running install in fact IS the way to generate and update package-lock.json . This implies that there is a bit more going on than just grabbing specified versions.

On the other hand, running npm ci just deletes node_modules folder and installs versions exactly as specified in package-lock.json .

Besides that, I have seen in some Twitter posts and the plan for the future seems to be convergence between them so that npm install is as fast as npm ci when the conditions are right.

Other CI servers

Lately I have been using mainly Travis CI but there are many other great CI servers like Jenkins, Teamcity, Bamboo and others which support wide range of flexible configurations.

In case you’re using other CI server, simply adjust step which is responsible for preparing node environment to use specific ( 5.7.1 ) npm version and npm ci command to install dependencies.

What about reported security issues?

Original 5.7.0 release introduced very serious issue and corresponding HN thread with comments like…

It seems to have completely broken my filesystem permissions and caused me to have to manually fix the permissions of critical files and folders.

This destroyed 3 production server after a single deploy!

Luckily this behavior was fixed by following 5.7.1 release with official response…

This release reverts a patch that could cause some ownership changes on system files when running from some directories when also using sudo.

So please, proceed with caution and preferably test in non-critical environments first.

What if I am using Travis CI caching for node_modules folder?

As generously shared by the Tim Allen and Morgan Roderick, in this twitter thread, Travis CI builds can be made faster by leveraging caching feature.

Travis CI can preserve whole node_modules folder between the builds. Every subsequent build then performs just incremental changes on top of that during npm install which may lead to even faster builds on some projects.

In contrast, npm ci always deletes and re-fetches all dependencies as specified in package-lock.json . This should in theory prevent any possible corruption of the already installed dependencies.

As always, use what makes most sense for your particular situation

And we’re done!

I hope you will use this tip to get much shorter build times for your projects too! Please support this article with your 👏👏👏 to help it to reach wider audience and follow me on 🕊️ Twitter to get notified about newest blog posts 😉

Also, feel free to check some other interesting frontend & Angular posts…

And never forget, future is bright