Azure: Pipeline caching

Exploring pipeline caching on Azure and it’s shortcomings

In late 2019, Azure DevOps finally released a feature to enable caching on pipelines. Pipeline caching on other platforms like AWS, Bitbucket is pretty straightforward and works like a charm.

Pipeline caching on Azure is slightly different and I’d say it’s still a work in progress. This means on some occasions it might not work as expected.

This article will explain how pipeline caching works on Azure DevOps and address the gotchas.

Cache@2 Task

This task has 3 input arguments -

1. key (required): unique key to identify the cache
2. path (required): path of the folder or file you want to cache
3. restoreKeys (optional): If key fails, this will be the fallback to find cache

Examples

1. Caching NPM’s shared directory (Build)

2. Caching Cypress binary (Test)

Cypress is a test framework whose binary goes into hundreds of megabytes (500–800 MB) and hence this below example shows how to cache it.

3. Caching with conditions

In most cases, you’d want to skip some steps if the cache is successfully restored (hit). For example, a step that installs dependencies can be skipped if the cache was restored. This is possible using the cacheHitVar task input.

Post-Job: Cache Task

Once you add caching to any build pipeline, Azure automatically adds a post-job cache task.

Don’t be surprised by this additional automated step. It takes care of uploading cache contents (the first time) & checks if the contents weren’t changed through the course of your build.

Gotcha #1

Cache not working? or not getting a cache hit?

Make sure your build is passing. If it fails, caching won’t work and post-job cache step “passes” however it doesn’t do anything (no logs is an indication).

Gotcha #2

Cache hit/restore still not happening? Check if you see this warning in the Post-Job Cache step.

##[warning]The given cache key has changed in its resolved value between restore and save steps;

If yes, then make sure none of your build steps between the Cache and Post-Job Cache step change the cache key value.

For instance, we use the below value as the cache key for our NPM example.

npm | $(Agent.OS) | ${{ parameters.WORKING_DIR }}/package-lock.json

So let’s say there is a npm publish step in the build which changes the value of package-lock.json. In this case, you’ll never get a cache hit because package-lock.json (hash) is part of the unique cache key.

There are two workarounds — either remove the change causing step or remove package-lock.json from the key.

Gotcha #3

The documentation states to use ${Pipeline.Workspace} for defining relative paths on the hosted build agent.

However, not all folders on the hosted build agent are exposed via default variables. Therefore at times you might need to specify the absolute path like the above snippet — /home/vsts/.npm

Gotcha #4

Caching has restricted scopes at a project, pipeline and branch level.

Example: Let’s say you added caching for Pipeline A on develop branch. Now, if you create a new branch called feature/A and open a pull request against develop. The first pipeline run for this branch will have a “Cache miss” because it’s a new branch.

Thus caching is less useful when used with short-lived feature branches.

*This is mentioned in the docs but can be missed

Gotcha #5

On Windows build agents caching seems to be very slow for cache sizes greater than some threshold. This defeats the whole purpose of caching as you add caching to speed up builds and not the other way around.

So if you see builds taking longer than what it used to before caching, don’t worry you haven’t done anything wrong. It’s a bug which will be addressed soon.

Other FAQ’s

That’s it folks, happy caching! Feel free to drop in any questions in the comments.

If you liked this article on Azure, you might find the below article helpful too