Using AWS S3 as a simple cache service

S3 is great for file storage, but it does so much more as well. I love using S3 as a simple caching mechanism for any stateless functions that need to save some ephemeral data to keep state.

Traditionally, you would use in-memory caching tools like Redis for this, and Redis does still have its place as it will be faster than retrieving data from S3 in almost every case. However, if milisecond performance is not a concern, S3 is a cheap, low-effort and simple to implement alternative.

Some basic caching helpers

// cache.js

const { S3 } = require('aws-sdk');
const s3 = new S3({ region: 'eu-west-1' });

const { CACHE_BUCKET } = process.env;

const get = async(key, defaultValue = null) => {
  try {
    const { Body } = await s3
      .getObject({
        Bucket: CACHE_BUCKET,
        Key: `${key}.json`
      })
      .promise();
    return JSON.parse(Body.toString());
  }
  catch(e) {
    // File might not exist yet
    return defaultValue;
  }
};

const set = (key, value) =>
  s3
    .putObject({
      Bucket: CACHE_BUCKET,
      Key: `${key}.json`,
      Body: JSON.stringify(value)
    })
    .promise();

module.exports = {
  get,
  set
};

You would use them like this (be sure to define the CACHE_BUCKET environment variable):

const cache = require('./cache.js');

// ...

await cache.set('my-key', { message: 'Hello, world!', boolValue: true });

const value = await cache.get('my-key');
console.log(value); // { message: 'Hello, world!', boolValue: true }

Storage types

In our example code above, we’re storing values as JSON objects. Of course you’re able to store any type of content (binary, plain text) as you see fit for your purpose.

S3 Lifecycle Rules

Using S3 as a cache works especially well in combination with S3 Lifecycle configurations. This will allow you to automatically delete cache resources after a certain period of time has passed.

An example on how to set that up in CloudFormation:

Resources:
  S3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: 'my-caching-bucket' # Set bucket name
      AccessControl: Private # We don't want these cache files to be publicly available!
      LifecycleConfiguration:
        Rules:
          - Id: ExpirationRule
            Status: Enabled
            ExpirationInDays: 2 # Will delete items after 2 days

You can add a Prefix or TagFilters condition to only apply this auto-deletion rule to objects in the bucket that have keys that start with specific prefixes, or attach tags and filter by those.

Extending methods

If you find yourself using this method of caching data a lot, you might want to expand our little cache.js helper with a few additional methods. S3 is quite flexible, so it should be relatively easy to implement functions that:

• Delete cache keys
• Delete cache keys that start with a prefix
• Tag cache keys and allow for deleting or retrieving all keys with a specific tag

Further reading

• AWS::S3::Bucket Rule Cloudformation documentation
• Whitepaper: Best Practices Design Patterns: Optimizing Amazon S3 Performance