Serverless APIs for Web Page Testing

Overview

Let’s go serverless! In this article I am going to develop a serverless API to scrape and test out web pages under different resolutions. This is relatively straight-forward using open source libraries and AWS Lamdba.

Below are some screenshots of Medium.com on what I want to achieve. The serverless function captures the web page in the desired resolutions and responds with an image of the page.

The Code

Let’s use a top town approach to understand the solution. Below is the Lambda function to achieve what I want to do.

Puppeteer is the Node library providing APIs to control and use Chrome/Chromium for crawling and generate images or PDFs of web pages. When you install Puppeteer, it downloads a recent version of Chromium (~170MB Mac, ~282MB Linux, ~280MB Win).

For use with AWS Lambda, I need to use chrome-aws-lambda to avoid the limits imposed. As you can see from package.json below, Puppeteer is installed as a development dependency. If you go through chrome-aws-lambda implementation, you can see that it first check if Puppeetter is available and try to use it. As such for local development Puppeetter is used but when deployed to AWS environment the Chromium binary bundled with chrome-aws-lambda is used.

With the Chromium browser, it is easy to capture the web page in different resolutions and save it as an image using the setViewport andscreenshot method.

Image Handler

With the image captured then I need to encode and return the binary output with the correct MIME type. This is handled by the image handler which either return the binary output as image/png or JSON string with error message whenever there is an issue.

Serverless Framework

I use the serverless framework which lets me develop and deploy serverless applications to AWS, Azure, GCP & more.

Lambda Configuration

For the Lambda function, I configure it in serverless.xml and expose it through API Gateway.

pngScraper:
    handler: scraper.pngScraper
    timeout: 30
    layers:
      - { Ref: ChromeAwsLambdaLayer }
    events:
      - http:
          path: scrape_png
          method: get

Few things to note here

• handler: Javascript file and the function in the file.
• timeout: Default time out is 6 seconds and I configure it to be 30 seconds.
• layers: I package chrome-aws-lamba code into a Lambda layer. More on this later.
• events: Here is where I exposed the Lambda function as API.

For better performance you can also configure the memory size and provisioned concurrency but this may increase your bills. Refer to this documentation for more details.

API Gateway Configuration

Since the output is in binary, I also need to configure API Gateway to return binary media. In my case I configure it to allow all media types but you can be more specific on the media types to return, e.g. image/png, application/pdf, etc. More details are available in the documentation.

apiGateway:
    binaryMediaTypes:
      - '*/*'

AWS Lambda Layers

AWS Lambda layers allow you to configure Lambda function to pull in additional code and content in the form of layers. A layer is a .zip file archive that contains libraries, a custom runtime, or other dependencies.

With layers, you can use libraries in your function without needing to include them in your deployment package. Layers let you keep your deployment package small, which makes development easier. You can avoid errors that can occur when you install and package dependencies with your function code.

In my case, using the following commands from the website, I am able to bundle chrome-aws-lambda into a layer.

git clone --depth=1 https://github.com/alixaxel/chrome-aws-lambda.git && \
cd chrome-aws-lambda && \
make chrome_aws_lambda.zip

With the zip file, I can then configure the layer in serverless.xml.

layers:
  ChromeAws:
    name: ChromeAws
    compatibleRuntimes:
      - nodejs12.x
    description: Chrome AWS Lambda
    package:
      artifact: layers/chrome_aws_lambda.zip

In the Lambda function, I make a reference to the layer.

pngScraper:
    handler: scraper.pngScraper
    timeout: 30
    layers:
      - { Ref: ChromeAwsLambdaLayer }

Since chrome_aws_lambda is bundled into a layer, I exclude the library from the deployment.

custom:
  region: ${opt:region, self:provider.region}
  stage: ${opt:stage, self:provider.stage}
  bundle:
    forceExclude:
      - chrome-aws-lambda
      - puppeteer-core

  serverless-offline:
    location: .webpack/service

Deployment and Testing

To test the serverless API locally, just use the serverless offline or sls offline command.

Development Testing

# sls offline --verbose

.......

Serverless: Using configuration:
{
  "packager": "npm",
  "packagerOptions": {},
  "webpackConfig": "node_modules/serverless-bundle/src/webpack.config.js",
  "includeModules": {
    "forceExclude": [
      "aws-sdk",
      "chrome-aws-lambda"
    ],
    "forceInclude": null,
    "packagePath": "package.json"
  },
  "keepOutputDirectory": false
}
Serverless: Removing /Users/XXX/workspace/development/alpha2phi/serverless/grabql-api/.webpack
Serverless: Bundling with Webpack...
Serverless: Watching for changes...
Serverless: Starting Offline: dev/ap-southeast-1.

Serverless: Routes for pdfScraper:
Serverless: GET /scrape_pdf
Serverless: POST /{apiVersion}/functions/grabql-api-dev-pdfScraper/invocations

Serverless: Routes for pngScraper:
Serverless: GET /scrape_png
Serverless: POST /{apiVersion}/functions/grabql-api-dev-pngScraper/invocations

Serverless: Offline [HTTP] listening on http://localhost:3000
Serverless: Enter "rp" to replay the last request

From the browser, enter the following URL for testing, and you should be able to see the screenshot I showed in the beginning of this article.

http://localhost:3000/scrape_png?url=http://www.medium.com&width=400&height=300

Deployment to AWS

To deploy it to AWS, use the serverless deploy or sls deploy command.

# sls deploy

Serverless: Bundling with Webpack...
Serverless: Excluding external modules: chrome-aws-lambda@^5.5.0, puppeteer-core@^5.5.0
Serverless: No external modules needed
Serverless: Packaging service...
Serverless: Layer ChromeAws is already uploaded.
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Skip uploading ChromeAws
Serverless: Uploading service pdfScraper.zip file to S3 (80.37 KB)...
Serverless: Uploading service pngScraper.zip file to S3 (80.37 KB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
....................
Serverless: Stack update finished...
Service Information
service: grabql-api
stage: dev
region: ap-southeast-1
stack: grabql-api-dev
resources: 18
api keys:
  None
endpoints:
  GET - https://do5jnnqkob.execute-api.ap-southeast-1.amazonaws.com/dev/scrape_pdf
  GET - https://do5jnnqkob.execute-api.ap-southeast-1.amazonaws.com/dev/scrape_png
functions:
  pdfScraper: grabql-api-dev-pdfScraper
  pngScraper: grabql-api-dev-pngScraper
layers:
  ChromeAws: arn:aws:lambda:ap-southeast-1:450266975445:layer:ChromeAws:2
Serverless: Sucessfully updated to v2.20.0

With the provided endpoint I can then test out the API.

In case you want to view any info related to the service, use the sls info command.

Serverless framework makes it very easy to monitor the deployed API. I can use the following command to check the CloudWatch log.

# sls logs -f pngScraper  -t

PDF Generation

Currently the API returns a PNG image. In case you want to generate PDF, use the PDF handler.

Summary

As you can see it is relatively easy to develop, test and deploy serverless API. However, you need to design it carefully as each cloud provider has its limitations. The current API is synchronous and if immediate response is not required, it is good to do decoupling and make it asynchronous. I shall talk more about this in future article.

The source code for this article is available here.

You may also want to check out the following articles.