The best way to package your AWS Lambda function’s dependencies

The most frequently encountered challenge with AWS Lambda, as seen on Stack Overflow, is the ‘No module found for import.’ This often arises from inadequately packaged dependencies for Lambda functions. If you’re seeking the best method to package your dependencies for a Lambda function, you’re in the right place.

Lambda merely provides the execution environment, be it for Python, JavaScript, or any other language. You’re responsible for specifying the packages your Lambda function requires. It’s a common misconception that popular packages like ‘boto3,’ essential for programmatically accessing AWS services, are automatically included in the Lambda execution environment. However, nothing is included by default, except native Python libraries like os, sys, and logging.

For instance, if you’re working on a Lambda function to read Parquet data from S3 and load it into RDS PostgreSQL, you’ll likely need packages like ‘boto3’ for AWS service interaction and ‘pyarrow’ for handling Parquet files.

There are two ways to package your Lambda function code. One option is to include both your Lambda function and its dependencies in a single archive for upload. The other option, which we’ll explore in this article, involves separating the dependencies and core functionality by using Lambda layers.

We’ve opted for the second option as it promotes reusability of Lambda layers across multiple functions. This approach allows you to manage the core functionality in the AWS console while updating dependencies separately in the foreground. Unlike the first option, where code and dependencies are bundled together, this separation makes it easier to handle and view the code in the foreground.

Unit testing Lambda code locally before uploading to the cloud can be challenging. Option 1 may lead to multiple code uploads during the build process. Therefore, we’ve chosen Option 2 for its support, flexibility, and layer reusability.

Let’s dive in and package the necessary dependencies for a function that reads data from S3 and loads it into RDS. This article will focus on creating the Lambda layer, providing a comprehensive guide for developers.

1. Get a AMI Linux 2 Ec2 instance.

2. Connect to the Ec2 Instance.

3. Execute the following code in the EC2 command line

• Install the python version based on the Lambda execution environment.

sudo amazon-linux-extras install python3.8

• Get/Download the script to install pip in the EC2 Instance

curl -O https://bootstrap.pypa.io/get-pip.py

• Install the ‘pip’ for the current user .

python3.8 get-pip.py — user

4. Create the directory named python.

mkdir python (I)

It is not optional to name the directory as python , it is mandatory why?

5. Install the dependencies in the python directory.

python3.8 -m pip install boto3 psycopg2-binary cryptography -t python/

6. Zip the dependencies

zip -r <Any name>.zip python

e.g. zip -r python-boto3-psycopg2-crypto-layer.zip python

I have named it is as ‘python-boto3-psycopg2-crypto-layer’ as it will make sense for me when I need a same packages for other lambda function and can reuse it.

7. Copy the Zipped file to s3.

aws s3 cp python-boto3-psycopg2-crypto-layer.zip s3://abc-dev01-incoming-files

Oops! Error!

When you didn’t attach any role (i.e. permissions to access any AWS services) to EC2, It will try to search for IAM credential (sercret key id and key) is present in AWS config. The following error depicts that credntial error.

The EC2 instance you created doesn’t have permission to write to S3, in our case we need to push this zip file to S3 as subsequently we are going to point this location and create a Lambda layer. And, creating lambda layer or pushing the new version of layer also EC2 will not have permission.

Okay, how do we fix it?

EC2 Instance needs to assume the role when it is calling other resources of AWS, to do that we need to create a policy and attach that policy to the role.

Create the Policy (lambda-layer-push) with the following definition

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::abc-dev01-incoming-files",
                "arn:aws:s3:::abc-dev01-incoming-files/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "lambda:PublishLayerVersion",
                "lambda:GetLayerVersion",
                "lambda:UpdateFunctionConfiguration",
                "lambda:GetFunctionConfiguration",
                "lambda:DeleteLayerVersion"
            ],
            "Resource": "*"
        }
    ]
}

Let’s break down the policy you provided:

"Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::abc-dev01-incoming-files",
                "arn:aws:s3:::abc-dev01-incoming-files/*"
            ]
        }

This policy allows access to read/write/list the (s3:GetObject/s3:PutObject/s3:ListBucket) objects in the bucket arn:aws:s3:::abc-dev01-incoming-files.

{
            "Effect": "Allow",
            "Action": [
                "lambda:PublishLayerVersion",
                "lambda:GetLayerVersion",
                "lambda:UpdateFunctionConfiguration",
                "lambda:GetFunctionConfiguration",
                "lambda:DeleteLayerVersion"
            ],
            "Resource": "*"
        }

• lambda:PublishLayerVersion: Permits publishing a new version of a Lambda layer.
• lambda:GetLayerVersion: Allows fetching information about a Lambda layer version.
• lambda:UpdateFunctionConfiguration: Grants the ability to update the configuration of a Lambda function.
• lambda:GetFunctionConfiguration: Enables fetching the configuration details of a Lambda function.
• lambda:DeleteLayerVersion: Allows the deletion of a specific version of a Lambda layer.

Create the role (lambda-layer-push-role) and attach the policy (lambda-layer-push)

Trust policy (what are all the resources can assume this role), in our case EC2 so the policy would be (It will be auto created when you pick EC2 as service during the role creation).

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Okay, now let us link this role to our EC2 instance (select the instance, go to security, and Modify IAM role)

select the role you just created and update.

All set , let us try to copy our layer to s3.

Awesome, now it is done.

Let us create the layer and push our layer version 1.

aws lambda publish-layer-version --layer-name python-boto3-psycopg2-crypto-layer --content S3Bucket=abc-dev01-incoming-files,S3Key=python-boto3-psycopg2-crypto-layer.zip --compatible-runtimes python3.8 --region ap-southeast-2

Successfully, created the layer:python-boto3-psycopg2-crypto-layer we can use it in the lambda function now.

Note: AWS Lambda has a maximum deployment package size limit of 250 MB, which includes the function code and any additional layers. This limitation is in place to ensure efficient deployment and execution of Lambda functions in the AWS environment

Have questions or thoughts? Don’t hesitate to share them in the comments below! If you found this article helpful or enjoyable, show your appreciation by giving it a round of applause 👏. Happy learning and looking forward to hearing from you!