Automating a Lambda Function Deployment
ACM.52 Creating a Lambda function to generate a Job ID
Part of my series on Automating Cybersecurity Metrics. Lambda. Network Security. The Code.
Free Content on Jobs in Cybersecurity | Sign up for the Email List
In the last post I explained that we are going to create a cryptographically secure job ID.
We’ll generate this ID in an AWS Lambda function. We’ll extend this function in the next post. This is all part of the cloud security architecture I’ve been working on step by step.
I already explained how you can create a batch job manually in the AWS console:
In this post we’re going to take a look at automating our Lambda Function with CloudFormation:
This will be very simple Lambda function so we won’t need much of what is listed here:
You can read the details of each configuration item in the above documentation, but here’s what we will need:
Architectures: arm64
I explained in the prior post that arm is a bit less expensive and will work for our needs in this function.
Code
We’ll specify the code for this simple function in-line. There are different ways to deploy the code for your Lambda function but I’m keeping is simple for this initial post.
FunctionName
We’ll give our function a name.
Handler
The handler is the name of the function or method in our code that the Lambda function initially invokes. This value is required in our case.
You would think that the value you pass into the handler in the CloudFormation stack should match the name of your code function, but when you’re adding code in-line in the CloudFormation template the value is “index.[your function name].”
So in our case we’re going to write some code with a method called “handler(event, context)” and we need to set this value to “index.handler”.
Role
We will need to specify a role if the Lambda function. This is required, even though at the moment we don’t to add any permissions. We will need to add permissions in the next post.
Runtime
We will specify that we want to use the latest version of Python.
I wrote a bit about this post on the Lambda runtime in this post on Container Runtimes:
VpcConfig
I explained how a VPC Configuration may be useful for security reasons in a prior post on LambdaNetworking, but we’re going to leave this off for the moment.
KMS and Environment variables
I’m not going to use environment variables here so we don’t need a KMS key, but one of the things I check for on cloud security assessments and pentests is exposed environment variables. If you use environment variables, use a KMS key to encrypt them and restrict access to the Lambda role — in almost every case. More on this in an upcoming post.
Create the Lambda Role
The first step will be to create our Lambda role. We already created a Lambda role template for this purpose so we can just reuse that:
We can update our deployment script for roles with the following command:
lambdaname='GenerateBatchJobId'
deploy_lambda_role $lambdaname $profileNow run the deploy script:
./deploy.sh
Recall that this CloudFormation creates a role with a trust policy that allows a Lambda function to assume the role.
Create the Lambda Role Policy
Our job needs a policy, even if blank so we’ll create that and add it to our Lambda folder.
Navigate to the folder where we are creating Lambda functions and create a new folder for our batch job:
mkdir GenerateBatchJobId
We can copy the code from our other batch job and modify the permissions to save time.
mkdir GenerateBatchJobId/cfncp TriggerBatchJob/cfn/TriggerBatchJobLambdaPolicy.yaml \
GenerateBatchJobId/cfn/GenerateBatchJobIdPolicy.yamlEdit the new policy to remove the permissions in it we don’t need. The interesting thing here is that we don’t need any permissions yet for this Lambda function but we have to add a statement to our policy.
What can we do? Add a deny all policy statement.
Copy the deploy script from our existing job to the new batch job:
cp TriggerBatchJob/deploy.sh GenerateBatchJobId/deploy.shEdit it and add the new policy deployment code.
Python code to generate a random job id
Now we can add some code to our python function. I wrote about creating a cryptographically secure random value for our job ids in the last post.
Luckily the Python documentation has a code sample for exactly what we need to do:
The warning at the end warns us to never store passwords in plain text. What we are creating here is move of an ID than a password but it is used to get into a system. More on that recommendation to follow.
We can put the following code into a test file with a .py extension:
Test the code few times and yes, it does produce a random value:
By the way, I should mention the the longer the value you create, the harder it will be for an attacker to “brute force.” I explain that concept further in my book at the end of this post.
CloudFormation Template
Next we can create our CloudFormation template to deploy our job.
Note that there are multiple ways to deploy Lambda files. We can put the code inline within our CloudFormation template. We could create a zip file with the code init. Alternatively we could create a docker container. I’m going to start with the simplest approach for now. I’ve added the Lambda code to my CloudFormation template. Generally I would recommend one of the other options to separate application code from infrastructure code.Here’s our template:
Deployment script revisions
Add a deployment script to deploy our Lambda function and role
Is the above code going to deploy successfully? No. Why not?
Who is allowed to deploy Lambda functions?
We are using our IAM profile which is only allowed to deploy IAM resources. So we’ll need to think about who, in our organization we want to be able to deploy application code such as Lambda functions and Batch job code.
We could add this permission to our developers group since they are the ones writing the code. But what will we do when we get to the QA and production environments? We would need a separate script to add the role to QA in a QA environment and Prod users in a prod environment if you decide to allow any users to manually run CloudFormation stacks.
A better approach: Define a role that is allowed to deploy applications, and deployment permissions for application code. That way you can allow different people to assume the role in different environments. You might not allow anyone to assume the role in production at all. You might have a deployment system that uses that role instead.
Hopefully once the code leaves production the deployments are automated and use the role without human interaction, but if we are requiring MFA to allow the deployment to proceed, the QA team would provide those credentials in a QA environment. An ops team could provide the credentials in a production environment.
If the ops team cannot change the code but only execute deployment jobs, that would prevent attacks such as the Solar Winds breach because hopefully your QA team would noticed the inserted code during the testing process and the integrity of the code can be checked end to end as it flows through your software development lifecycle (SDLC).
Here’s where I differ from what I consider an insecure approach to DevOps by some who argue about the definition. To me having Dev and Ops mean the same team that writes the code deploys the code in production is a security nightmare waiting to happen. There are reasons why this segregation of duties exists in many security standards. It also is common sense if you stop to think about things like insider threats and preventing inadvertent errors.
My definition of DevOps: You write code to automate your operations and deployment functions. This automation occurs in a development environment. It is validated and tested in a QA environment by a QA team. It is used in a production environment by an operations team.
You test your deployments in QA in such a manner that QA can’t change any of the code to make sure your deployment code works. You may also test your deployments in a staging environment that matches production to ensure your production deployment is not going to fail due to some configuration in production that wasn’t in QA.
By the time the code is ready to deploy in production you know it will deploy successfully. You can also verify your code hasn’t changed from the point it left your development environment to prevent security incidents like the Solar Winds breach.
Application Deployment Permissions
Let’s say we are going to name the cli profile “appdeploy” and add that to our code for now. We’re also going to change the “service name” to AppDeploy. I don’t know if service name is the right term for this prefix on our CloudFormation stacks. Perhaps it should be “group name” but in any case it’s a set of stacks related to a certain group that is allowed to deploy them.
Create a role for application deployments
What’s the easiest way to create a new role for application deployments? We can create a new group and role with a few lines of code. Assuming we are in a development environment now we add the developer to the new group.
In the IAM directory structure, navigate to the group deployment script.
Add a line to add a new group and three lines to add the user to the group.
Create a Policy for the group so it can assume the new group role:
In the role deployment script add one line to add the new role.
Create a policy for the role that allows deployment of AppDeployment CloudFormation stacks. The name will need to match what our generic deployment stacks expect which is the role name with “GroupRolePolicy.yaml” at the end.
AppDeploymentGroupRolePolicy.yaml
Run the deployment scripts in each directory.
./deploy.sh
Problems with AWS CloudFormation, Bash and Parameter-Overrides
I’ve written before about having problems with spaces, bash, and parameter-overrides here:
I also wrote about how someone has been trying to sabotage the search engine rankings of that post for whatever reason and how I’m trying to fix it here:
Something odd happened while testing my code for this post. When I tried to create the new IAM resources above, I started getting errors with code that has been working up till now. I test all the code before I check in and made no changes to that code so why was it failing now? Did someone at AWS make a change? Or did I really just not see this blatant error all the other times I tested?
Who knows. I just want to prevent it in the future. In order to ensure all my code related to parameters works exactly the same way across the board, I created a new shared function. Whatever change I made while testing some other change to the shared functions broke other functions that were passing in parameters in a slightly different way.
The function creates the parameter string which gets passed into the function that deploys a CloudFormation stack:
I put this in my shared_functions.sh file I wrote about in an earlier post in this series so it’s already automatically included in my deploy.sh file.
I add parameters to the parameter string like this now:
Then I use that string like this when creating my stack:
I got rid of the deploy_iam_stack function I left in to keep my code functioning. That was in part, the source of the error. I had to change all my deployment stacks to stop using that and use the common stack deployment function instead in my common functions file.
I still need to revise the way parameters are passed in a few places. But for now, my app deployment role and adding users to it is working now.
Update: When I went to use the above code in my KMS deployments where I have a KMS key description with spaces, I realized there was still a bug in this code. See the upcoming posts and next push to GitHub for updates. This bug really just demonstrates that Bash is not a good choice for this purpose and hopefully in the future I will resolve this another way, but we have to start somewhere in a new AWS account.Verify the IAM deployment
Verify that all your resources were created successfully.
- User Group named AppDeployment
- Developer is in the user group
- The AppDeployment group has a policy with the expected permissions
- AppDeployment role exists
- AppDeployment role has the expected policy
Always verify your deployment. This is where I found out I had the errors I didn’t initially see that I just wrote about and a few others that I had to fix.
Set up the new CLI profile.
Now set up the role profile as explained in prior posts so our Lambda deployment can assume the new role. I wrote about that here:
Deploy the Lambda Function
Now we can attempt to run the Lambda deployment script. If you have set up your AWS CLI profile correctly with a policy defined as above you would see this error next:
Our Lambda function role needs the iam:PassRole action. What is that? That is a whole topic for another post. For now, I will just mention that you should be very careful and explicit when granting permission to pass roles to compute resources. For now, return to the AppDeploy policy and allow that role to pass the Lambda role to the Lambda function.
Test our Lambda Function
Now you can return to the console and test your Lambda function the same way we did in this prior post and see that yes, our code generates a random ID.
Consider the approach
Now, you may have been able to log into the console and quickly drop in some code and deploy this Lambda function. But what we’ve done throughout this series is not just “build things” but to think about how we can build, deploy, and operate our applications securely in a cloud environment. That takes advanced preparation and thought over and above writing the code itself.
I’m not done with this Lambda function. Simply deploying a code is not that useful. I have a reason for generating this code. Follow for updates.
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2022
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab