Fixing Application Errors Inside a Container Used By a Lambda Function and Redeploying It
ACM.301 The benefits and risks of redeploying a container used by a Lambda function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Check out my series on Automating Cybersecurity Metrics | Code.
🔒 Related Stories: Lambda | Container Security | Application Security
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the last post we looked at errors caused by architecture mismatch between a container and the hardware running your Lambda function.
After we resolved that error we got a different error which I happen to know is an error with the configuration of the code inside the container. That configuration is defined by the Dockerfile used to deploy the container, not the Lambda.yaml CloudFormation template we created to deploy our Lambda function.
In this post, I’m going to show you something really cool about deploying Lambda functions with containers. It’s also kind of scary if you don’t understand the security implications of this coolness. Let’s start by taking a look at the error message and resolving it.
Fixing container configuration errors in a Dockerfile
We got past the architecture issue in the last and got an . What is this “app” error all about?
Well as it turns out, copying code from online online examples might not always work so well if you miss a step. Actually, I didn’t miss it but I renamed a file that I should not have without adjusting other parts of the configuration.
Here’s the example I’ve been using:In the demo code in the post above the author created a little Python program in a file called app.py.
That’s the code that is going to run when you execute the container and hence, the Lambda function that uses the container.
It looks like he’s copying that file into the Docker container using this command:
You can read about the COPY command here:
In the directory on the EC2 instance where I built the dockertest image, I’m going to add a directory called app and add the app.py file to that directory.
mkdir app
vi app.py
#copy and paste the example code above into app.py
#save the file
<esc>:wq!Then I’ll rebuild the docker file using my build.sh file that has the docker command in it.
#docker build -t docker-image:dockertest .
./build.sh
Now I can run my push.sh script and pass in that image ID to push the image to ECR.
./push.shI created this push script in an earlier post. I’m working towards reusable scripts but for now it’s in the directory where I’m testing this image.
Verify the ECR repository has a new version of the docker file.
Re-deploying a Lambda function that uses a container
Now here’s where it gets interesting. If you redeploy the Lambda template, CloudFormation does not recognize the underlying container has changed. The template has not changed, so CloudFormation reports that there are no changes to deploy.
What’s really cool is that you can update your containers without redeploying your Lambda function. That’s good because recent tests with a Lambda function using a container showed that the deletion of the Lambda function took 15–20 minutes. Deploying the new function took about 3 minutes. Ugh. Put in an #awswishlist request to speed that up.
But luckily you can update your Lambda function simply by uploading a new container if all you need to do is update the code. Here’s how it works:
After you upload a new or updated container image, Lambda optimizes the image before the function can process invocations. The optimization process can take a few seconds. The function remains in the Pending state until the process completes. The function then transitions to the Active state. While the state is Pending, you can invoke the function, but other operations on the function fail. Invocations that occur while an image update is in progress run the code from the previous image.
The caveat to the above is that if you are monitoring for changes to your infrastructure and applications using CloudFormation drift detection, you’ll need another way to monitor for those changes. The idea of immutable infrastructure needs some additional thought when it comes to using containers for Lambda functions, but I’ll save that for later.
I still think this is a bonus and I’m definitely going to try it out and possibly use it across the board for cloud deployments. I would love to be able to use the same container for Lambda, Batch, and Fargate, for example.
File permissions in containers
The next error I get says that permission is denied to access my file.
Recall that I created a local folder named app and put the app.py file in it. The Dockerfile copies that file into /home/app/app.py.
The location is defined at the top of the Dockerfile:
The FUNCTION_DIR variable is used in the COPY command:
There’s a command to change the permission of the entry point here:
However, I don’t see a command to change the permissions of the app.py
Let’s change the permissions of the whole app folder using a recursive chmod command. I need to run this after the line where I COPY the files to the container.
Note that the command is prefixed by RUN which allows us to run commands in the container. Those commands run when the container is built, not when it is executed as explained in a prior post. Use the link above for the COPY command to read more about the RUN command.
Next, I rebuild the docker image. So about this warning:
I added the above command to upgrade pip. We also might want to use a newer version of Python but for this test I’m going to use it.
I still get this error:
What’s happening is that we haven’t added a separate user to the docker container so it’s running all these commands as the root user. This post has some information about fixing that problem. I’ll address that more later.
https://stackoverflow.com/questions/68673221/warning-running-pip-as-the-root-user
Do not run the last line in that example at the linked page unless you actually need that functionality. Be careful with answers on stackoverflow and make sure you understand what each line does. Not all answers that “work” are the right answer.
Also note that you might have to run certain commands as root and then downgrade to the new user. More on all that later.
Redeploying a container used by a Lambda function
So where is the “state” of the function mentioned in the above documentation when you push a container to ECR? I looked all over the Lambda dashboard for my function and I can’t find it. I can see the state when I run this command:
aws lambda get-function --function-name dockertest
Let’s write a little loop to check the state as we push the new image to ECR and try to test the function again. I could have taken more time to check the states and exit appropriately but this is just a quick test:
So I am monitoring the state of the Lambda function like this:
I pushed the new version of the container to the ECR repository. The state never changed.
I retested the Lambda function and it failed. The container for the Lambda function was apparently not updated. Was it still in the process of updating? If so, the state never changed.
I just happened to notice this button on the Lambda dashboard:
I don’t recall reading about that, but I pushed it. The Lambda state still never changed.
But now the Lambda function works.
I’m not sure if the state change was so quick my script did not pick it up or the documentation is incorrect. In any case, you may want to be aware of this version and deployment issue if it matters to you. If you have multiple functions working together that could lead to some tricky errors at the moment of deployment. You may want to get to a point where you cease all new requests (using a queue or something) during the time your deployment for new containers starts and stops, if you have a potential issue.
Also, about that Deploy new image button. Where was that in the documentation?
I used this query on Google and got no relevant results (at the time of this writing).
"Deploy new image" site:aws.amazon.com
Perhaps the updated documentation is in the works.
Is there a related CLI command?
I don’t see one.
I think this functionality is a bit new so perhaps it is still in the works.
Re-testing the container Runtime Interface Emulator (RIE)
Now that I know the container works in Lambda, I am going to test what failed before when trying to test this image locally using the Lambda Runtime Interface Emulator (RIE). I’m using the command from the blog post referenced above. I put it in a file called localtest.sh.
Same errors as before:
Perhaps it has something to do with the fact that the RIE was compiled for x86 and I’m using arm. I wrote about those types of architecture issues in the last post.
Here’s the documentation for the Runtime Interface Emulator (RIE):
Ah, yes:
The runtime interface emulator is available on the AWS GitHub repository. There are separate packages for the x86–64 and arm64 architectures.
Once again, it seems like the error could be determined by the code potentially and a better error message might be possible.
I’ll retest in the next post with the proper emulator. :-)
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2023
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
