Adding Error Handling to Bash Custom Lambda Runtime

ACM.306 Using Bash trap to capture and handle errors in bash scripts

Part of my series on Automating Cybersecurity Metrics. Container Security. Lambda. Deploying a Static Website. The Code.

Free Content on Jobs in Cybersecurity | Sign up for the Email List

In the last post I created a Lambda custom bash runtime that runs in a container, based on an AWS tutorial, but it’s missing a few things.

In this post I want to add error handling to the custom runtime. If you recall from my post on custom runtimes, there are two APIs we can call to send errors to Lambda.

Read this post first to understand the APIs you need to work with but note that I fixed some issues in a separate post. There’s a link at the bottom.

Initialization error — Log errors that occur during the initialization process.

Invocation error — Log errors that occur during the processing of the Lambda event.

I started looking into the trap command as a way to trap errors. This led me down a rabbit hole. I have some references at the bottom. Essentially you can trap exit codes in bash and run a command or call a function. So I looked at a sample and wrote a little script to do this using the ERR exit code. I figured, divide by zero is an error so I’ll use that to produce an error and that should work, right?

Wrong. For whatever reason that doesn’t return the “ERR” exit code.

As you can see from the code above, if there is an error that produces the ERR signal code in bash, it will be trapped and run the send_err function. That function will exit. The line after the error should not run, but when the signal code for ERR exists the function send_error should be printing out some messages and it’s not. I want to call the Lambda API to log an error when that happens and this isn’t working.

Now if I simply put some gibberish on that line instead of divide by 0 it works:

You can see that my send_error function executed:

So apparently exit code 127 is “ERR” but divide by zero produces some other exit code. So I thought, well I’ll just add all the bash exit codes. What are they? Finding a definitive list was not simple. There’s a link below. Which of those codes do I really need? That’s a long list.

What if I could figure out what the exit code was? I tried to produce a bunch of lines with all the numbers that could be an exit code to check for those numbers using a for loop to spit out lines of code for all possible exit codes. But this resulted in an error because trap won’t run with an invalid exit code. Apparently trap knows what all the valid exit codes are. That was a hacky approach that was going to take too long.

Then it became obvious. I should just check for exit and a non-zero code. First let’s run the function for any EXIT signal. I left some other codes in there just because.

Notice now when I run the code it calls my function on exit and the exit code is 1. According to the lists referenced at the bottom of this post, SIGHUP is exit code 1 and I have SIGHUP in my list above, but it’s not working. So what is the proper variable for 1 on Amazon Linux? No idea but we can just solve this problem another way.

First let’s test what happens when we comment out the error like this:

With no error or abnormal exit, we get exit code 0 (zero).

So let’s just capture every exit and only send the error if the exit code is anything other than 0.

Run the code with no error. The call to the Lambda API would not be executed in this case.

Now add back in the divide by 0 error.

Replace the divide by 0 error with a gibberish value (command does not exist error):

That’s pretty cool. We’ll have to see how it works in practice.

Now to handle which API gets called I need to know whether we are initiating the function or processing. We can add a state variable to our code like this:

Now insert our gibberish command in the initialization phase.

The Lambda Initialization Error API gets called in that case.

Insert the gibberish command in the processing phase portion of the code.

The Lambda Process Error API gets called in that case:

That seems to work but we’ll obviously have to do more testing to make sure nothing falls through the cracks. Let’s move this over to our custom runtime client code.

I add in the Lambda error API calls and set the error messages. It’s at this point that I realize the main difference between an initialization error and a processing or invocation error is that the latter passes in the Request ID. For that reason I need to move the processing state after the point where we have a Request ID.

I could actually just check if the REQUEST_ID exists but that leads to various bi-products and code paths I don’t want to deal with so I’ll leave it like this.

As in prior posts I build the container using my build script:

./build.sh

I push the container to ECR using my push script:

./push.sh

I push the deploy button in the Lambda console. There must be a way to automate this but I’ll figure that out later.

I test my container again:

It still works. I haven’t triggered an error yet so that makes sense. Let’s insert an error and see what happens.

I insert the line gibberish and repeat all the above to build, push, deploy, and test my new container.

Yay. I see the error message from my code above and it correctly sent an initialization error. Notice there is no Request ID in the log output.

Let’s move the error to the processing section of the code.

Yay. Process error.

We also get the Request ID:

Now I can remove the gibberish test line to break the function and move on. You could add other things here as well as noted in the documentation such as the X-Ray tracing ID. I might get to that later…so much ground to cover!

Update: after testing this further it had some issues which I adddressed in this post. I also moved the error handling to a separate file which can be included in the base image of any Bash Lambda container.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

The best way to support this blog is to sign up for the email list and clap for stories you like. If you are interested in IANS Decision Support services so you can schedule security consulting calls with myself and other IANS faculty, please reach out on LinkedIn via the link below. Thank you!

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
Author: Cybersecurity for Executives in the Age of Cloud
Presentations: Presentations by Teri Radichel
Recognition: SANS Difference Makers Award, AWS Security Hero, IANS Faculty
Certifications: SANS
Education: BA Business, Master of Software Engineering, Master of Infosec
Company: Cloud Penetration Tests, Assessments, Training ~ 2nd Sight Lab

Like this story? Use the options below to help me write more!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Clap
❤️ Referrals
❤️ Medium: Teri Radichel
❤️ Email List: Teri Radichel
❤️ Twitter: @teriradichel
❤️ Mastodon: @[email protected]
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
❤️ Buy a Book: Teri Radichel on Amazon
❤️ Request a penetration test, assessment, or training
 via LinkedIn: Teri Radichel 
❤️ Schedule a consulting call with me through IANS Research

My Cybersecurity Book: Cybersecurity for Executives in the Age of Cloud

References: