Mirroring a GitHub Repository to GitHub with AWS Secrets Manager

ACM.324 Creating a common function to retrieve a secret value used by all Lambda functions to retrieve their own secrets

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: GitHub Security | AWS Security | Application Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post I had to pause to consider the exponential risk that is possible if you share credentials and access between cloud providers. It all depends on the threat model.

In this post, let’s remove those ugly hard coded GitHub credentials from the post prior and get them from Secrets Manager. I’m also going to create a reusable function that any Lambda can call to get the value of one of its own secrets. Recall that we have a standard naming convention from a prior post that makes this possible, without writing custom code for each individual Lambda function. This is another example of the power of abstraction in cybersecurity. We can make things easier and prevent misconfigurations with this approach.

Recall that my explanation and definition of abstraction for cybersecurity differs from some others you might have heard. I’m not sure who started talking about it first, but I’ve been explaining this concept in my blog posts long before I read the other use of the term. That use of abstraction is interesting, but to me, not as powerful as what I’m talking about here in this blog series.

Let’s see how we can make this work in some common Lambda code that uses AWS Secrets Manager.

Add the Secrets to Secrets Manager

The first step is to add the secrets to Secrets Manager. I’m all for automation but manually copying and pasting those secrets into Secrets Manager limits the chance they will end up in the wrong log somewhere. I may eventually automate the secret creation but set the values manually. The biggest risk with the copy and paste approach is the chance that an attacker has access to the memory of the system where you are copying and pasting.

Recall that my secret name is the same name as my Lambda function with “Secret” at the end. By using a common naming convention we can create a single block of code that can be used by every Lambda function that uses this pattern.

I create two key-value pairs for this particular Lambda function: github_pat and github_ower. I explained what those values are and used them in the last post where we cloned the repo with the values hardcoded into the code (bad, bad, bad.)

Once the values are set we can retrieve the secret in our function and use those secrets instead of hardcoding them.

Getting the name of our function

I want to dynamically retrieve the name of the function. This is a bit tricky because there’s no standard way to get the name across the test environment and in AWS. The problem is that the Lambda Runtime Interface Emulator seems to be returning a generic function name instead of a specific one as far as I can tell. We’ll have to work around that.

Here’s what I did.

First, I check if the environment variable exists and use it, if it does.

If not, I check the function ARN, which is ideally what I’d use if the Lambda RIE returned the correct name. I use the function headers for that.

Note that there’s some kind of special character returned in the ARN used in the RIE. I don’t know what it is, I just removed it with this line:

Note that the same thing happened when running this in Lambda, however it was not only the function name. We have to be very careful with quotes in any values in this code.

I also learned something interesting about dashes in sed commands:

I have some information about sed here:

If the name equals the generic “test_function” name returned by the RIE, then we’re using local credentials. I’ll get the function using the STS get-caller-identity command that doesn’t work when the function is deployed to Lambda.

Since we’re using credentials above, we have to call get_creds before using this function to make sure the credentials are set.

I put this into a common, reusable function: get_function_name in a file called function_name.sh.

I added that file to the Dockerfile as shown in prior posts.

A reusable init function

Now I don’t want to have to source multiple files in every function I create. I want to limit that as much as possible. But I want to keep my file names related to their purpose. If someone needs to edit the code to change how getting a function name works, they should not need to edit the file related to credentials. This makes code reviews and tracking changes easier.

So I’m going to create a new init.sh file and call my get_creds and get_function_name functions from that.

Then I can call init() from my Lambda function code (which is a bash function). And if I need to handle any other initialization tasks later, it’s obvious where to put those now.

Notice that I echo the function name from within the init function and also inside my function handler to make sure the environment variable is set correctly.

I execute the RIE in one window and test my function in another as shown in prior posts and it works:

Just FYI, I also stepped through and tested that the correct function name was set at each point along the way as well.

Perhaps I could alter the RIE somehow to pass in the correct function name locally, but I don’t want to simply trust a value passed in by the client. I also want to be reasonably sure I’m using the correct credentials.

I want this all to be derived dynamically in a way that cannot be altered, though if our permissions are set up correctly, a function can only ever access its own secret. 😊

Getting a secret value

Now that we have our function name, we can use that to retrieve our secret, which has a name of the function plus Secret at the end.

I create a secrets.sh file.

I add a function get_secret_value with the secret key (name) to retrieve as a parameter.

For my initial test I simply echo back the secret, using the FUNCTION_NAME environment variable.

Now hopefully you see how our naming convention helps us quickly deploy Lambda functions with proper secrets management without recoding all the secrets access code over and over again.

I include the secrets.sh file in my Dockerfile:

I have to source the code in my function and call the get_secret_value function.

I execute this bit of code to make sure it’s working so far and it is.

I get back the full secret value with two key-value pairs.

“{\”github_pat\”:\”github_pat_xxxxxxx\”,\”github_owner\”:\”xxxxxxxxx\”}”

Now I want to retrieve the value of the specific key I’m passing in. I’m going to install jq in my Dockerfile.

As explained in a prior post I noticed something interesting about yum repositories on AWS at this point.

Then I can use jq to parse the secret value for the specified key:

I test that and it works. Now I can add the code for the owner and test that as well.

That works. Now I can move on to altering the rest of my function to use those values.

Now you might be thinking, as I am, yes but you have to call the get secret function every time you want to get a value. That’s extra overhead. Yes, it is. I could store the secret in an environment variable, but then the secret is exposed in various ways and hangs around longer than I need it.

I could encrypt the environment variables but that’s more work and overhead for my function. Also, the encryption is useless if someone has access to the key that can decrypt the environment variables.

That’s one of the things I look for on penetration tests — seeing if I have read only access in an account what types of secrets I can uncover, sometimes via Lambda environment variables.

I’d like to keep the secrets with the function itself and clear them when I’m done using them.

A more viable approach might be to allow passing in multiple values to retrieve at once. Perhaps passing in an array of keys and getting back the array of values. That could lead to various parsing errors and injection attacks as a result. I am already thinking about the injection attacks that are possible in the above code as it is. I’m going to keep it simple unless I find that I need the performance gains of an alternate approach.

And on that subject of injection attacks, maybe I’ll revisit that later but for now I’m going to move on. In this case, I’m trusting myself not to inject bad values into Secrets Manager. Your scenario may be different. I may test some things related to this down the line.

Using the Secrets Manager Values

Now that we have our values we can get rid of the hardcoded github_pat and github_owner in our code and use the values from Secrets Manager instead.

Delete those awful hardcoded secrets! Don’t forget to delete any code that prints those secrets to logs as well, like any echo commands in our case, or print statements in other languages.

Instead we can use a common function to get a secret value for any Lambda secret.

Once we are done, we can clear out the values.

Do you see what I’ve done now? Any time we deploy a function we can use the same code that does the following:

• Deploys the Lambda function with our common naming convention.
• Deploys a secret with the common naming convention.
• Deploys a role and policy that allows a Lambda function to retrieve its own secret.
• Use a common base container to call a common function to get any secret value added to AWS Secrets Manager for that function.

You don’t need to repeat, recreate or review the above code over and over again if it has not changed.

The only code that will change is the code in the red box above where the function retrieves its own secrets and does something with them — and someone will have to add the secret values to AWS Secrets Manager.

All the time I just spent creating this reusable container for any Lambda function will save me countless hours later. In a large organization, it could prevent numerous issues with secrets in code because new developers don’t know how to use Secrets Manager or that they even should. You’ve just made everyone’s lives significantly easier.

You can focus all your time on testing this one function for potential injection attacks and think through the process of handling these secrets appropriately instead of people doing things an exponentially different number of ways.

That’s the power of abstraction.

With the power of abstraction comes one downside. You have to be incredibly careful about updating this code if a lot of things depend on it. You also need to test it thoroughly for any security flaws.

I’m going to continue testing this code as I go. Feel free to send me a note on LinkedIn or Twitter (ok X) if you find any issues, or submit the issue on GitHub once I check this in.

Of course, I wrote this in Bash as a proof of concept but you could do this in any language, and on any cloud provider that has a secrets manager and a way to include reusable code in a cloud service.

One other note — putting credentials in logs is not ideal. I’ll probably address that in a separate post.

Fixing AWS CodeCommit Access

You may have noticed I commented out the line that pushes the code to AWS CodeCommit and X’d out the credentials. I mentioned on Twitter that I already deleted those credentials because I don’t like that approach.

We could fix the AWS CodeCommit credentials the same way by storing them in AWS Secrets Manager, but as I mentioned in a prior post, standard git credentials do not provide a way to leverage MFA. I’m going to attempt to show you how I’ve used MFA in automated processes in an upcoming post, but I have to see if it works in Lambda as expected. Stay tuned.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab