Enforcing MFA Each Time a Lambda Function Runs (First Attempt and Set Up)

ACM.329 MFA to clone a GitHub repo via a Lambda function — first attempt

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: Lambda | GitHub Security | Container Security.

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post I attempted to use a Service Control Policy (SCP) to enforce MFA to invoke a Lambda function, but it didn’t work in all scenarios.

There’s a problem with that approach anyway. Once you have an active session, you can continue to take actions for that session for as long as the session is valid. I want to require MFA every time the Lambda function gets executed, which is something you might want for a highly sensitive batch job or Lambda function.

What would you do to prevent a data breach?

Sure you might not want that during development where you are repeatedly running and tweaking code. But when you are running an action that could lead to a data breach that costs the company millions of dollars, the few extra seconds to input a code might be warranted.

As always, nothing is perfect but this approach requires entering an MFA code to run a Lambda function.

App Codes vs. Yubikey — consider the use case and the risk

I already explained some issues with a Yubikey code via a command line tool. Once an attacker gets onto your machine they may be able to generate codes or change the configuration of your Yubikey with a command line admin tool.

For this reason, I’m using a virtual authenticator — an application that generates a one time code that I enter to run the Lambda function. As mentioned in a prior post I have a separate device for generating codes than the one I use to surf the web and read social media posts or email attachments. I also have my iPhones in lockdown mode.

About the multiple factors in MFA

Using an application to generate and enter a code does have some risk. An attacker may be able to obtain a code via keylogger when I’m entering it.

But what if they don’t have the credentials with which to use that code?

MFA has multiple factors of which the code is one. The hard coded credentials are the other. If we can keep one or the other away from the attacker, MFA is doing its job.

Designing systems with segregation of duties

You can use segregation of duties to further protect the credentials. Perhaps you have one person on your team who is responsible for the credentials associated with your process and a separate person who is responsible for the MFA device.

This is on top of other controls like networking and the IAM permissions to invoke the Lambda function. Think about how you can protect your assets and segregate duties by allowing different people to access different parts of the system but no one person has access to everything.

Reviewing the mechanisms for using MFA with the AWS CLI

In my prior posts I showed you multiple ways to create a profile for the AWS CLI.

User with credentials

If you are running the AWS CLI as a user, you can add that user’s credentials using the aws configure command. No MFA will be required. If you try to require MFA for an action, any requests from that user will fail to pass the MFA requirement. (In theory — I may have found an issue below after I wrote this.)

~/.aws/config file

[profile USER]
region=us-east-1
output=json

~/.aws/credentials file

[USER]
aws_access_key_id = xxxxxxxx
aws_secret_access_key = xxxxxxxxx

Configure a role with an MFA device

You can configure a profile for a role that requires MFA to assume by adding a profile with a role name and the MFA device ARN to the profile in the config file.

You will need to either add credentials for the role profile in the credentials file as shown above, or reference the credentials of another profile as shown below.

~/.aws/config file

[profile ROLE_W_ARN]
role_arn = arn:aws:iam::xxxxxxxxxxxxx:role/DevOpsRole
source_profile = USER
mfa_serial = arn:aws:iam::xxxxxxxxxxxx:mfa/john_mfa
region = us-east-1
output = json

~/.aws/credentials file

[USER]
aws_access_key_id = xxxxxxxx
aws_secret_access_key = xxxxxxxxx

You can also add an external ID as explained in other posts to prevent a confused deputy attack for cross-account access.

Assume a role and dynamically configure a profile

The other approach was to use STS assume role to assume a role and dynamically configure a profile. In that case, you don’t have to specify the role or the MFA device in the configuration. The configuration file only needs the region and output. The credentials file will have short term credentials in it. You also need the user or role configuration that is allowed to assume the role.

~/.aws/config file

[profile DynamicProfile]
region = us-east-1
output = json

~/.aws/credentials file

[DynamicProfile]
aws_access_key_id = xxxxxxxxx
aws_secret_access_key = xxxxxxxxx
aws_session_token = xxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxx//// etc.

This configuration is nice because you don’t have long term credentials here for the assumed role, but the credentials can be leveraged as long as they are active, if stolen. This approach allows you to run CLI commands using a profile in the standard way with short term credentials. There are other ways to assume and use roles with AWS STS but this method works when you have a tool that expects an AWS CLI command with a profile.

All the above options are explained in my other IAM Posts.

Configuring roles profiles dynamically using the AWS CLI

We can configure roles dynamically with the AWS CLI in any of the above formats as explained here:

That means, if we have the appropriate values, we can dynamically configure and use a role in our Lambda function, unless there is some restriction in Lambda that prevents this.

New resources to support MFA via Lambda

For now I’m going to manually create the following to test this proof of concept approach to enforce MFA via a Lambda function

A new secret named GitSecrets.

A new user:
username: dockertest
console access: no
AWS developer key: yes - stored in secrets manager
MFA: yes - named dockertest
Permissions: Permission to assume a new role.

A new role the user can assume with the following permission:
AWS CodeCommit and SecretManager for a secret named GitSecrets

GitSecrets

I’m going to manually create a Secrets Manager secret called GitSecrets and move the git information from my Lambda secret to the GitSecrets secret.

I’m going to remove the git information from my Lambda secret and add the credentials and MFA device ARN for my new user.

dockertestAssumeRole

I’m going to call my new role and policy dockertestAssumeRole — when I automate it will still be possible to keep things generic as possible using a consistent naming convention.

I am going to have to complete the trust policy after I add the user:

I’m also going to have to create a policy to add to the role but for now I created it with no policy and named it dockertestAssumeRole.

I get a warning about an overly permissive trust policy, which is true. I haven’t specifically limited who can assume the role because I still need to create the user.

Next I create the policy. I grant access to all actions for AWS CodeCommit for a specific policy for now. Once I run the Lambda function, I can review the logs to see which actions are actually required as I showed you how to do in other posts.

I also add read only access for GitSecrets.

dockertest user

Next I create the user.

I’m going to opt to create a new policy.

Choose the STS service.

Grant access to the role we just created.

Name and save the policy.

Refresh the user policy list screen and select the policy we just created.

Now I can go back to dockertestAssumeRole and fix the trust policy.

Add a principal.

Add the dockertest user and your own account number if following along.

Don’t forget to add /user in a user ARN.

Also note that I should not have used ForAllValues above so I removed that.

Now the warnings are gone. Update the policy.

Create a developer access key id and secret key and add them to the dockertestSecret. The docker execution role will be able to retrieve those credentials.

The browser has a huge attack surface. AWS Identity Center has cross account role issues and now way to enforce MFA on role assumption for a single user as AWS IAM did in the past. I’ve gone over those things in great detail in prior IAM posts so I refer you to the link above.

This post will show you how to protect AWS automation credentials to some degree, though there is always more we can do. This is a pretty good starting point. Nothing is foolproof but this is better than credentials on a developer machine, in a browser. Yikes!

Think LastPass breach. I wrote about that here.

So yes, we are enabling these credentials for the AWS CLI.

Return to the same page and assign a virtual MFA device.

Name it dockertest.

Add the mfa_serial value to your secret with the ARN of the virtual MFA device. We need this here if we opt to allow the MFA device to exist in a separate account, in which case we may not be able to calculate the serial number. Alternatively you could require the MFA device to be in the same account, give it the same name as the Lambda function and calculate the ARN. I’m on the fence as to which way I’ll go long term but for now my secret looks like this.

Assuming a role that requires MFA with the AWS CLI

We will dynamically generate a CLI profile for the user credentials stored in Secrets Manager for the Lambda function. That is not an actual user but a principal created specifically for this purpose.

We’re going to use the AWS CLI assume-role command to assume the role that is allowed to access the AWS Lambda secret with the GitHub credentials and access AWS CodeCommit.

We’ll need to pass in a token associated with the appropriate MFA device ARN.

We’ll use the temporary credentials we get back to configure another CLI profile that can run commands with the privileges or the assumed role.

Here’s the assume-role command. I already had code from prior posts to do all of the above so copying that over was pretty simple. I also had to modify the secrets code a bit.

I’m going to cover some issues with AWS CodeCommit in the next post.

Here’s the assume-role command we’ll be using.

Altering the Lambda Function

Head back to the container code we’ve been working with that leverages the custom Bash runtime and alter the function in handler.sh. I started that container here and added improvements in subsequent posts with improved error handling and other functionality.

I can obtain the secrets in the Lambda secret now instead of the git information now in GitSecrets as shown below.

I temporarily echo out the values but immediately remove that once I confirm they are working.

Now I can try to assume the role. I’m going to use the AWS CLI assume-role command above. But how can I get the token? We’ll pass it in as a parameter. That means the token may be accessible in certain places, such as a one time nonce used in a web request. It can only be used once, and the person using it also requires the credentials.

I’ll add a function to get the token from the request:

Note that I need a few new validation functions for numeric values and a length of 6. I wrote about those here:

I updated my test file to pass in the code:

Now I was going to calculate the assumed role but at this point, I decided there’s no easy way to get the account number so I’m going to get it from Secrets Manager as well.

I can add a lot more validation to all of this like checking to see if the role is the correct name at the end and in a proper ARN format, but for now I want to see if this even works.

I retrieve those values and configure an AWS CLI profile for the user credentials:

I used my existing code to assume the role:

Obtain the temp credentials:

And configure the role profile via the CLI:

I ran an AWS sts get-caller-identity with the role name profile to confirm I had assumed the role but as I showed in a prior post that doesn’t work in AWS Lambda so I removed it after the test.

aws sts get-caller-identity --profile $rolename

I clear the variables I don’t need.

I was cruising along and started getting this error which took a fair amount of time to resolve. I added the code to get the git credentials from the new secret.

I got this error.

An error occurred on line 55. Exit code 254. 28cbf148–2094–4492–8be5-bfd3dcc05f29 An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::464339214996:assumed-role/dockertestAssumeRole/dockertest is not authorized to perform: secretsmanager:GetSecretValue on resource: GitSecrets because no identity-based policy allows the secretsmanager:GetSecretValue action

So I’m going over and over my IAM Policy. I did this quickly and forgot about the KMS key. The role needs permission to use KMS and the KMS key policy needs to add permission for the role so I added that.

But no that wasn’t it.

Some problem with the format of the ARN for GitSecrets. I created this all in UI so it seems like a UI bug that generated or allowed an incorrect secret ARN in the policy. I’m not sure but I removed that for now as this is mainly a proof of concept of the MFA functionality.

Once I had the secrets my code worked as it did in the last post to access GitHub without putting credentials in the URL.

With the exception of the speed bump with the secret ARN, my existing code worked like a charm until I got to CodeCommit, which requires some additional configuration. I’ll cover that in the next post.

Next… the moment of truth. Does this work in Lambda? I pushed and deployed the container to the Lambda function we’ve been working with in this series.

I have to edit my test event to pass in a code:

Read only file system…I was wondering if that would work. We can’t write to the credentials and config file because they are read-only in Lambda.

Well, we can change where the files exist for credentials using two environment variables.

I also delete those files at the end of the function.

The problem is the caching of the temporary credentials for the assumed role. In fact, I wish those were not written to disk at all. I would like to capture them in memory and write them somewhere myself but I don’t see a way to control that here. #awswishlist

Or here:

So I build and push and deploy my new container with those environment variables and I pass in a bad code, thinking that the role won’t be assumed due to a bad token (that error works as expected locally). Then the file won’t be written. But I can test the rest.

Once again, no. We cannot reach STS due to a timeout.

I presume AWS is preventing role assumption as a security mechanism but as you can see here, this would actually improve security not harm security.

Update: I went out to have a cup of coffee and started thinking…did I really check the networking in that last post where AWS STS got blocked? I think I did but I can check again just to make sure. As it turns out there was an issue in that AWS let me select subnets for my VPC endpoint that were not actually accessible. This caused an incredibly long timeout. But even after fixing that I still hit another error and was not able to get this work. I wrote aobut that in the next post.

Even if the network issues get fixed, we still have an issue if the assume role function writes the credentials to a temp file in a location we can’t change. I thought about trying to change the home directory for the user running the lambda function in the container but I am not sure that will work either — or what all the implications would be. Other things might break. Anywho.

More later.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab