User-Specific Secrets on AWS: KMS and MFA with Developer Credentials

ACM.84 Granting an IAM Group permission to use a KMS key in a Key Policy

Part of my series on Automating Cybersecurity Metrics. Metrics, MFA, Passwords and Encryption. The Code.

Free Content on Jobs in Cybersecurity | Sign up for the Email List

We’ve been working on adding a user-specific secret in Secrets Manager in the past few posts and considered how to deploy secrets in a manner that supports non-repudiation.

The problem we have from the first post is that we changed our KMS policy to allow the Developers Group Role to use the KMS key. Although our user (Developer) can describe the key, the user cannot get the key because that IAM user does not have permission in the KMS Key policy to use the key.

Modifying our KMS Key Policy to support multiple users

Here’s our next challenge. Our KMS key script that we’ve been using up till now expects one input for a decrypt user. We need to allow multiple users to use the key to decrypt their own secrets based on the design in the last two posts. I explained that we are not using a per-user key due to the cost in a prior post, but you could use that option in a high-security environment.

We can fix this issue by altering our key policy to take in a list instead of a single ARN. In addition we can loop through our users in our Group and add them to the policy, similar to how we added users in a Group to a trust policy as described in this post:

Essentially we want to allow a Group to use a KMS key but AWS policy documents don’t support groups as principals so we have to loop through the users of the group and add each one to the key policy.

For starters, I want to try something I didn’t try for the role policy in the above post. I want to use the existing KMS template but pass in a list. I can simply change the type for my encrypt and decrypt ARN to a CommaSeparatedList

There’s another thing I need to change. Since we are now passing in a list instead of a single value, I need to remove the dash below.

Modifying policy statements to support a comma separated list

We need to remove the encrypt and decrypt ARNs from the describe key statement since we were individually listing and referencing three separate roles. I presume we can’t pass in two lists, though we could try. It is probably just easier to add DescribeKey permission to our encrypt and decrypt statements and remove those two single values from the describe key statement.

When I tried this out, no luck. It seemed as if KMS key policies do not work the same way that a trust policy does. Upon further testing, it turned out to be some other as of yet still unknown issue I didn’t care to pin down. It would also be nice if this error message pinpointed the line in error in the policy document.

To verify this problem was not just a typo (which I guess it was) I restored the key policy to what I thought it was in the first place from a local backup but I still got the above error. Hmm. So I pulled my code back out of source control that I know works. It updated successfully.

One more try to modify the KMS policy to make sure it wasn’t just a typo.

To be extra careful, I updated and tested the policy changes one at a time. I removed the encrypt ARN from the describe key policy and removed the dash in front of the encrypt ARN in the related policy since it is now a list. That worked!

OK now do the same for the decrypt ARN.

That also worked. Next copy and paste the DescribeKey permission to the statements for encrypt and decrypt.

That works also. So I must have had some extra space somewhere because that’s exactly what I thought I just did. AWS Policy Documents and lack of helpful error messages are one of the trickiest things to get right on AWS.

So here’s what we have so far. Encrypt and Decrypt ARN parameters are now a comma delimited list:

The describe key policy statement only has the root user:

The encrypt statement references the encrypt ARN parameter with no dash and the statement has the DescribeKey permission now:

Same for the Decrypt statement:

Now, so far I was just testing my existing deploy script as is. I haven’t gotten the list of users from the group to pass in. If we want to allow the KMS user to do that we’ll need to add the GetGroup permission to the KMS group role. Add this permission and deploy the role.

Next we need away to get the list of users to pass in as the encrypt and decrypt ARN.

We can reuse the code we used before to get a comma separated list of IAM users. Now I notice that I forgot to add a profile so I will add that now. In addition we are now using this code in two places so I will move it to the shared functions file.

Next we need to get the list of users via our new function and pass it in to create the key policy using that list.

At this point I get the following error. This error is misleading and is not exactly the problem, though the error is related to parameters:

An error occurred (ValidationError) when calling the CreateChangeSet operation: Parameter 'ServiceParam' must be one of AllowedValues

Taking a look at the parameters. They do not look correct at all so I have a typo somewhere.

I had set the service to “secretsmanager”.

…but that’s not what is listed below. It’s using the first word of the description as the service parameter and the second word as the description parameter. So obviously I didn’t set one of the parameters correctly if at all. It’s missing so single words from the description are getting pulled up into other values (all part of the issues with spacing an bash function parameters when using the AWS CLI I’ve written about before…)

What are the CloudFormation template parameters?

DecryptArn in my above parameters is not an ARN or a list of ARNs. So first of all, this is not working:

To test this function independently I created a test.sh file and once I started moving the code over to test it I realized that I never set the group variable. Oops.

After setting that variable so the group gets passed into the CloudFormation template, now my parameters look correct.

And, that worked.

Be nice to people who use your code — and yourself!

But what was the real problem with my code? I forgot to add the check for missing parameters in my get_users function to give me a nice error message that clearly indicates the problem. Head on over to our function and add the missing checks.

Next I re-tested the key deploy.sh script to make sure I did not introduce an error.

Now we head over to the key alias deploy script and add code to deploy an alias and test it:

That works:

OK now we can test if you our user can get the secret from secrets manager.

Recall that we created a profile for our developer in a prior post.

aws secretsmanager get-secret-value --secret-id Developer --profile developeruser

Access to KMS is not allowed.

An error occurred (AccessDeniedException) when calling the GetSecretValue operation: Access to KMS is not allowed

Problems with our key policy conditions

What are we missing? This is a frustrating error message because it doesn’t tell us if the problem is in the IAM or the KMS policy. When I review them they both appear to have permissions for KMS and our key. Let’s head over to CloudTrail. Ah, but once again the conditions policy does not seem to be correct. When I created the key I passed in secrets manager as the service, but it does not have secrets manager in the condition. It would be nice if the error message from AWS mentioned the condition specifically and provided a more accurate error message (#awswishlist).

To be perfectly honest, I fixed this code once before but when I went back to look at the code, my changes were missing. So I get to “practice” fixing this error again.

Back to CloudFormation to validate that secretsmanager was passed in for the service. My parameter is correct but this logic is incorrect. Do you see the problem?

Should say ServiceIsSecretsManager. I based my subsequent logic on the name so need to fix that as well by swapping the order in my Fn::If statements. And of course after due that I get an error about a typo.

Now I have to redeploy all my keys that were using that incorrect template. This is why having a test environment and test scripts is so important before pushing changes to production.

And now…my developer user can retrieve their own secret from secrets manager.

MFA revisited

Recall that two posts ago we couldn’t add MFA to our user-specific secrets policy. Let’s revisit all the actions our user just took and see if MFA exists for any of them. I had to wait a bit for the information to show up in CloudTrail but when it did I get the following list of actions for the above command. Also interesting that KMS does not show what is being decrypted — in this case a Secrets Manager Secret. That would be nice.

Let’s see if MFA is found in either of the above. Here’s where things get interesting.

GetSecretValue has no MFA flag.

The KMS request does, but it’s set to false.

The odd thing is that I am still enforcing MFA with a boolean in my AWS IAM User secret policy:

So apparently:

• Adding MFA for a user in the CLI configuration does nothing (which I kind of already covered before).
• Adding a condition to enforce MFA for kms:Encrypt and kms:Decrypt in an IAM policy does nothing if a user for IAM user programmatic access? Is that a bug? Or am I missing something…

There are two potential ways we could resolve this problem to enforce MFA — and I say potential because one of them relies on the above condition that doesn’t seem to be working. You’d need to test this out to know for sure:

• Allow the user to assume a user-specific role which requires MFA for role assumption. Change all of our code to allow each user-specific role to access it’s own secret.
• Only allow the users to obtain their secrets from the AWS Console, in which case MFA can be enforced to log into the console.

Neither one sounds extremely appealing to me at the moment, but let’s say we did want to allow console-only access to obtain the secret. We would need to give the user console access when created (update the user template and deploy script to include those options). We’d need a way to enforce access to the secret only via the AWS Console.

The only possible way I see to do that is via the user agent in the request and anyone who’s ever done penetration testing 101 knows that is super easy to bypass. I could submit a programmatic request and change the user-agent in flight on it’s way to the service (after it leaves the CLI or any other tool I’m using).

Alright. So we’re back to creating a user-specific role. And now this all seems like a bit much as a hacky workaround. Well, we should have a user-specific secret. We just cannot enforce MFA easily. Maybe I’ll revisit this later.

NOTE: Later when I tested this same thing out, it appeared that the Developer CLI credentials were disallowed when MFA is enforced and only allowed console access. If you are counting on any of this MFA enforcement, test it thoroughly!

IP Restrictions

There is one other condition we could add. We could restrict the policy to a specific IP address or IP range. If you use the Developer Bastion Host idea in a prior post, you could lock down the IP to the user’s VM IP address within your VPC. The problem with that is that IP addresses are ephemeral and change. If we allow access to a single IP address and the developer restarts their VM (EC2 instance) then the IP address will change and the developer would not be able to access their secret.

Additionally, the VM I’m using is currently reporting the public IP in the request. If the IP changes it may grant access to an IP completely unrelated to our account, though the person wouldn’t have access due to IAM permissions. There are restrictions on elastic IP addresses — you can only get so many, so it would be difficult to lock down a specific secret to a specific IP address.

However, if we start using a VPC endpoint, we may see that our request comes through at a private IP address belonging to our developer VPC. In that case, we can restrict access to our Developer VPC and the Developer VMs deployed it it. This would not prevent developers within your organization from accessing secrets based on IPs but it would prevent stolen developer credentials or sessions from being used to access developer secrets outside your developer VPC.

Let’s say someone accidentally sent logs to AWS support as explained in this post:

There’s a rogue insider at AWS who tries to use that session to get access to developer secrets. We lock down deployment of EC2 instances and other compute resources so that is only possible with MFA. In that case, the session credentials could not create a resource in your account to run the code from the VPC with an allowed IP address. An external request should have a public IP. So even with a valid session, the credentials could not be used to access the secret.

I’m speculating here. We don’t know for sure until we test it out. This is an example of how weaknesses in one aspect of your security controls can be offset by other security controls. Designing security controls can be tricky, depending on how secure you want to be!

I always try to keep it as simple as possible, and in my mind the simplest thing for me would be if AWS would fix the aforementioned issues that got us to this point. For now, I’m going to leave what we have in place and possibly revisit it later. Maybe I’ll come up with a better idea. :-)

Testing other profiles CANNOT access the secret

Whenever you test code and especially code related to IAM policies, don’t just test that it works. Phew. Done.

Test that the things that shouldn’t work don’t.

What happens if we switch to our IAM Profile:

How about our KMS profile? Same thing.

So presuming the KMS user and IAM user do not have permission to alter the resource policy (which is not true in our current implementation for the IAM user but I explained how to fix it in the last post), then only the Developer that owns the key should be able to retrieve it and use it to login to EC2 instances with that SSH key assigned.

If we create a per-user EC2 instance, we can lock down the instance and key to a specific user.

What if the security team needs to access that instance? There is a way to change the SSH key for an EC2 instance in case of a security emergency. You will also want to understand who can do this and ensure that people who shouldn’t be doing this are not:

We still have work to do:

• We don’t really have non-repudiation. We’ll need to think through the architecture of our IAM policies a bit more.
• I eventually opted to give Developers console access due to limitations with roles and developer keys related to MFA and user-specific permissions. For that we had to update the developer and think through how we are handing out credentials. More issues with non-repudiation.
• I found some issues with this implementation when I finally got through all that so I could test user access in the console.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2022

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab