Create a KMS Key for the Sandbox Account with Organization Role

ACM.209 Spoiler Alert: It doesn’t work. Avoid this.

Part of my series on Automating Cybersecurity Metrics. The Code.

Free Content on Jobs in Cybersecurity | Sign up for the Email List

In the last post I created some functions for assuming AWS IAM roles via the command line.

Now one of the things I tried to do initially after creating a Sandbox account was to create a KMS key in that account. There were a couple of ways I could go about doing this.

First, I could have the Sandbox user create the key, but the Sandbox user would need to use CloudShell, run the commands locally, or manually spin up an EC2 instance and configure it to create the key.

There are so many catch-22’s in Cloud Security Architecture. I want to create this secure thing, but first I have to create that secure thing. Which one should I do first?

I didn’t want to overthink this and I’ve got my OrgRoot user set up. The fastest way for me to get this done is to create the key with the OrgRoot user. Or so I thought. This post too way longer than it should have due to what appears a KMS policy check that caused me grief. I ended up abandoning this approach but I hope this complication will reach someone on the KMS team who will make this easier or improve the error message if I am doing something wrong.

TLDR; It didn’t work. You have to deploy a key with credentials used to administer the key. Which I don’t agree with. So if you have a deployment system deploying KMS keys — you have to give that deployment system credentials access to administer all your keys? Perhaps I did something wrong, but I ended up using the role for key administration to deploy the key.

Here’s what happened and the error message I think CloudFormation and/or the KMS team should address. And until they do, you should probably have a separate team in a large organization or at least a separate role in a smaller organization for administering and deploying KMS keys. Think through how your deployment system might be used to access all your encrypted data. I have to think about this a bit more and the possible attack paths as a result of this restriction before I can definitively say it is a problem, but it feels like a problem at the moment.

We can use our existing KMS key template to create a KMS key. That is not the problem. The problem is ~ who can create the key? What if I want one role to create the key and another role to administer the key after that point?

We are going to use this key for EC2 instances so we’ll follow the example in this post:

We can use our new assume role functionality in the last post to deploy the key.

Recall that we deployed a developer key for EC2 instances like this:

For our purposes we can use the same approach, but we will use the Sandbox admin user for the encrypt and decrypt role.

Modify the Sandbox deploy script to add that code:

Run the deploy script. It is at this point that I realize that I need to add a profile to my command to get a stack export in the shared function. Otherwise the default profile will be used to run the script and only looks in the current account for the stack.

This is one of the downsides of using CloudShell. I was using the default credentials for the shell and it wasn’t actually using a CLI profile, jsut a variable named profile. Now if I try to use this code in CloudShell it’s not going to work without adding credentials — which I don’t want to do because of all the issues I mentioned already when using AWS CloudShell.

What I would do instead going forward is to immediately create a Sandbox account and work from there probably but I have to think about it a bit more. At some point we need to deploy the first OU, account, and role. So maybe that initial deployment is created via CloudShell and then from there on out an EC2 in a sandbox account in a restricted network. More on that in later posts hopefully as I flesh this out further.

Back to the problem at hand.

Here’s where I get an error as I move into a multi-account environment:

Let the fun begin.

The key template is expecting a KMS admins ARN to exist in the same account where the key exists. Now my long term plan is to deploy KMS keys from a single account managed by KMS admins. But for the purposes of resolving this issue in the Sandbox account, I’ll “just” deploy the KMS admins role to this account. Then it will work right?

I wrote about the KMS admins role I created here:

This is the code we want to use in the Sandbox account to deploy the role.

However, if you followed along I later simplified that for multiple roles:

We should be able to source the role_functions.sh file and call the deloy_group_role command above to create the role.

That code depends a KMS admins group.

To administer the key we’ll need to add the Sandbox admin user to the KMS admins group.

No problem, we have code already written for that!

Then we can add the user to the group as in the existing group code, but instead of the KMSAdmin we’ll add the SandboxAdmin.

Here’s where I find a bug in my prior code. I wrote about the premonition I had about this issue when I wrote the code and said I’d fix it later. Later is now.

Take a look at the key template and the Cloudtrail condition. We need to adjust that to get around this problem.

Hokey? I don’t know but it works. I added a condition to check if ServiceParam equals CloudTrail. Then if the service is not CloudTrail I just change StringLike to StringNotLike for the first part of the condition.

Next error — “You will not be able to update the key policy in the future”

Are we having fun yet? This was the biggest waste of my time. I hopefully am saving anyone reading this from a similar fate.

This error is frustrating because it is inaccurate and it also restricts the ability to segregate key creation from policy management. I wrote about that here.

Obviously our KMS administrator role has access to modify the key. We’ve been using that same role throughout this blog series and I haven’t modified the permissions in the role policy.

This was going to be a simple post. But no.

Long story short, I tried to assume the KMSAdmins role using the Sandbox organization admin role which allows cross account access because I was mistakenly thinking that was the profile I created for the Sandbox Admin user. The org role can’t assume the KMSAdmins role.

Oops.

So I fixed that and tried the deployment using the Sandbox Admin user credentials and that use is allowed to assume the KMS Admin role.

Anyway it’s a whole long painful story but here’s the result.

I made the function to create a cross account role generic to set up a role profile for any cross account role. I’m sure that will be helpful in the future but I didn’t end up needing it. That was in my last post as well.

I created a separate function to get the account number for an account name since I needed that code in multiple places.

I created a separate function to get the appropriate values to formulate the organizational role name to work with my new generic function to create a cross-account role.

Then I modified the deployment script to use all of that and tried to assume the KMSAdmins role to deploy the Key, but like I said, it didn’t work.

The error:

An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/xxxx-Sandbox/OrgRootSession is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::xxxxxxxxxxxx:role/KMSAdminsGroup

So I changed the deploy script to only deploy the IAM resources:

Deploy the key with the Sandbox Admin user

Back to what I should have probably done in the first place.

Login as the Sandbox Admin.

Open CloudShell.

Try to run my KMS deployment script.

And….now CloudShell it doesn’t work.

And…I’m done with CloudShell. I am using a user that is an administrator in the account. I don’t have any SCPs limiting CloudShell. I don’t know what the problem is and I’m tired of weird CloudShell issues.

At this point I decided to just manually deploy an EC2 instance to get things done and complete my automation as I go. AWS has been touting CloudShell as a more secure option than long term credentials but I disagree. I also do not like this particular CloudFormation restriction in AWS. It is not accurate because the role I assigned as an administrator has full KMS permissions for all KMS keys in the account. I should be able to deploy that with an alternate set of credentials.

Just a security note while I’m here. If you are going to use long term credentials enforce the use of private networks and MFA. Limit permissions in policies. Currently I have a couple of sets of credentials in my account with too much access, but I’m going to fix that. Stay tuned.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

The best way to support this blog is to sign up for the email list and clap for stories you like. That also helps me determine what stories people like and what to write about more often. Other ways to follow and support are listed below. Thank you!

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
Author: Cybersecurity for Executives in the Age of Cloud
Presentations: Presentations by Teri Radichel
Recognition: SANS Difference Makers Award, AWS Security Hero, IANS Faculty
Certifications: SANS
Education: BA Business, Master of Software Engineering, Master of Infosec
Company: Cloud Penetration Tests, Assessments, Training ~ 2nd Sight Lab

Like this story? Use the options below to help me write more!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Clap
❤️ Referrals
❤️ Medium: Teri Radichel
❤️ Email List: Teri Radichel
❤️ Twitter: @teriradichel
❤️ Mastodon: @[email protected]
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
❤️ Buy a Book: Teri Radichel on Amazon
❤️ Request a penetration test, assessment, or training
 via LinkedIn: Teri Radichel 
❤️ Schedule a consulting call with me through IANS Research

My Cybersecurity Book: Cybersecurity for Executives in the Age of Cloud