Allowing Users to Start Encrypted EC2 Instances in the AWS Console

ACM.97 Requiring MFA, encryption, and disallowing network misconfigurations that expose admin ports and data

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics. The Code.

🔒 Related Stories: AWS Security | Security Architecture | KMS

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last two posts I ended writing a “quick” delete script which wasn’t so quick due to all the dependencies and the order of things. The last post covered dealing with circular dependencies.

Now, where was I? Oh yes, I was trying to use the Developer user created with CloudFormation for whom we autogenerated a password to login into the AWS console and view the user-specific secret we created that contains the SSH key to login into an EC2 instance that we encrypted with a customer KMS key and deployed into the networking deployed with CloudFormation.

Click on each of the links in that last paragraph to see how we got to this point and find the related code in GitHub.

Somewhere along the way I changed a role that caused AWS to delete the ARN in my key policy that caused me all sorts of problems that led me to create a deletion script to try to delete and redeploy all my resources including the invalid trust policy that started the whole problem.

Allowing users to start EC2 instances in the AWS Console

The first thing we need to do is to start our EC2 instance so we can log into it (if it is stopped).

Navigate to the EC2 dashboard and start the EC2 instance we created for our developer. You might see something like this. If you do, you are not logged in as the Developer user the permissions we defined earlier in this series. You are logged in as some other user. That’s OK let’s test it out.

Note that I tried to start the instance as someone other than our Developer user. If you have been following along, what do you think happened? The instance didn’t start. Unfortunately, there is absolutely no error message in the AWS console which is completely unhelpful. In the past, you would get an error message that didn’t really make sense, but at least you knew something was wrong. Now you get nothin at all? Not too user-friendly. But due to experience in the matter I know what the problem is.

Let’s go take a look at CloudFormation logs. Although I thought this was going to be a post on networking primarily, I have to spend time explaining KMS again. Hopefully AWS will make this a bit easier soon.

What is troubling is that when you are using the default view in CloudFormation which is events that are not read-only, you don’t get this item.

However, if you clear the box, which I would expect would give me all events, you don’t see the error:

If you add false to the box you will see the error:

You can also search on the Event source: kms.amazonaws.com and you would see the error. Seems like a bug in there because when you remove the filter to view all events you should get the KMS error. But anyway, click on the error. In this case I’m using an SSO user but it doesn’t matter. If I was using an IAM user it would be the same problem.

“User: arn:aws:sts::xxxx:assumed-role/AWSReservedSSO_xxx/xxx is not authorized to perform: kms:CreateGrant on resource: arn:aws:kms:us-east-2:xxxx:key/xxxx because no resource-based policy allows the kms:CreateGrant action”

As you may recall we added an encryption key to our EC2 instance. The encryption key has a policy that only allows our Developer role and the AppSec role to use the key for decryption (and again, would prefer we could limit it to the Developer user as explained in prior posts but we cannot due to the way KMS works.) The user I tried to start the instance with cannot because they don’t have access to the encryption key.

Instead, we have to start the EC2 instance while logged in as our Developer user. Navigate to the AWS Console. Wow. This is not pretty.

What does “API Error” mean? Seems like it could says something more along the lines of a permission error, which is what is going on here. We need to give our Developer some EC2 permissions. Which ones? Do we need Load Balancers right now? Do we want to allow them to define static IP addresses (EIPs?) Should we allow Developers to create new SSH keys (Key pairs)?

Beware of the * in your IAM Policies!

I like to say a star in your policy is an asteRISK. Sometimes they are necessary but often they are not. Especially when it comes to users. I will usually try to check AWS policies for configurations that are too broad on cloud security assessments and pentests by evaluating such policies to see what they are doing and if the asterisk is necessary.

In addition to what is shown here — *all networking* — on AWS is granted through EC2 permissions. If you add ec2.* to a policy you are allowing that user to change any networking within your AWS account.

I like to tell people that when I was a developer in an on-premises environment, I could not even expose a database or file share to the Internet it would not even be possible. I didn’t have access to change networking that would expose those resources to the Internet. Now organizations are giving developers who have not been trained to properly deploy networking the ability to control it. In fact, in some cases they are deciding they don’t need it at all! And we wonder why so many cloud resources are exposed to the Internet unintentionally.

Developers can learn networking and deploy it properly. If they can learn how to create complex systems in code, they can deploy networking. However, you should still have separation of duties and proper governance in your organization to protect against rogue insiders and unintentional misconfigurations.

Generally, you have a smaller team responsible for networking who takes the time to understand all the details, nuances, and ensure your logging is properly enabled. This should make developers’ lives easier so they can focus on application code — which is what many of them want to do anyway!

So in this case, I’m going to add EC2 permissions to my Developer user, but not the permission to edit networking EIPs, an other things the Developer doesn’t need right now. I will grant read only permission for EC2 to help them troubleshoot issues and the ability to run an EC2 instance. Here are the list of commands for EC2:

Permissions to start and stop an EC2 instance

I’m going to add permissions for:

ec2:describe*
ec2:start-instances
ec2:stop-instances
ec2:terminate-instances
ec2:reboot-instances

Let’s see how that works. As I mentioned before, I’m writing the code on the fly as I go rather than fix it and then write about it. We may need something else related to IP addresses but let’s try it.

To which policy should we add our EC2 run permissions?

Now as you may recall, our developer user is part of the Developers Group and has permission to assume the Developer Role. We are currently logged into the AWS console as the Developer. Where should we add our permissions? Let’s consider our options.

Role policy:

The user has not currently assumed a role. They are logged in as their own IAM user. If we assign the permissions to the role, the user would need to assume the role before they could start an EC2 instance. That would be ok at the moment, but I will show you in another post why I would prefer to assign this policy at the point where the user IAM principal is present.

User-Specific Policy:

Do we need to add this to the user-specific policy? No. All developers will be allowed to start instances and so it doesn’t make sense to put this in the user-specific policy. Even when I get more granular that should not be necessary.

Group Policy:

In this case, the way we are logged in now, our Group policy should apply. Let’s add these permissions to the Group, but we are also going to require MFA. By doing so we will prevent users from starting EC2 instances using the AWS CLI with these permissions. If we want to allow a user to programmatically start an EC2 instance, we will use an AWS role for the reason’s explained in prior posts related to programmatic actions with developer credentials and the inability to enforce MFA. More on that later.

Adding our Policy Permissions to run EC2 instances

Our existing Group Policy only allows our AWS user to assume the Developer Role. As you may recall it’s a generic policy used by every group.

We don’t want to give these permissions to every group, but we may want to allow more than one group to run EC2 instances. Let’s create a generic policy for this purpose that we can limit to the groups that need these permissions.

Copy the existing group policy:

cp GroupPolicy.yaml GroupRunEC2FromConsolePolicy.yaml

Modify the actions and for now, we’ll set the Resource to “*”.

Recall that we created a separate function to deploy a group policy in our group_functions.sh file.

I subsequently changed all the groups to use the same group policy template — GroupPolicy.yaml. Let’s modify this function to pass in the template name instead.

Move the line that sets the template in this function to the function that deploys the group. Then pass in the template name as a parameter to the function.

Now we can use our generic deploy_group_policy function to add our run EC2 policy to our group.

Modify deploy.sh and deploy it.

One other issue. If we deploy both the group policy and the group EC2 policy using the above code both CloudFormation stacks have the same name since we are using the name of the group as the stack name. Let’s change that.

First, we will add policy name to our create_group_policy function and use that for our stack name:

We’ll pass in the policy name in our group creation function:

And from our deploy script:

Now return to the console where our Developer is logged in.

You can tell which Developer you’re using by looking at the top right of the AWS Console.

Refresh the screen and now we can see more information:

We can’t see load balancers and that’s fine. We don’t need that functionality just yet.

Notice that the Developer user can see one running instance. that is the instance I’ve been using to deploy all the scripts so far. I’m going to switch over to using the developer machine in a minute. There is one stopped instance and that is the EC2 instance we deployed with the script from an earlier post.

Click on Instances. Notice the user cannot see Alarm status.

What is that? Look at the list of instances with our other user:

No alarms are set. Click the +. Here we could create CloudWatch alarms and alters if some criteria is met.

The Developer user would require some additional CloudWatch permissions and we don’t need that yet so it’s ok the way it is.

Try to start the instance again. (Crosses fingers.)

Phew. That’s not pretty AT ALL.

For some reason the message is encoded. I think I explained how to view those before. I documented it here for future reference:

Well, here’s what we get. The Developer user is not allowed to perform the ec2:StartInstances action. But why not? Didn’t we grant that permission?

CloudFormation allows deployment of invalid policies

Let’s go back and review our policy. Both our new policies are applied to our Developer Group:

Aha. CloudFormation allowed us to deploy a flawed policy. This has happened before. We see that this policy has 4 errors when we view it in the console. I refer you to my earlier post where I suggest that AWS should be using the same function that validates the policy here in the console that they use to deploy the policy via CloudFormation.

When I created my policy I was using the format of the commands used by the AWS CLI. The commands are in a different format in AWS policies. Unfortunate, but c’est la vie. We need to use StartInstances and change the other actions as well to the correct format.

Let’s try this:

And now…the Developer user can start the instance:

Oh but wait. The instance is in the Stopped state. That’s confusing. The console shows us that the instance started but then it goes to a pending state and then stops again.

Problems with KMS (again)

We have a single KMS key template that we use to deploy all keys. You would expect it work consistently across AWS services, no? It doesn’t. I have had to make modifications already when Parameter Store didn’t support the “via:Service” option. Hopefully that will be added soon. Now what’s different about EC2?

Return to CloudTrail.

User: arn:aws:iam::xxxxx:user/Developer is not authorized to perform: kms:CreateGrant on resource: arn:aws:kms:xxxxx:key/xxxx because no resource-based policy allows the kms:CreateGrant action

The error message says the problem is in the KMS key policy (a resource policy as opposed to an IAM policy. Let’s go check that.

That permission doesn’t exist. We didn’t have to add that permission for other services like accessing SecretsManager or ParameterStore. So why do we have to add it now?

What does this CreateGrant action do anyway?

Following the links for more about grants we read:

Grants are commonly used by AWS services that integrate with AWS KMS to encrypt your data at rest. The service creates a grant on behalf of a user in the account, uses its permissions, and retires the grant as soon as its task is complete.

Services give temporary permissions using the CreateGrant action. We don’t really want our Developer giving out permissions. What else does CreateGrant do?

When you create a grant for a KMS key, the grant allows the grantee principals to call the specified grant operations on the KMS key provided that all conditions specified in the grant are met.

I really don’t think I want my user creating any grants. This, especially, does not sound good:

Be cautious when creating grants and when giving others permission to create grants. Permission to create grants has security implications, much like allowing the kms:PutKeyPolicy permission to set policies.

Wow. We really don’t want to add that permission anywhere we don’t have to per that description. But at the moment I cannot start an instance without it.

Let’s take a look at our CloudTrail message again. This is an interesting property of the user identity block:

"invokedBy": "ec2-frontend-api.amazonaws.com"

I wonder if we can use that in a condition. Let’s review conditions one more time because we had some problems with them earlier in our key policy.

It looks like we can restrict this to the EC2 service the same way we did with Secrets Manager in an earlier post if this sample code is correct:

Return to our key policy. Add ec2 as one of the services that can be passed in via our ServiceParam Parameter.

Next I’m going to add conditions for each service:

Earlier we also had to add CreateGrant when deploying the EC2 instance. I made a comment indicating that was the only reason it was added here:

Instead of adding this unnecessary permission in all cases, I’m going to conditionally add it if the service is EC2.

Using the AWS::NoValue pseudo parameter above to optionally set the value or set no value at all.

I’m going to swap my if statement for the condition in the decrypt ARN block as follows to check for KMS first otherwise use the via service option. If via service doesn’t work for a service (like it did not for Parameter Store last time we checked) then have to pass “kms” in as the service parameter.

I’ll use my conditional addition of the CreateGrant permission in the decrypt statement as well.

Now…let’s see if this actually deploys.

Potential problems when you rename a CloudFormation stack

It seems to be…but now we have essentially changed all four key policies. We’ll have to go back and test that they all work. At the moment I have commented out the deployments for Batch and Lambda since I’m going to change those in a bit. I’m going to test starting the EC2 instance and accessing secrets in Secrets Manager.

I tested to see if the user could still access their own secret after the KMS key for DeveloperSecrets got updated and that worked.

I still couldn’t start the EC2 instance. Do you know what I forgot?

I have to pass in the correct value for ServiceParam which is ec2 for the DeveloperComputerResources key. I also renamed the stack and the key and changed the description to be specific to EC2.

What happens when I rename a CloudFormation stack? It doesn’t update the old stack it creates a new one. When that happens I have to make all these other changes as well:

• My SSO user can’t delete the KMS key and alias so I have to run my scripts to do that in the KMS/scripts directory.
• I have to edit any stack references that are used to get the key ID from that stack output which includes my AppSec Role Policy, Developer Group Policy, and EC2 Instance template.
• Then I have to redeploy all those resources in addition to the key with the new stack name.

Fun, huh? Be careful and test thoroughly when renaming CloudFormation stacks that reference each other.

I can tell which stacks I need to delete by attempting to delete the key. For example, this is my EC2 stack:

Updated EC2 template.

Updated AppDeployment group role KMS policy:

Now here is something else interesting. I would expect that I would need to grant my IAM user permission to use the KMS key through an IAM policy. I don’t. I added these permissions initially with access to the specific KMS key. KMS permissions are not required here.

Then I removed the above permissions. I could still stop and start the EC2 instance.

The only other KMS permissions I assigned to this User was through the user specific script snd that access if for a different specific KMS resource. So apparently you do not have to add IAM KMS permissions to use a Key to start an EC2 instance. You only have to add the permissions in the KMS Key policy.

I then added this new policy to my delete script.

Now we need to see if we can login with our SSH key autogenerated and stored in Secrets Manager in user-specific secret in an earlier post.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2022

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab