AWS CLI Session Compromise
ACM.145 Threat-modeling AWS assume role temporary credentials
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Check out my series on Automating Cybersecurity Metrics. The Code.
🔒 Related Stories: Cloud Governance | IAM | AWS Security | Okta
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In my last post I wrote about who is responsible for what when it comes to AWS security.
I had to stop and write that because I’m going to link to it below when it comes to some of the threats I’m going to cover in today’s post, which is all about sessions. Specifically, I’m going to take a look at how AWS tracks CLI sessions and how attackers can obtain and abuse them.
Why might an attacker want to obtain a user’s AWS CLI session? Because if they can obtain a user’s CLI session credentials, they can do whatever the user is allowed to do — without requiring MFA. That’s because the user had to provide the MFA prior to obtaining the session credentials. After that the session credentials are no longer required.
What might an attacker do with session credentials? Here’s one example: I wrote on CreateUser privilege escalation and backdoors.
I also wrote about a potential mitigation here… but wait, there’s more.
Before I can got to the more (there’s [at least] one other threat we need to consider), I need to cover the topic of AWS CLI sessions and temporary credentials — what they are, where they exist, and how an attacker might obtain these credentials.
AWS AssumeRole action
When you assume a role you get some temporary credentials that get rotated periodically:
Once the credentials exists, MFA is no longer required. The credentials can be freely used from that point on with no MFA prompt. The credentials exist for whatever amount of time you have configured, or the default.
These short term credentials and sessions cannot be revoked in all cases, but you can change the permission of the IAM principal that uses them. When you restrict the permissions for a role — you would affect all sessions for that role, not just the session of a single identity. I hope that AWS will provide better mechanisms for revoking a specific set of temporary credentials for a single session in the future. #awswishlist
AWS CLI
Where do those credentials exist? It depends how and where you request them. If you are assuming a role with the AWS CLI, the temporary credentials are accessible:
- In the CLI detailed debug output, as I explained in this post:
- Potentially to an attacker that has phished an SSO user as explained in this post:
- Over the network when requested
- In memory
- Cached (stored on disk) on the host that requested them
An attacker may also trick a host into requesting and presenting these credentials externally, using things like open redirects, SSRF, and DNS rebinding which I’ve spoken about in presentations before, but that is not the topic at the moment. We are talking about sessions that required MFA to activate but that step has already been completed. The temporary credentials or other session identifier can freely be used with no MFA requirement.
Let’s look at the potential places a malicious actor could obtain your session credentials in more detail for the last three bullets that I haven’t already written about.
Over the network (in transit)
In this case, when I talk about “over the network” I’m talking specifically about intercepting the credentials in transit when your EC2 instance (or any other resource that can obtain temporary credentials) requests the temporary credentials and the response comes from the AWS service that delivers the credentials to the compute resource.
I can think of a few types of access an attacker could obtain your credentials this way. The first would be that the attacker would have to be on the compute resource (EC2 instance, container, Lambda function, etc.) and be able to intercept traffic between the EC2 instance and AWS using a proxy or network sniffer.
In order to do that the attacker would likely need to get malware onto the machine, and if they can do that it’s probably easier for them to just read the cached credentials. If they couldn’t read the cached credentials or are trying to be more stealthy or something they might try to read the network traffic via a sniffer or proxy on the compute resource itself.
Another attack could occur if the attacker could somehow redirect your request for these credentials to some sort of proxy before getting the AWS service that provides the credentials. Let’s say an attacker sets up a compute resource with a private IP in your private VPC. They somehow trick your computer to send your temporary credential requests to their computer resource instead of the metadata service.
The attacker forwards your request along and then sends the response back that requires your MFA code. Then when you send your MFA code the attacker grabs your code, obtains the credentials, and maybe sends you back an error so you try again and get a different set of credentials.
The above will not be possible if you have zero trust network rules in place to prevent this, even on your local network. It could be that somehow this is prevented by the AWS hypervisor or by using some sort of host identification mechanism like a private key that prevents such an attack. I’m not going to explore whether this is or is not truly possible here, just explaining what an attacker might try to do. That’s the type of thing I would dig into in a penetration test or bug bounty — if one existed. #awswishlist
The other point of attack would be internally at AWS. If the traffic was not properly encrypted and protected internally at AWS, an AWS insider might be able to sniff and intercept the traffic once it leaves your system and passes through the AWS systems that handle your request for temporary credentials. That portion of securing the credentials falls in the AWS portion of the AWS Shared Responsibility Model, which I showed you how to analyze more in depth than this basic photo in the last post. This is the type of detail you won’t get out of the diagram below, but we do know that once the request leaves our control, it is AWS’s responsibility to secure it.
In Memory
Various types of malware read memory on a host through a myriad of methods I explored in a reverse-engineering malware class and an advanced penetration testing class. I don’t always have time to perform these attacks on a penetration test but working on some additional automation so I can do more in less time. As I wrote in my last post, the more basic testing and reporting can be automated (part of the reason for this blog post and my related efforts) the more time I can spend on customized, advanced attacks. There are many.
I presume when you request the credentials and before they are cached they are likely in memory as well as whenever you need to use them. You could explore this further on whatever operating system you use with different types of tools used for forensic memory analysis. There are also penetration tools that grab information out of memory such as Windows tickets that are used frequently in attacks on Windows systems. The same concepts and types of tools can be applied to the AWS credentials while in memory.
That said, again, there’s probably a simpler way to get them in most cases — the credentials are often cached on the host that requests them. Let’s look at a few different ways you can run the AWS CLI.
Where do AWS CLI credentials get cached on hosts that require them?
Let’s take a look at different implementations of sessions for the AWS CLI. There’s a typical place where sessions are stored on an AWS EC2 instance. Before I show you that, I want to see if AWS CloudShell works the same way and if it is easy to find the temporary session credentials in AWS CloudShell.
AWS CloudShell for an AWS SSO user
Note that with AWS CloudShell you do not need to request temporary credentials, unless you are trying to use some role other than the one you are using in the AWS console. Other blog posts explain how to do that, and if you specifically request credentials, they are going to end up wherever you store them — in memory, environment variables, or on disk. That’s not what I’m trying to find out in this post, but something I will definitely be covering later in more detail.
I ran through a series of commands and as you can see I didn’t have to put in any additional credentials or an MFA token. I could run these commands simply because I logged into the AWS console (and I need permission to run the command I ran, which is aws ec2 describe-regions). The MFA token is required on login, not upon running the commands. It doesn’t seem like there would be a way to require MFA for a particular action.
In this case, somehow related to my login and clicking a link on the SSO dashboard for a particular account and permission set associated me with an AWS role. I presume somewhere behind the scenes an AssumeRole command got executed on my behalf. Let’s find out.
aws sts get-caller-idenity
Yes, my identity running the AWS CLI commands is a role which I would expect based on my description of using AWS SSO with the AWS CLI here:
Where are the credentials? They could be cached on disk. The could also be in environment variables. Let’s look around.
Hmm. Not exactly what I’m seeking. I want to know what my credentials are. Let’s dump the environment variables.
This is more interesting. But I’m not going to take this any further. As you can see there’s a credentials URL and a token. That’s interesting but doesn’t work the same way the credentials work on compute resources you instantiate. I’ll leave any further exploration as an exercise for the reader, within the bounds of the AWS penetration testing terms. CloudShell is not in the list and AWS doesn’t have a bug bounty. #awswishlist
Note that remote code execution flaws have been discovered in both Azure and Google Cloud Platform CloudShell implementations — here are a few examples (and hopefully AWS has addressed all similar threats):
Now let’s take a look at CloudShell for a non-AWS SSO (standard IAM) user
It’s pretty much the same. It seems that any attacks would be applicable to both.
Restricting access to CloudShell
You may or may not want your users using CloudShell. It is super handy. It likely has similar controls to the AWS console itself. However, it probably has a larger attack surface and probability that an attacker could abuse it since it runs in a browser. You also may have users that you only want to grant access to see things in the console rather than execute commands.
If for some reason you don’t want your users to be able to use CloudShell you can restrict it:
You could also limit the use of CloudShell to less risky actions. Or you could limit it’s use to a short period of time for risky actions, in which all activity is monitored, such as I will probably demonstrate in a few upcoming posts.
Session credentials on an EC2 instance — SSO
Here’s where things get interesting. I’ve been using a host created on AWS from an SSO account, but I never went through the process of establishing an AWS CLI credential session with AWS SSO. I don’t particularly like the process. Yet, when I look at my EC2 instance, I see cached credentials for AWS SSO in the following directory:
/home/ec2-user/.aws/sso/cache
Curious.
If you look in those files, those are, in fact, what appear to be session tokens. I wonder how they got there. I don’t believe I have ever gone through the AWS SSO CLI authentication process. I never took the code I got back from the AWS SSO process and entered it in the browser. So how did those credentials get there?
Also, my blog post came out on January 9th, and the credentials above are from January 10th. I definitely wasn’t using AWS SSO to obtain the credentials after I wrote the blog post. I generally also write my blog posts a day or two before they are published. This is somewhat concerning, but I’m going to just leave that there and move on. I don’t plan to user AWS SSO users for a while because I’m going to be testing other things with IAM users. If I got money to investigate every interesting finding I ran across in a week or even a day…
Running commands with an AWS developer key and secret key
When I run commands on the EC2 instance the way I’ve been showing you, I use developer credentials assigned to a normal IAM user. I assume a particular role I created for automation purposes.
When I assume that role to run my desired commands, I have to enter my MFA token, even if I’m logged in somewhere else like the AWS console.
Now, you could trick me into entering a token somewhere, but it just feels like that’s harder to pull off than the AWS SSO approach. Also, this is not my final solution. It is only a stepping stone towards and end result.
The unfortunate thing is that I have to store those credentials on my host in a file called:
/home/ec2-user/.aws/credentials
That’s where your credentials end up when you run the AWS configure command. But if you have designed your IAM roles such that a user must enter an MFA token to run the commands, then the attacker cannot simply run them using the credentials. They also need MFA. And hopefully you rotate these credentials frequently and monitor for use of credentials previously rotated.
So After you enter your MFA code, you get back your temporary credentials, what happens to them? Take a look over in this directory:
/home/ec2-user/.aws/cli/cache
Aha. These are temporary credentials.
If you look at one of those files you’ll see:
{"Credentials": {"AccessKeyId": "xxxxxx", "SecretAccessKey": "xxxxx",
"SessionToken": "xxx..." ...You could leverage those tokens anywhere to run commands and they will not require MFA since it took MFA to obtain them.
It’s not any different than logging into a web application where you have to login, enter your MFA token, and then once you are logged in you can take further actions without providing MFA. You have what’s called a session. As long as that session is active you can take actions on that website. Any attacker who can get your session token or id can do the same, using your identity, without MFA, if there are no network restrictions on the actions taken with your session identifier. Web site session tokens must be handled with great care and protected.
The same is true of an AWS CLI session. And that’s what attackers are targeting to take actions on your AWS accounts in a lot of security incidents these days.
How can we limit risk if a session is compromised?
At some point, anyone can be phished or tricked to click on a link that grants an attacker access. There’s no way we are going to completely eliminate phishing, though there are a lot of things we can do to limit the number of successful phishing attacks.
So I’m going to assume that at some point an attacker can get session credentials somehow.
The best thing you can do in a cloud environment is the following:
Design your IAM credentials so that it takes more than one person to perform risky actions, such as I did in this post:
I am not addressing all the other considerations an controls here that I’ve covered in other posts like encryption, networking, and secrets management to limit the blast radius. Those things will also limit what an attacker can do with your credentials and potentially help you spot an attack (if you are monitoring suspicious activity). Additionally, you need to have processes and controls to keep systems up to date, which I haven’t covered.
But probably your number one control, since credentials are the number one thing attackers use to break into and take actions on your systems, is to limit what any one set of credentials can do, even if those credentials are assigned to the same user. The risky credentials are only used as needed and when performing other actions, use less risky credentials.
Now, although I’ve designed my system so that the system requires two roles to create a user and grant access, I still have another concern. Hint: It’s in one of the screen shots above. I’ll write about that tomorrow.
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2023
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
