It’s Friday.

ACM.340 Sometimes you just need to think on a complex technical problem for a bit

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: AWS Security | Secure Code | Batch | IAM

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Today my brain is tired and I’m watching the birds in my backyard. It feels like it’s been a bit of an unproductive week. But I guess, like Thomas Edison trying to create the lightbulb, I ruled out some things that don’t work. At least not the way I would like.

Here’s a painted bunting for your viewing pleasure. These guys are really shy and this is the best photo I have so far. But perhaps, like my job orchestration strategy, I will have a clearer picture later.

In the last post I pondered my options for running some simple batch jobs. I determined that Batch was just too complicated for my needs and Lambda won’t let me do something really simple — assume a role with MFA.

I really don’t understand why this doesn’t work and keep wondering if I did something wrong.

Plus, I’m not really sure about the security of Lambda for MFA codes anyway. It was possibly a short term solution upon further investigation.

So I’m stuck. Kind of. I have some ideas for simplifying the process of running batch jobs but every time I start writing it down it feels like it becomes overly complex again. I’m pondering all the issues and sometimes when you get stuck — you just have to do something.

There are the things you know and the things you are trying to figure out. These are the things I know.

Containers for Micro-Templates

I have this idea that I can run a container to deploy each resource. That aligns with the concept of my ideas about microtemplates, a concept I made up. These templates run with a role that requires MFA to assume.

I’m not sure how well that will work until I try it. But I’m going to give it a shot. I have this idea of deploying a single resource to a single container so when you update it you don’t have to update everything else you’re running. It’s aligned with the idea of microservices.

Catch-22: To get credentials you must have credentials

The sticking point is basically that to get the temporary credentials, you have to first have some credentials that are allowed to do that. What should those credentials be? Should you have different credentials for every role you want to assume?

And whatever user or process that can obtain the temporary credentials can also use the credentials. I was thinking about restrictions to prevent that and my brain started to melt so I’m saving that for later, if needed. Perhaps you simply have to restrict the credentials that can assume the role to those who are allowed to do the thing the job is doing.

Automation User Credentials

Here’s my initial thought. What if you have credentials that can assume a list of automation roles — but only with MFA. In order to assume a role the MFA token must be provided so the credentials alone can’t perform automation tasks.

That means if someone steals the credentials somewhere along the path to the automation, that’s not enough to do things. The user attacker also needs to obtain a token to perform role assumption with MFA.

Recall that we need user credentials to perform MFA role assumption, so we need to create a user somewhere, somehow that can provide the token that works with the credentials used to assume the roles with MFA.

Now, if I want segregation, I have to create separate automation credentials for each role I want to assume. I have to have an IAM user that can assume an IAM role with MFA and a KMS user that can assume a KMS role with MFA — if I want to require separate MFA devices for different role assumptions and limit certain users to performing certain actions.

A single automation role that can do anything is risky.

I could, if I only had two administrators, potentially add two MFA devices to the automation user and require a different device for the two different roles.

But what happens when I have 5 different roles? That doesn’t work because I can only associate two MFA devices with a user. That seems a bit hokey anyway.

So what I’m thinking is that for each role assumption, there is a set of user credentials that requires MFA to assume each role.

I don’t want those credentials floating around on developer workstations so those credentials will be pulled by an EC2 instance role at runtime. The code will be provided by the user when executing the job that starts the EC2 instance.

So long story short, I’m going to create a user for each role and store the long term credentials (which can be rotated frequently) in Secrets Manager and assign an MFA device to each user. A related trust policy will allow that user to assume the related role with MFA.

For example, we have jobs that run IAM resources so I will create the following:

• An IAMAutomation user with an IAMAutomation MFA device.
• Credentials for that user stored in an IAMAutomationSecret.
• An IAMAutomation role that requires the IAMAutomation MFA device to assume the role.

For KMS:

• An KMSAutomation user with an KMSAutomation MFA device.
• Credentials for that user stored in an KMSAutomationSecret.
• An KMSAutomation role that requires the KMSAutomation MFA device to assume the role.

And so on.

Creating the resources before any resources exist

Who creates the above credentials and how? Because you don’t have any of your DevOps resources setup presumably when you create these roles?

It’s some kind of script. It could use containers, but you’d need to use local containers because you haven’t yet set up ECR to pull down containers from there yet right?

So what I’m going to do long term is run containers that need credentials passed into them. If running on a job-specific EC2 instance it can pull down the containers and pass in the credentials.

If you’re running the container stand alone, you can obtain the temp credentials and pass them in yourself.

I want to make sure the containers work by just running them or in an automated process that pulls them from ECR. Hopefully.

So for my initial testing I can simply obtain temporary credentials and pass them into any containers I’m testing.

Reorganizing resources

I want to reorganize my resources. I may have mentioned this before. But essentially it will align with the CloudFormation resources list with a few exceptions.

I don’t know why EC2 includes networking. Those two things should be separate. I suppose that is a legacy thing.

But for the most part the resources will be grouped in folders aligned with the CloudFormation resources.

For example, AWS has a section in the above documentation for Identity and Access Management (IAM).

So in my Resources folder I’ll have an IAM folder and the role with the associated name — IAM — can deploy any of the resources in that folder.

Then I’ll have a /containers folder.

I can have a /containers/cfn/iam folder for all the containers that deploy those resources.

Then I’ll have a /deploy folder. The deploy folder will orchestrate the jobs — and I haven’t quite figured out how I’m going to structure that yet.

I may change my mind on any of the above. I’m not sure but that’s how I’m thinking about it.

This is all related to my naming convention which already names all CloudFormation stacks with names aligned with the above.

In the end I had [PROFILE]_[RESOURCE]_[NAME] but I’m going to use the role name instead of profile going forward. Profiles will be auto-generated and match the role name.

All CloudFormation output export names matching the resource name will return an ID or ARN — whichever is appropriate.

Obtaining the automation credentials

Somehow, we need something that has permission to obtain the automation credentials. So I’m going to spin up an EC2 instance with a role that is allowed to access the automation credentials.

It cannot use the automation credentials without the MFA token. It also doesn’t have the credentials hardcoded into the instance. It pulls them from Secrets Manager when it starts up.

The user trying to run the automation and the instance has to provide the token when the instance starts. The token is used to assume a role and get temporary credentials. If the token is not provided, the instance logs an error and stops.

The automation role obtains temporary credentials and stores them in a secret.

After the role obtains the temporary credentials it sets the automation credentials to null.

The instance passes the temporary credentials to a container that it runs, plus any parameters provided to the instance at startup.

The container runs, reports any errors, stops, and then the instance terminates.

Somehow I need to track the job state as it runs and once it is complete…something I’ll get to later.

Rotating the automation credentials

Since the credentials are stored in Secrets Manager we can rotate them frequently. However, if someone has a means to get these credentials, they can still get them after they are rotated most likely.

Is it reasonable to run a container for every resource deployment?

Using a separate container for each resource that gets deployed may be too expensive. It’s definitely more expensive than using Lambda functions at first glance. But as mentioned we can see if it’s possible to use spot instances.

I’m not sure about ECS with clusters and autoscaling. If we don’t need much scalability perhaps this option saves money. It definitely should save time compared to managing and deploying ECS for simple jobs. I don’t need APIs, I just need to run scripts.

I could think of a way to run multiple containers on a single instance in the future to complete a job. You could pass in multiple tokens for different roles to run a job, for example and store multiple sets of temporary credentials. But then should you just have one role? Pros and cons?

I also lose some information about which EC2 instance ran the job if I use separate credentials. I need to track job IDs, who ran the job, the instance ID, etc.

By the way, I don’t particularly like SSM because I think it’s a great option for a C2 channel if the credentials are not managed very carefully. I thought that immediately the first time I saw SSM. Apparently others did too because that’s what PACU has built into it. Had these guys at my meetup in Seattle and they demonstrated their C2 channel and I immediately knew they were using SSM. I asked them afterwards and my hunch was correct. Unfortunately, the author of this tool is no longer with us, but his legacy lives on.

I also like the idea of encrypting my EC2 instances with a customer managed key on a host I can completely configure and lock down (eventually — I’m going to start simple.)

For the extreme security use case you could also consider an instance with an enclave:

Will this even work?

I don’t know. Let’s find out.

So what I was doing before I got down this rabbit hole was to run a simple set of jobs to deploy a static website. What I want to do first of all is clean up all the code I’ve written so far and check it in with a big caveat that it’s a POC mess. :)

Then I’ll try this new approach and try to finalize deploying my static website resources. Or something like that. I’m thinking out loud — or on a keyboard essentially.

Off to clean up and check in.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab