Structuring Accounts For A Common Job Execution Framework

ACM.433 Revisiting the catch-22 of deployments from the root account

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Part of my series on Automating Cybersecurity Metrics. The Code.

🔒 Related Stories: AWS Security | Application Security | Abstraction

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post I created a common job execution container for running batch jobs.

Creating a Generic Job Execution Container

ACM.432 Reducing complexity and creating a flexible container that can run any type of job, not just deployment jobs

medium.com

As I started working on the next script I hit this problem. The job execution role needs to access the SSM parameters so they need to be in the same account where the role was created.

Then I decided the parameters needed to be in the same account as the EC2 execution role so it could pull the list of parameters and secrets, etc. But I forgot about my prior dilemma and was trying to use a role from a different account to execute the job.

In the end, the easiest solution (and least complexity to prevent security misconfigurations or issues) is to ensure that the EC2 instance, EC2 instance role, parameters, secrets, and job execution role are all in the same account.

That led me to think about in which accounts I wanted that job execution framework to exist.

Components of our Job Execution Framework

Based on the last post here are the components we need to deploy resources using the job execution framework I’m creating.

An EC2 instance
An EC2 instance role
Job configuration SSM Parameters
A user that can run jobs with an MFA device
Secrets with credentials associated with the user
A role that can execute a job

So I will need to deploy the above in any account where I want to run jobs.

In addition, to the above, the EC2 instance needs access to an ECR repository to pull down containers and possibly a source control repository. It’s simpler if all those things are in the same account. For now I’m going to say they will be.

I don’t need all that in every account because someone executing a job can assume a cross-account role to take action in another account. So I only need these things in the accounts where jobs get executed, not all the accounts where jobs take actions. Limiting job execution to a subset of accounts will make it easier to manage, monitor, and secure deployments.

Deploying resources in the root management account

Now I need to think through the scenario of deploying this framework in the root account for things we need to deploy in the root account. Do I even want to do that?

We want to use the root account as little as possible. Actions in that account are not governed by service control policies.

When I set up the account, I create a new user with some scripts that do not require any special configuration.

I do that because when you start with a blank slate, there’s nothing there. You have to deploy all the infrastructure you want to use to deploy things. I’d need to deploy all those resources for the job framework to use it.

The root user runs these scripts to create a break glass user in case we can’t access the IdP which I’ll eventually get back to deploying and configuring.

Initially that new user created what is currently named the “root-org” account. That account is used to deploy the rest of the organization. What I actually want is to get the job framework into that account and use it from there.

Once I create the org-admin that user can set up the whole job framework and any other authorized user or role can use it.

But what I was trying to do in the first place was deploy the org-admin. Do I really want to set up the whole job framework in the root account?

No.

All I really need to deploy in the root account are the scripts above, the new account, and the new user that has permission to take actions in that new account to deploy everything we require for the job execution framework.

I was having that root user in the root account deploy the root-admin user and then the root-admin user deployed the new account. That way if you needed to make a change to the account you could do that with the root-admin user. What I think I am going to do is simply use the root user to deploy the initial resources in one quick script and make sure the root user has access to those resources.

The “org” environment

The other thing I have been thinking about is how do you deploy resources in the governance and security account? Where do you put the jobs and the repositories accessed to deploy those resources? Are they just production repositories or something else?

I also run security assessments and penetration tests out of other accounts. I would consider those “production” accounts but I don’t want those accounts to have access in any way to my organization or my production applications. I don’t want anything to infiltrate my organizational repositories that should not.

So really I have separate “production” environments which I want to have separate job execution frameworks, code, and containers. I was trying to figure out where I was going to deploy the jobs, parameters, roles, secrets, and repositories.

That led me to a change to my organizational structure. I’m going to create a separate “org” environment with the “org-admin” account below that. Everything else is mostly the same, but governance and security fall under this new org parent OU.

The org-admin account will be where all the job execution framework components exist. Similarly the nonprod-admin account will be where all the job execution framework components exist for the nonprod environment.

Separate jobs deploy resources in the “org” environment from the jobs that deploy things in the “nonprod” environment. People who deploy things in the “org” environment can test them first in the “nonprod” environment. The credentials used in the nonprod environment will be in a separate account from those used in the org environment (or production, or penetration accounts, etc.)

The one thing I would need to think about is how code moves from non-prod to production, for example. Rather than a “push” which is normally what it tends to be called when you move code from dev to QA, for example, I think it is going to be a “pull” from the org-admin account to production. An execution job in the org-admin account can orchestrate that deployment pull.

So after way too much noodling over all of this, my next step is going to be to revamp the initial script I run when I create a new account to create the new root user, the organization, the org OU, the org-admin account, and the org-admin user, role, and group.

Once I have those resources, I can build out the job framework in the org-admin account. Then I can use that to redeploy the organization and resources using the new and improved job framework.

I also have some ideas I wanted to look into related to that initial script and AWS may have just made that easier. So I’m going to explore that in the next post. I think. I keep starting a topic, hitting an issue, and realizing I have to implement something else first. But generally you start from the beginning and back to the beginning is where this problem led me. So pretty sure that is what I’m going to deploy in the next post because there’s no place left to go up the chain. 😆

Separate Repositories for a Job Execution Framework, Job Images, and Job Configurations

ACM.434 Also building docker containers with files from different directories or contexts and problems with buildx on…