Separate Repositories for a Job Execution Framework, Job Images, and Job Configurations

ACM.434 Also building docker containers with files from different directories or contexts and problems with buildx on Amazon Linux

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Part of my series on Automating Cybersecurity Metrics. The Code.

🔒 Related Stories: AWS Security | Application Security | Batch Jobs

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post I explained how I want to restructure my accounts to support the job execution framework I’m building and why I need a separate container for the initial organization deployment — because there is no job execution framework when you initially start deploying.

Here’s where I make some changes I’ve wanted to demonstrate for a while. I want to create separate repositories for each job and the job configurations so people can create and run new jobs without changing the core job execution framework.

I’m going to create separate repositories for separate components. That provides the flexibility to easily add new types of jobs (containers) which I need to do now to create my root management account initialization container.

AWSJobExecutionFramework — A repository for the framework code that executes jobs using a common set of code. When someone deploys a job they can start with this reusable code base. This code base has the code to run the jobs only, nothing job-specific.

job-awsorginit — This repository contains a Dockerfile for the initial container image used to deploy the organization, initial user and role, and the first account and organization administrator in a new AWS account. You can’t use the job execution framework when the job execution framework hasn’t been deployed.

job-awsdeploy —This repository contains a Dockerfile used to build a container image that uses the job execution framework — meaning it requires an SSM parameter that contains the job configuration and a secret with credentials in the same account to deploy the job.

jobconfig-deploy — This repository contains the configurations for deployment jobs. In order to deploy a job using the awsdeploy job, the job configuration needs to first exist in SSM Parameters store. This repository maintains the job configurations separately from the code that runs the jobs.

Later I can add things like:

job-awssecurityassessment — A job to run all the tools I use and have created to assess the security of an AWS account, for example.

jobconfig-awssecurityassessment — a repository that contains the configurations to run security assessment jobs.

job-awsamibuild — A job to create new, approved AWS EC2 Amazon Machine Images for an organization.

jobconfig-awsamibuild— a repository that contains the configurations to build Amazon Machine Images. Multiple configurations can be added here to deploy different types of images or the same image in multiple environments, for example.

Separation of duties

If you’re tracking with me here, you can see how you could separate duties and only allow certain people to change certain repository code. Someone could always propose a change to another repository to make that code easier to use with a pull request if they can see the code — without having access to make the changes themselves. That way your core security functions are protected while people can freely configure new jobs as needed using approved framework code.

Accessing files in different directories in a docker build

Now I have a problem. Docker build will only allow you to access files in the same context as the Dockerfile you’re trying to build. What that means is if I’m trying to build the Dockerfile that’s located in my job-awsdeploy directory, then I have to put all the files that I want to include in my image into that directory.

I have common job execution framework code that I want to include in every container executed by the job execution framework. But I also have code related to the specific job in the repository where the Dockerfile is located.

I don’t want to copy the other repository code into the same directory or it may be copied inadvertently to my job container. In addition I want to be able to pull down updates to the job execution code easily from GitHub or CodeCommit as the code changes. So I want to leave that code in its own directory associated with the git files.

How can I solve this problem? I can use multiple build contexts as shown in this post. It’s also explained in the Docker documentation but not so clearly. Unfortunately this site tries to force you to look at a lot of ads.

Essentially what you can do is use buildx which contains some extensions to docker build. I can specify multiple named “build contexts” (file locations) when I build my container.

docker buildx build -t [tag] -f [Dockerfile location] \
--build-context framework=[framework path]

Then I can reference those contexts when I copy the files using the from directive.

From [base image]
WORKDIR /job/
COPY execute.sh /job/execute.sh
COPY --from=framework /container/resources/ /job/resources/
COPY --from=framework /container/shared/ /job/shared/

You can include multiple contexts if you need to as well, each with a different name:

docker buildx build -t [tag] -f [Dockerfile location] \
--build-context framework=[framework path] \
--build-context external=[some other external resource]

Install buildx

To use buildx you need to install it. On Amazon Linux:

sudo yum install docker-buildx-plugin

Check the version:

docker buildx version

If you’re not using the most up to date version the contexts feature may not be available to you.

Unfortunately installing this feature on Amazon Linux gives me an older version. To use this contexts feature I had to manually upgrade buildx.

You can find releases here:

I found a work around on this AWS post in theory:

Specifically this code:

export DOCKER_BUILDKIT=1
docker build --platform=local -o . git://github.com/docker/buildx
mkdir -p ~/.docker/cli-plugins
mv buildx ~/.docker/cli-plugins/docker-buildx
chmod a+x ~/.docker/cli-plugins/docker-buildx

Many problems here. You are pulling code from GitHub directly. Over the Internet. You’ll have to open up a private network or use a NAT. I don’t see a checksum here to verify the integrity of the files. It also takes a long time to run. It doesn’t work.

The reason it doesn’t work is that it’s using a deprecated protocol, namely git:// which tries to access GitHub on port 9418 and I presume is unencrypted.

While trying to run that I got the same error as when I tried to run buildx alone and I found the code producing this error:

ERROR: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to read dockerfile: failed to load cache key: failed to fetch remote git://github.com/docker/buildx: exit status 128

I tried changing to https and that didn’t work either, so I consulted the buildx instructions for a local build.

None of the options worked. I ended up running this, even though the third option seems like it should have worked:

cd ~
git clone https://github.com/docker/buildx.git
cd buildx
sudo make install

Then I got an error because I needed access to some CloudFlare domain so I had to fix that. Later I had to allow yet another IP range for Python.

Then it killed my little instance. I had to free up some space and try again. Nope, I need more memory and CPU.

All this because I can’t get buildx from AWS repositories via yum.

So I have to stop this instance, create an image, and launch a new instance with more memory because that process is apparently managing swap memory. With more memory shouldn’t have the swap issue. Right?

Not what you thought you’d be reading about in this post? Me neither.

I run a medium instance. That’s much better. Top CPU usage I saw was 15%. No swap.

That finished in maybe like 3 or 4 minutes.

Hmm. I am still seeing the old version:

To fix that I ran the following commands (output from make install above):

mkdir -p ~/.docker/cli-plugins #no sudo
sudo install bin/build/buildx ~/.docker/cli-plugins/docker-buildx

After doing that I get the updated version:

Unfortunately still getting the error I started with:

So at this point I run the command to get the docker version details:

docker version

My system has version 20 and the latest is 24.

So I run another update because I’ve recently made some network changes. For some reason an AWS update requires access to cli.github.com (???) I really wish AWS would make this stuff work from a local yum repository but more on that later.

I get a bunch of updates.

But no docker updates:

But after unsuccessfully trying to follow the docker documentation to add the docker repo to my configuration, I found this obscure comment:

I added this comment at the very top of my Dockerfile:

# syntax=docker/dockerfile:1

That fixed the problem so I could build the container.

Build the container to deploy the root files (awsorginit)

Here’s the code I end up with to build the root container.

First, I have the new directories associated with the above repo so for this test I’m using these two contexts:

• job-awsorginit
• AWSJobExecFramework

I rearranged the directors in the AWSJobExecFramework as follows:

• container — everything that goes into the container image
• scripts — scripts to build the container image and push and pull the image, etc.
• ec2 — still has what will be the ec2 userdata code
• gen_code — needs some updates but still has code generator code
• testexecute.sh — still the same but I’ve added a bit of code to test the container build with multiple contexts. I will need to change this further later to support the awsorginit job which doesn’t use the full job execution framework.

job-awsorginit/Dockerfile

I might be able to get rid of some of these resources later but the main goal was to try to get the new context directives working. I’m in developer mode, not security mode right now. 😉

job-awsorginit/execute.sh

This file needs to be updated to include the existing root script plus a few additional resources. I’ll cover that in a later post. It’s the file that gets executed when the container runs and is specific to this job so it lives in the job repository, whereas the run.sh file is the generic entrypoint that lives in the job execution framework respository and should be the same for every job.

AWSJobExecutionFramework/scripts/build.sh

The build script presumes that you have cloned the job execution framework repo and the job images you want to build to the same folder.

In other words, I have the following in a directory:

/AWSJobExecFramework
/job-awsdeploy
/job-awsorginit

The test file still lives in the root of the AWSJobExecutionFramework and all file paths are relative to that root directory (because bash).

The script now expects an image name (job name) to be passed in as the only argument. Using that name and the expected directories, the script can calculate all the necessary paths.

Since we are using different contexts and the Dockerfile doesn’t live in the current directory, we specify the docker file location with -f.

I also use the build-context directive to define two build contexts — one for the job execution framework and one for the job (container image) directory.

testexecute.sh

When I run that I get a list of jobs and copy and paste the job name I want to run.

I enter y to build the container:

For some reason the container seems to take longer to build but at any rate it works (once I got all the file paths right).

I should note that although it says it copied the files I have not yet verified that and had some issues with CloudShell that I will write about in the next post.

Building the awsdeploy image

I can use the same process to build the awsdeploy container image. I still need to adjust the Dockerfile the same way and I was working on the execute.sh file for that job when I got sidetracked once again with other things to fix. That’s all on the way.

Running jobs

Once I get my job execution framework set up, I can use my testexecute.sh script as is. However, for my initial job I’ll need to handle the credentials a bit differently. Also, I still can’t push and pull a container to and from ECR until I set up ECR. I wrote a while back about this idea of having “environments” and so I’m getting back to that — building out a whole environment to run jobs — because after I do that things should be easier.

The whole environment issue sidetracked me from finalizing my web deployment and the whole environment issue led to an issue with my organization setup so here we are. Hopefully I’ll get all the things done I wanted to do in the first place. I think I’m close.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2024

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab