Standard, Configurable Repositories to Run Jobs

ACM.442 Creating a flexible architecture and configurable code repositories

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Part of my series on Automating Cybersecurity Metrics. The Code.

🔒 Related Stories: AWS Security | Application Security | Batch Jobs

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post I explained how I eliminated some code and gonig to do that a bit more in the next couple of posts.

In this post, I want to simplify my repository structure used to run jobs.

TLDR;
I'm going to break down the repository as follows:

exec - the repository that has the code that accepts credentials and runs the job.

jobs - the repository with the containers, scripts to manage them, and individual job-specific code.

resources - shared templates and scripts for deployments to a particular platform such as AWS or GitHub.

configuration - the SSM Parameters values and code to deploy them.

Standard Repositories

I started to think that my repository structure is a bit too complex and I want to simplify it.

I want configurable repositories so someone using the framework can basically override any one of the containers to create and run their own jobs.

As a reminder, I have separate repositories because organizations might want to delegate management of different parts of the code to different people.

However, it also helps when someone wants to override one part of the execution framework but use the rest.

For example, let’s say that you want to use the framework but provide your own templates. You can specify a different git repository for the resources and the configurations but leave the rest in tact.

I’m going to break down the repository as follows:

• exec — the repository that has the code that accepts credentials and runs the job.
• jobs — the repository with the Docker files and job-specific code.
• resources — shared templates and scripts for deployments to a particular platform such as AWS or GitHub.
• config — the SSM Parameters values and code to deploy them.

I’m going to prefix the above with 2sl-job for my base repositories. Someone could clone and rename the repositories or use some but not all of the repositories.

Here are my repositories now:

Let’s say you have your own deployment framework but you want to use some of my templates. You can easily use just the templates.

Or maybe you want to override the templates and configuration but use the deployment framework.

You can pick and chose what you want to use.

gitignore

Of course I will add a .gitignore file to all these repositories:

Running a job with the above repositories

In what I was previously calling my “init.sh” script I clone all the repositories. I’ve made those repositories variables that you can change and I’m actually calling the file run_local.sh instead of init.sh. I’m going to simplify my container jobs in the next post and all the jobs will use the same file to kick off the job if running locally (versus inside an EC2 instance or possibly a Lambda function in the future if I can get MFA working).

It may change slightly by the time you see it in GitHub but that’s the idea.

Instead of adding a new repository for every job, the jobs for a single environment will all be in the “jobs” repository. As I started to work with my prior structure it became clear that would be too cumbersome.

Now all the file references can reference those variables and you will be easily able to swap out the repositories as needed. In fact, I may do that for things I don’t want to publish publicly in various environments. I might have a private jobs repository that uses all the rest of the resources, for example. Or I may have private resources used by jobs unrelated to what I’m publishing here. This is a much more flexible approach.

I still need to test it all out but that’s where I’m headed.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2024

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab