Components of an AWS Batch Job
ACM.6 Considering the composition of our batch jobs
This is a continuation of my series of posts on Automating Cybersecurity Metrics and one of my posts on
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Part of my series on Automating Cybersecurity Metrics. The Code.
🔒 Related Stories: Batch Jobs | AWS Security | Cybersecurity
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the last two posts I explained how batch jobs can help cybersecurity:
and why you might want to use MFA with batch jobs:
You’ll need to configure a number of things to create a batch job on AWS. I went through a quick tutorial to create a batch job manually in the AWS console just to get the idea what I’d need to think about when constructing a batch job. Those components are listed on this page:
Compute environment: The compute resources you want to use for your batch job. That means what AWS compute resources you want to use like Fargate (containers) or EC2 (VMs). You can configure the compute resources to have different configurations which will affect your batch job performance and cost.
Job Definitions: A template that defines what will happen when a job runs. I provided a lot of examples of what you might want to do with a batch job in my initial post on batch jobs for cybersecurity:
Jobs: The template (job definition) defines what a job will do. The job is the actual execution of your template or job definition.
Job Queues: Job queues handle the scheduling and management of multiple jobs. You can associate multiple compute environments with a job queue and assign priorities to jobs.
To get a feel for these components, I just ran followed the tutorial and manually created a job to see how it works.
Thinking through building the jobs the way I want is a lot more complex than that. I have concerns about who can kick off jobs when and how will I know if they were successful or failed? Where will the logs end up? Who can view the data I pass into the batch jobs? How much will it cost? Getting something to work is not architecting a solution, but in order to understand the components of your architecture, you’ll need to start somewhere.
My experimentation with batch jobs is going to be a bit free form, as time allows, and intertwined with other things I need to get done, but hopefully if you follow along you’ll understand where I’m going — and how I think about securing things in the cloud.
As I mentioned in another blog post I wrote, my code is written in phases. The first phase is always quite rough to flesh out a concept and improve it over time. I usually try to go back and reduce the chance for errors and simplify my code after the fact as I wrote about in this post (hopefully part of an upcoming software security book also on my to-do list):
The code I’ll present is not production ready and can always use improvement. I’m figuring things out as I go. I already mentioned on Twitter I wrote a JSON template parser to create reports which I may share but it’s very rudimentary and specific to my needs.
But before I can get to all that, we need to think through some issues related to how we will run our jobs. How will we assign permissions, protect data, and do we have any network considerations?
AWS provides some sample batch jobs such as this one which grabs a script from an S3 bucket and runs it:
Another option is to use a docker container. I need to run things that are a bit more complex than a single script. I want to have my batch jobs assume a role with limited permissions and install some libraries to help carry out tasks. Right away, I decided that I will use containers, not a single script.
The other benefit of a container is that there are various methods we can use to ensure the integrity of our code as it passes from dev to QA to prod, something I talk to clients at IANS research about a lot. Failure to ensure code integrity was the underlying cause of the Solar Winds breach so we’ll want to think pretty carefully about that if we are running sensitive batch jobs.
Docker containers it is! If you want to get a feel for AWS Batch and the elements involved, run through the tutorial above or run the sample CloudFormation templates. I’m going to be focusing on containers in upcoming posts as before we can create a batch job with a container — we need a working container.
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2022
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
