A New Strategy for Batch Job Orchestration

ACM.339 Thinking through my options and a simpler approach for my particular use case

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: AWS Security | Secure Code | Batch | IAM

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post I was sorting out my Batch roles and trying to run a test job. I wanted to minimize costs so I chose options to that end — and I never could get the job to run. Perhaps if I chose a wider range of resources I could have got the job running but I already don’t like the fact that I cannot choose the smallest instance types for simple jobs so I started rethinking my options.

Jobs are not always big and monolithic. They can be, but there are jobs that consist of many smaller executions to complete a process. Different steps in the process may require different resources and different permissions.

In the case of cloning my GitHub repo I need a tiny amount of resources. A Lambda function would probably be ideal but I cannot do what I am trying to do with a Lambda function. I also wonder if encrypting an EC2 instance with my own customer managed KMS key would be cheaper.

I wanted to use AWS Batch or AWS Lambda because it would be easier and maybe cheaper. But is it?

One of my goals was to lock down a container to a task-specific role. The other was to orchestrate a bunch of containers — which will be clearer later. I of course want it to be secure, and I want it to cost as little as possible. I like the idea of sizing my resources to match my job. Deployment jobs may be tiny. Different types of penetration testing jobs require either more memory or more compute depending on what the test or scan is doing.

So let’s see. For the tiny jobs, the cheapest instance I can get for AWS Batch is the c6g.medium. The very cheapest instance types are not available to me. I also have to run an ECS Cluster. How much does that cost?

Well this is cool. With ECS you only pay for the EC2 resources.

There is no additional charge for Amazon EC2 launch type. You pay for AWS resources (such as Amazon EC2 instances or Amazon EBS volumes) you create to store and run your application. You only pay for what you use, as you use it; there are no minimum fees and no upfront commitments.

What’s the smallest instance type I can use with ECS?

With ECS I can choose the “t” instance types which cost less, as I showed in a prior post:

So perhaps I can just use ECS instead of AWS Batch.

But do I really need a cluster if I’m just running small batch jobs?

It really depends on what I am doing, but right now I want to run a tiny job and a medium size instance seems like overkill. It also seems to be unavailable — or something. I couldn’t get the job to run. The time to configure and troubleshoot a new service is also a factor.

If I had to schedule a bunch of financial jobs, I might want a very robust scheduling process. But my needs are far simpler.

What if I just launched the cheapest EC2 instance type for a tiny job? What if I used spot instances to run my jobs? Hmm. Any jobs using the same role (like IAM jobs for example) could run on the same instance.

The only issue is that I’d need to track the success or failure of each job. That’s what Batch is doing for us. But if I use an automation script to run a bunch of jobs sequentially then perhaps I don’t care about all that.

Batch is just turning out to be way more complicated than I need, and Lambda won’t let me do what I want in my container. ECS feels like overkill for my immediate use case.

My container already works. I’ve proven that. So perhaps I just need a custom way to run containers one after the other or trigger the deployment of the smallest instance that can run a single container. Or all the containers using the same role can run on the same instance. I’ll have to think about that as I’m testing.

What’s the smallest instance type and how much does it cost?

What if I use spot instances?

I manually created and tested to see if I could choose a spot instance for the smallest instance type and it appears that I can.

There are whole companies built on monitoring spot instances and trying to schedule workloads cost-effectively. In any case, I can try out this theory because I already have a container that starts with one role and then assumes a role to run a job.

The thing is, I can probably skip the whole first role altogether. In fact, I’ve done that in the past. I ran an application to perform a task (or batch job if you prefer) in a container and assumed a role using MFA to run it.

So I’m back to square one and what I used to do in the first place. I just need to streamline my process a bit more.

Essentially what I want to do is to put each function to create different resources in its own container or perhaps there’s a container with all the IAM functions, for example. You can instantiate the container to deploy a resource.

Then I can string together a bunch of calls to different containers that have different permissions to complete a process. Something like that.

I might end up using queues but for now I’m going to keep it simple.

I also want to reorganize my source code repository, so I’m thinking about checking in what I have and starting fresh.

But first I’m going to remove the Lambda code from my container. I’m also going to get my container and EC2 instance setup and logging to CloudWatch so I can track the status of jobs.

There are so many options for how I could do all of this. I’ll start simple and move on to more complex things as they are required.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab