Reorganizing AWS Batch Service Documentation

ACM.335 Documentation is key to getting anyone to use your cloud service — or all of them if you can maintain consistency

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Part of my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: Batch Job Security | IAM | Container Security | Deploying a Static Website

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post I spent two days trying to sort out the AWS Batch documentation to understand what IAM roles and policies I need to create.

Which leads me to this post.

I’ve spent way more time than seems necessary trying to sort out the AWS Batch service documentation. I took some time to organize it for anyone who is trying to understand AWS Batch below.

Additionally, these recommendations apply to any AWS Service. If people can’t understand what your service does or how to use it — they won’t. When people told Steve Jobs no one would pay for music via the Internet because they could just steal it, he said “They will if you make it easy.” That’s from this book on Jobs by Walter Isaacson.

Not to be confused with AWS Batch Jobs.

😊

But in the words of Einstein — things need to be simple enough, but not too simple. I need to be able to maintain security in my AWS account. I need to know how those things work and be able to track and maintain the policies and governance in my account.

And I need to understand how these things work for all AWS services, by the way, not just AWS Batch.

It would be so much easier to do if AWS had consistent documentation across services to give me the information I need, both as a developer trying to deploy the service for the first time, and as an architect or security professional trying to design the enterprise use of the service properly.

To be honest, Azure has the most consistent documentation menu and UI layout at the moment. GCP used to have a really good overview page for each service, but it seems to have changed. I explain how those things work below.

It seems like every AWS service should have a consistent menu and headings that makes information easier to find. There should be a consistent to things like the following in the left menu:

• Storage and encryption
• Networking
• IAM
• Best Practices — for security, performance, and cost management
• Troubleshooting

Storage, networking, and IAM are pretty much the three pillars of security controls. Then there’s architecture — how it all works together. Let me see these things at a glance.

The documentation should be as simple as possible and not redundant with other documentation across the platform. But not too simple so I have to reverse engineer things to really understand how they work. Wizards are for newbies, not architects.

Well-secured and performant platforms are generally designed by architects, and no matter how much AI you throw at it, that will always be true to some degree. AI can help but I doubt we are anywhere close to it taking over. So explain it to me in a straightforward manner so I can get things done.

Give me references to common things in a consistent manner like AWS CLI commands and IAM actions.

But most of all organize the documentation in a clear manner so I can understand the service, what it does, how to use it, the architecture, and the security, performance, and cost options easily.

As I wrote in another blog post, access is most commonly the sticking point. Fix that. AWS Batch has been one of the most challenging services I’ve tried to use or write about recently and I didn’t really expect that. I figured it would be easier Like Lambda. I just want to run a job. But I need it to be more secure than Lambda appears to be able to offer at the time of this writing. I like the idea of encrypting my compute platform, for example. I want to run roles with MFA.

This access issue can be addressed better by every AWS Service I have reviewed, I think. I’ve provided what is needed for IAM for Batch below, as an example. That is applicable to any service.

If I were to rewrite the AWS Batch documentation an fix the UI I would write it like this:

[THE FOLLOWING SECTION COULD EXIST FOR EVERY SERVICE]

Batch Service Overview

• Create a consistent overview page across services. GCP used to have a great example of this but they have gone backwards in their documentation. The overview page should include a diagram with architectural components and links to the documentation for each component. It would be nice if all AWS Services had this consistently at the top of the documentation and even instead of the marketing page or at the very least, linked from it.
• FAQ
• Remove the page that explains how to set up an AWS account, etc. and link to a global single source for that. It doesn’t need to be repeated in every AWS Service a different way.

[BATCH SPECIFIC SECTIONS LINKED TO FROM THE OVERVIEW PAGE]

Batch Compute Environment

The compute environment defines the compute resources on which your AWS Batch job containers run.

You can choose from the following compute environments

• AWS Elastic Container Service (ECS)
• Kubernetes
• Fargate
• A single EC2 instance (node)

These three headings should move under Compute Environment. Why are they under scheduling?? They also seem a bit redundant.

• AWS Batch on AWS Fargate
• AWS Batch on Amazon EKS
• Elastic Fabric Adapter

Possibly because the UI is mis-designed and should have the parallel and single node options under compute environment. At least, it doesn’t make sense to me. Choose to run your jobs on a single node or multi-node compute environment. Remove redundant configuration in the job definition UI that is also in the compute environment UI.

Jobs and Job Definitions

• All the information about containers and their configuration

Triggering and Scheduling Jobs

• All the scheduling links including the job scheduling and job scheduling policy links should fall under one category, plus the Event Bridge Link and Orchestration should fall under this category.
• Job queues fall under job scheduling

[THE FOLLOWING SECTIONS COULD EXIST FOR EVERY SERVICE]

AWS Security Responsibilities

Provide service specific AWS security information (not a link to the AWS Security home page or an archived white paper.) Cover security standards from NIST or CAIQ to show how the Batch service specifically secures customer data with diagrams

IAM and Resource Policies

• There should only be one section on IAM, not two.
• Remove and link to repetitive documentation from the IAM documentation.
• Create lists of roles, policies, trust policies, and resource policies required for each of the following, with sample CloudFormation templates with the minimal required permissions.
• Then add separate examples for expanded permissions and explain when they are required.
• Be very clear on which element of the architecture requires which role. Use a diagram.
• Included any recommended conditions in the policies. If it is a global recommendation for all services, link to a single IAM documentation page across services.

There should be a section for each of the following with a link to the above information if redundant or in the appropriate section if specific to that compute environment:

• ECS Roles and policies [list of roles and policies linking to examples]
• Kubernetes Roles and policies [list of roles and policies linking to examples]
• Fargate Roles and policies [list of roles and policies linking to examples]
• Single Node Roles and policies [list of roles and policies linking to examples]

Storage and Encryption

• Data flow diagram showing where you can use a customer managed key to protect data at rest and where you cannot for each compute option (ECS, Fargate, Single Node, Kubernetes).
• The data flow diagram should indicate any data that is not encrypted in the process in transit and at rest with a customer KMS key.
• Any information about data stored by the service and how to configure it
• EFS documentation
• ECS, single node, Kubernetes: link to encrypting an EC2 instance in EC2 documentation.
• Fargate: link to encrypting ephemeral storage and how to do that in Batch.
• Explain whether parameters and environment variables are encrypted end to end or not if a customer managed key is supplied.
• “Specifying sensitive data” should move here but possibly should just be a link on the page that describes encryption to the other service documentation.

Networking

• The networking is more straightforward to me, but provide documentation to keep everything on a private network with private IPs. Reference VPC endpoint and NAT documentation — don’t rewrite it. Explain what VPC Endpoints are required for each service such as KMS VPC endpoints if using encrypted AMIs.

Logging and Monitoring

• Move all the content about logging and monitoring (i.e. CloudWatch and CloudTrail) under this heading.
• Using the AWS Log driver should move here

Best Practices, Troubleshooting, and FAQ

• Security
• Performance
• Cost Management
• Troubleshooting

Global Features

Certain global features work across services and recreating the documentation is redundant — like tagging.

• Just say that Batch supports Tagging and provide a link to the global tagging documentation.
• Provide a link to CloudFormation resources
• Provide a link to the CDK documentation for this service
• Provide a link to AWS IAM actions for AWS Batch.
• Provide link to the commands or functions for the AWS CLI and each of the SDKs
• Provide which regions the services works in.
• Provide compliance information.

Anyway, I’ve been thinking about that for quite some time and wanted to write about it. This seemed like the opportune moment and example — and I hope this helps others understand AWS Batch a bit better as well.

Back to building my batch job…

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab