Threat Modeling for Containers

ACM.7 Security considerations for containers used by batch jobs

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics. The Code.

🔒 Related Stories: Container Security | Application Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last few posts we’ve been looking at how batch jobs can help security, MFA for batch jobs, and the components of a batch job:

The batch jobs we will create will likely involve containers based on preliminary investigation. We’ll want to make sure we secure them properly.

I answer a lot of calls for IANS Research about Kubernetes, AKS, EKS, GKE, Docker, containers, and everything related to it. For example, how do you secure your deployment pipeline? What about secrets, encryption, and network security? What about APIs that run on containers or “serverless”? Networking? Whenever you use a new technology or service you need to understand how it works, what attacks exist related to that technology, and what security controls are available to you to secure that technology.

Stepping through all the things I need to think about with an AWS Batch job is a long list. I’m going to start with one aspect of it — the container itself. I’ll touch on some related infrastructure issues, but I’ll dive more into those when we actually set up an AWS Batch job.

Need more help? If you want me to review your cloud and application architecture and make recommendations, schedule a call with IANS. For non-IANS customers contact me on LinkedIn for a penetration test or security assessment of your cloud environment and applications. If you are just following along and want to read what I write here it helps when people create a membership and follow me on Medium as that's pretty much the only way I get income from sharing this information. That income enables me to write more often. Otherwise, I'm happy to help out and writing about what interests me as time allows. Hopefully it interests you too and helps a little. :)

There are numerous ways to use containers and many security considerations which vary depending on the specific architecture you use. In this blog post, I’m going to start by thinking through some of the considerations for a container used as part of a batch job. We’ll need to look at all the various layers of security that apply to our batch job but it starts with the container and how it’s configured.

This blog post is really me thinking free form about the things I’ll need to address when I create my batch job and I’ll probably write about a lot of it in future posts. It’s not meant to be a complete list but it’s a starting point. I also share some stories below that you might want to know about if you’re operating in a cloud environment related to some of the controls if you are wondering if they are overkill.

We’re thinking about this container in relation to AWS Batch. If you’re running a container on Kubernetes, you have a whole lot more to think about when it comes to security controls. In fact, I just helped someone on a call related to batch jobs running on Kubernetes. If you opt to use AWS Batch, some of those concerns are offloaded to AWS by way of the shared security model in the cloud, but not all.

Which controls belong to you? I like to say that if you’re using a cloud platform, if you can see it, touch it, and change it, you likely own it. You'll need to make sure you secure it properly. 

However, you should really review the legal agreements you have with the cloud provider first and foremost, followed by the documentation, which is not necessarily legally binding. Talk to a lawyer about that.

Here’s what we know about how AWS secures AWS Batch and what you can do on your side to further secure your batch jobs:

When we’re thinking about the containers we’re using we will want to make sure we follow all the container best practices available to us though the CIS benchmarks. In my recent presentation at the IANS summit in Dallas I talked about CIS coverage. It’s a good starting point, but not enough. We’ll also to understand vendor guidance and all the ways in which containers may be attacked to try to ensure we can prevent those attacks.

Networking:

• You can configure networking within a container, but that is not that helpful if an attacker can get into the container and change it once they are in there. We’ll want to consider that threat.
• If we can run a batch job in a private network that can help ensure that attackers can’t get to containers from the outside or send traffic to and from the Internet.
• However, even if we can run in a private network, the containers likely need to communicate with DNS servers can be used for data exfiltration or command and control. That’s a topic I want to explore further if time allows. Other protocols like ICMP have been used similarly as you can read about in my paper on the Target breach written in 2014.
• A container in a private network operating in a cloud environment is trusting that the cloud environment is secure. What if cloud insiders obtain access the container? You might want to review this scenario with your cloud provider to see how they handle their insider security.

How much access does your cloud provider have? I was at an OWASP meetup in Seattle. A person started talking to me but did not ask what I do for a living (teach and test cloud security). He started talking to me about how he was proud of the architecture and the thought put into the security of that particular cloud platform. Then he went on to tell me that unfortunately, they didn't think through the ability for their own employees to bypass security controls and had some issues with that, but they were working on it. I asked him what he went and he explained a few of the issues. 

Ask your cloud provider how they prevent insiders from accessing your data for each cloud service you use. Better yet, get it in writing, in a contract.

• Besides insiders, what if an attacker were able to breach the cloud platform and get onto the cloud provider network without their knowledge? How secure would your batch jobs be in that case? We’ll want to think through that and try to construct an architecture that protects us as much as possible in that scenario. Networking is a key part of any architecture that helps you block and spot attackers — regardless of what the identity zealots tell you. You need both. Especially since stolen or abused credentials is your number one risk in the cloud.

Attackers in the cloud networks. Recently, while using Azure in preparation for a class, I set up very stringent networking using various service controls available for storage. At some point, I got an error message stating that I could not access my own storage because an IP starting with 20.x.x.x was not allowed to access the data. Odd. That is not my IP address. That was an IP address in the Microsoft IP range.

I submitted the error message to Azure support thinking maybe one of their services had a bug. I found many bugs while trying to use Azure in that last run through my class materials and while testing new services. 

The message I got back eventually from Azure support said something very vague about an internal service they couldn't tell me about discovering an incident which they were addressing. A security incident? I don't know. I don't use Azure for anything production-related so I just made a note of it and moved on. 

But if it was a security incident and I had just set up my storage with no networking controls, I would have not known that was happening. If it was a security incident, that just goes to show you that you shouldn't blindly test the cloud networks on which you operate. 

Even if it wasn't a security incident, something strange was happening because my access was being blocked due to access by an IP address other than my own by the networking restrictions I put in place at the service level. At a minimum I helped Microsoft find a bug of some kind. I was too busy at that moment to look into it further.

Bottom line: Mind your networking. If we were operating Kubernetes or docker in some other environment I’d be talking to you about other options but in this case we’ll be looking at what AWS offers in terms of networking controls for AWS Batch and what we can do on the container itself.

Secrets and Encryption:

• Will we need to access any user names, passwords, AWS credentials or otherwise from our batch job? We’ll want to think through how we protect those secrets.
• If we are deploying a batch job from dev to QA to production we want to use different secrets in each environment. I might not get to that topic here for a while or at all because I have a lot of ground to cover. But you can call me up and ask me at IANS Research.
• I wrote about the encryption fallacy in my last book. If we are relying on encryption, but an attacker obtains credentials and is able to execute the encryption and decryption commands in the cloud, then our encryption is pretty much useless. We’ll want to architect our security with that possibility in mind.

Given that I’ve already done some testing and a POC of using MFA with a container, I know that we’ll probably need to leverage some AWS credentials to assume a role that requires MFA. Unless I can think of a better solution, we’ll need to think through how we store those credentials and what happens with the MFA code we want to pass through to run our container and ultimately our batch job. I have some ideas how to do this but I’m not yet sure they will work. And those ideas lead to a whole new level of complexity and security considerations because we’ll end up passing around an MFA code.

One step at a time…let’s just start with how we can protect secrets on our container. I’ll address all those other concerns if and when we can actually get MFA working.

IAM ~ Identity and Access Management

Credentials are your number one problem in the cloud due to many reports such as those I reference in my IANS presentations. The next one I’ll be presenting at will be LA in October if all goes according to plan:

IAM considerations:

• How can we set up minimal permissions for our batch jobs?
• How can we introduce segregation of duties to minimize risk? I’m a huge fan of segregation and will be covering some ideas for that I’ve been playing around with lately on AWS.
• As I’ve already mentioned I’m interested in using MFA to kick off a job. Maybe someone at AWS is already solving this for me. :) #awswishlist
• How can we ensure there is no path to escalation that allows someone who can assume a role in a batch job to do something nefarious they wouldn’t otherwise be allowed to do? Not sure if I’ll get to this particular topic in this series as it’s more applicable to a large organization than to me in my own account, but if you are at a large organization it’s something you’ll need to consider.
• If an attacker can obtain access to a host running containers or access through a container to the host, the attacker can potentially leverage the metadata service to obtain credentials. We’ll want to take a look at if and how that might be possible and how to prevent it.
• How much could an attacker access if he or she obtains access to a single batch job? We’ll want to think about the blast radius, a term I describe in my cloud security book mentioned at the bottom of this post, along with the key things that can affect and increase it.

Container configuration considerations:

The configuration of the container itself and the operation of containers could also pose a security threat, beyond storing secrets and accessing cloud data via IAM roles. Some of these responsibilities belong to AWS but must fall to us. Here are just a few considerations we’ll need to think about:

• Is the container running as root?
• Could someone obtain access to PID1? (I will be using Amazon Linux.)
• We’ll want to make sure directories can’t be mapped to root or anything with higher permissions.
• No access to docker socket or any of the controls for docker or the underlying orchestration system.
• Ensure no secrets are available due to the way the container gets built or the underlying layers.
• Limit what an attacker can do once they get onto a live container.
• Integrity of our container code after it goes though the build process into production.

These are just a few of our considerations. Many books and resources exist on container security if you want to dive much deeper than I’ve just gone but we’ll look at this a bit more in upcoming posts.

Application, container, and infrastructure code considerations:

When it comes to the application running on the container, the code used by the application could allow an attacker to obtain access to things within the container or the network. This is a whole additional layer beyond the container itself and some of the things I test web applications for when I perform cloud penetration tests, along with everything else listed here.

• Security of all our deployment code used for the container, the application code running on it, and the cloud infrastructure.
• Rogue third-party libraries.
• Injection attacks of any kind.
• New CVEs discovered in the the libraries used by our containers (vulnerability management).
• SSRF, DNS Rebinding, open redirects, and other attacks that would allow an attacker to obtain access to resources on private networks even without direct access.
• In my next post I’ll write about how an application running in the cloud could be leveraged to obtain AWS credentials. That could be an issue for our batch jobs and we’ll need to be aware of the possibility and defend against it. Applications can be used as C2 channels to access the underlying host, in addition to the attacks in the last bullet that can provide access to AWS credentials.

Anything that you open up your network to might be a source that an attacker can send commands over and you will think it is normal, expected traffic. Any type of injection flaw may lead to some way for an attacker to obtain or leverage credentials. Any means of leveraging credentials on an application might allow an attacker to obtain access to data in our cloud environment. We’ll want to think about how we can protect shore up the code in our batch jobs to prevent attacks.

You can start with this series of blog posts I hope to turn into a book, which started out with me trying to refresh AWS credentials in Python and led to a discussion on injection in a later post:

This is not a complete list of the security issues you may need to consider in relation to your batch job. It will depend on your programming language, code, what libraries you are using, and what your application does. The list would change if you were using something other than AWS Batch. But this will help you understand some of the considerations you’ll have in terms of threat modeling for your batch job. This list provides some topics for my upcoming posts where I’ll write about how to address some of these issues.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2022

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab