Leveraging Containers in Lambda for Governance, Secure Configurations, and Portability

ACM.288 Containers everywhere for consistency and compliance

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: AWS Security | Container Security | Lambda Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post I pushed a container to my AWS Elastic Container Registry in preparation to deploy it in a Lambda function.

Now I want to use that container in a Lambda function and I’ll show you how to do that. But first I want to tell you about why you might want to use containers in a Lambda function. Now mind you, I haven’t done this at scale so this is an experiment and research for me as well as those who follow along as I go through this to determine any gotchas, costs, and issues with this approach. But I want to tell you who containers can help you with your cloud security in so many ways.

When I was recruited away from Capital One to help a security vendor move to the cloud, I said, “I’ll do it if I can architect things the way I want.” Initially, I had the most amazing boss who was on board with that, but after time at any organization, things change. I’m not a fan of politics and my experience at that company is one of the main reasons why I run my own company now, even if I could be making more money somewhere else.

The outcome at both companies in regards to security breaches was not so great at either place after I left. Not that I would have had the ability to stop those breaches given the organizational structure and dynamics. But in both cases I did my best to provide guidance and warnings where appropriate. And at one of the companies I designed a secure deployment system that both gave developers pretty free reign to write code while removing some key security risks that could expose the organization.

Did that security architecture matter? Well, here are a couple examples.

• I inherited an AWS account when I went to work there. I deployed Cisco Stealthwatch Cloud* because I know some people over there who used to work with the US Government and they had spoken at my AWS Meetup. The product immediately spotted rogue traffic going to Brazil. My team was doing nothing in Brazil at the time. (That changed later.)
• Secondly, a product manager set up an AWS account and spun up and deployed an EC2 instance that was almost immediately infected with ransomware. Let’s just say that opening port 9090 is a bit extraneous when all other ports are already open.

By the way, StealthWatch cloud has a new name now. I have heard some people say they are moving away from Cisco for various reasons, but the results in that instance speak for themselves. Perhaps a demo in your account might reveal something interesting. I haven’t used every product so I can’t compare them all, but it’s worth trying products before you buy them to see what they find because in both instances where this product got deployed, I got alerts to problems in those accounts (neither of which I set up or controlled, i.e. none of the governance I am writing about existed in those accounts.) You can get a free trial here — and I am not paid for this. Just telling you my experience.

In every case — security is hard. Even companies who seem to be doing everything right get breached. It’s frustrating when companies make obviously bad architectural choices and leave vulnerabilities wide open or test software in a risky manner that puts customers at risk. However, even if companies aren’t doing any of that and trying to avoid risk as much as possible, it’s really hard to close every single gap and make no mistakes.

I gave a presentation on Application Security, DevOps and cloud security at CounterMeasure up in Ottawa, Canada back in 2018 and one of my slides was “It only takes one mistake…”

This is where the principle of abstraction comes into play in security. I’m talking about my version of abstraction in cybersecurity, not a different angle on abstraction in cybersecurity I read about in a security class. There are multiple definitions or takes on abstraction. My definition is based on concepts of abstraction in software architectures.

If you can move all the things that are common about an architecture into one or fewer deployment scripts, there’s less change of a configuration error. If everyone is doing everything their own way then that could lead to problems. Designing security and infrastructure architectures with the concept of abstraction in mind can help your organization be more secure. You need some guiding principles and technical governance to really do this effectively — and people who know if and when to break the rules based on business needs and potential risk exposure.

Freedom within boundaries

But what am I even talking about? Let’s design by example. At the company where I architected a solution to move an on-premises application to the cloud, I wanted to allow developers to develop freely and fast, within boundaries where the organization would not be completely exposed. I have felt the pain of draconian reviews where I can’t get things to production in a timely manner, held up by one single person in an organization. That’s not fun.

On the other hand, I’ve seen people do crazy things at a bank like deploy packages off the Internet straight into production due to lacking governance. Yikes. (That was fixed, but not before I was called “paranoid.”)

How can we balance letting developers freely experiment and deploy things while taking away the things that pose the most risk — wide open networks, systems with unpatched vulnerabilities, and lack of proper encryption?

Well, there’s a whole host of misconfigurations that can be avoided by having developers deploy everything in a container. The developer has free rein in development environments to download packages from trusted sources (with security checks like those provided in GitHub and some others I’ll get to later.)

The developers do not have free rein to change the networking. The network is set up such that packages can come from trusted sources allowed on the network (a whole separate topic). A well-designed network also prevented one of our developers from deploying company code to DockerHub. He wasn’t trying to do anything bad but he wasn’t supposed to be doing that. So we got together and asked him what he was trying to do and showed him how to do it in an approved way. He could still do what he needed to do, just differently and without exposing company assets.

If developers have EC2.* permission in your account, they have permission to change your networks. Make sure you understand IAM permissions and like I said in the last post — the * should be spelled asteRISK — something that has been in my classes since I started teaching them. Please credit me if you use that. 😆

Anyway, here’s the other thing. If you deploy everything via containers, do developers need EC2 permissions at all in the environment where you deploy applications? No. In my case, I had a DevOps team that deployed the networking and EC2 instances in a secure and compliant manner. The developers could basically deploy whatever they wanted in a container (if it passed automated security checks and, when going to production, a review by a senior team member.)

In the development environment developers got a secure base container to start with and could deploy whatever they wanted on top of that. If they wanted to use some other type of container, we’d have to build a new, secure base (like arm versus x86).

Developer Workstations and EC2

Now at that company developers all worked on their own workstations but I have since come to love the idea of working on a cloud host — as long as it has proper networking.

If you can’t control the networking for a developer environment you can’t control things like the above scenario where a developer was pushing corporate code to DockerHub. So whatever solution you use needs proper networking as I’ve explained in many other posts.

You could have developer workstations in the AWS account where you’re deploying containers or in a separate account. In the same account and region means you won’t have to set up network peering for certain activities which could cost more money. On the other hand, your policies might be easier to write and you have inherent trust boundaries when you put the developer workstations in a separate account from the one where you are deploying your applications.

Network Security and How Containers Can Help

Just remember that if developers (or any technical person for that matter) can change that networking they probably will. I’ve seen it and it’s often not pretty. So lock that down and limit network changes to those who have been trained in network security and understand things like C2 channels, data exfiltration via DNS and other proxy-like services, and how exposure allows attackers to use stolen credentials and leverage security vulnerabilities.

I often explain that when I started as a developer, I couldn’t deploy all the misconfigured resources developers are deploying in cloud environments — because I had no permissions to change those resources.

And that’s why misconfigurations are so prevalent in the cloud. People are making changes who have never been trained on security and don’t understand the implications. Networking is one of the best ways to help prevent breaches WHEN credentials get compromised. Your network needs to be designed appropriately and securely. Proper networking makes alerting on suspicious behavior easier. Proper networking can limit the scope and blast radius of a breach.

If you presume that most web applications need 443 in and ephemeral ports out you can support like 90% of front end application use cases for networking. Beyond that you might use standard ports for access from web components to your middle tier and a handful of database ports from the middle tier to the database. For developer working in standard use cases no networking changes are required at the subnet and NACL level and you can lock out egregious access more quickly with stateless firewall rules.

Of course you have nasty things like Active Directory that use a billion ports if you try to deploy things like that but that’s beyond the type of thing developers are trying to deploy in containers for applications (I hope.) When I had to design and deploy the bastion host it was a one-off subnet that was only used for the bastion host and could only be changed by those with permission to redeploy that network because that was a critical security resource. But for the most part you can use standard subnets for most applications. Then you only have to deal with the security groups, and as I’ve shown throughout this series, that can be automated. That way requests for new applications can be handled more quickly with standard templates.

For those one off use cases where people are using random ports to access various things you have a work flow for approving such exceptions — the kind of workflow I worked in and adjusted for Capital One when I was there, with 5 people deploying networks for 11,000 developers. The complexity and your needs will greatly vary by the size of your organization — something I often talk about to IANS Customers who contact me on consulting calls.

Anyway, the idea is that you move the riskiest parts of your infrastructure to what I call a DevOps or Infrastructure team that is trained in security for those components and know what they are doing. The developers have free reign to do what they do best — write code! Without being blocked for most standard use cases when deploying a container in the network provided for them to use. For the one-off uses cases, hopefully you have a streamlined process for making exceptions.

Also — some developers do not like deploying networks. Networking is hard to get right. Like I said, developers want to write code! They want to get their ideas to fruition quickly and the thing they are trying to test is often not the network or the encryption key. Some strange people like me actually like deploying infrastructure, but for those who don’t, setting up an environment with a pre-defined network where it is easy to deploy and try out new things will make people happy.

I’ve noticed this most in the serverless community where people don’t want to deal with servers and VMS — and if they don’t want to deal with those things they certainly don’t want to deal with networking! Every person is different in terms of how deep they want to get into the infrastructure. Putting the right people on the right teams and making their jobs easier will make people happy. But remember, some people like me — like infrastructure problems and challenges. Maybe those are the people you want on your DevOps and security teams.

Experimentation

The other thing you’ll want to do is have a Sandbox where developers can deploy and test new things, probably including EC2 instances. Sometimes new services come out and developers want to see how they work. Additionally, it might be easier to learn something new by first clicking around in the AWS Console. I’ve also found that the AWS Console sometimes provides different and better error messages when you are trying to troubleshoot CloudFormation.

For these reasons it’s a good idea to have a developer Sandbox where some developers are allowed to try out and experiment with new ideas and resources that may not be allowed in your standard software development pipeline. Then the developer can demonstrate what they are trying to do and security teams can evaluate the risk of those changes.

I’ve already written about this but just a reminder of why a Sandbox account is a good idea and why it may be needed. I have some other posts on AWS Organizations and account structures here.

Portability with containers

One of the great things about deploying everything in a container should be portability. We are not quite there yet in all cases, but ideally you could take the same container from Kubernetes and deploy it in AWS Lambda. There are reasons why that won’t work yet related to runtime as I wrote about in this post:

But I think we are approaching this possibility and there is definitely the possibilty that your deployment system could help developers choose the correct container to deploy on different platforms. The same code deployed different places. Perhaps this could be done via a security best practice — multi-stage container builds.

Multiarchitecture containers are also possible, though I had some challenges with this last time I tried it. Maybe I will revisit this later.

In any case, you can provide developer secure base containers that align with your corporate governance and compliance requirements but still make it easy for developers to deploy their code. That’s the goal. Developers don’t have to worry about networking if they are following a standard deployment model but have an easy path to request changes if needed. A central team keeps the underlying platforms secure and up to date if you are using EC2 — or AWS handles some of that for you if you are using Lambda.

I think containers have a huge potential to reduce security risks and I hope to demonstrate this as I go. I have lots of ideas related to containers batch jobs, applications, security metrics, and deployments I hope to get to eventually. More on this topic as I work through additional posts on container and application security.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab