Container Escape Vulnerability in AWS Hot Patch

Update or mitigate now if you are affected (if you run containers, you probably are.)

One of my stories on Data Breaches and Container Security.

Free Content on Jobs in Cybersecurity | Sign up for the Email List

Thank you to David Jones at Security Dive for bringing this to my attention.

Palo Alto’s Unit 42 discovered a vulnerability in the AWS Log4Shell hot patch. This patch got applied to almost every type of compute resource on AWS, so you may want to update immediately if you are using an affected system.

Note that this vulnerability does not require you to be running Java or Log4Shell, so its impact could be potentially greater than the Log4Shell vulnerability itself to AWS Customers running containerized applications.

The patch is designed to automatically patch vulnerable applications. The problem is that the patch itself is not containerized, essentially. It runs in a process that has access to the entire host.

You can get into the weeds in this Palo Alto post:

Palo Alto offers the following mitigations in their article (quote):

* On standalone hosts, you can upgrade by running yum update log4j-cve-2021–44228-hotpatch (RPM) or apt install — only-upgrade log4j-cve-2021–44228-hotpatch (Debian).

* On Kubernetes clusters that installed the hot patch to every node via the hotpatch Daemonset, you need to manually connect to every node and update the installed hot patch via yum or apt. Note that only deleting the hot patch Daemonset doesn’t remove the hot patch service from your nodes.

* Hotdog users need to upgrade to the latest version.

Alternatively, if you’re confident that your environment is patched against Log4Shell, you can disable the hot patch service on a host by running sudo touch /etc/log4j-cve-2021–44228-hotpatch.kill. To disable Hotdog, run apiclient set oci-hooks.log4j-hotpatch-enabled=false.

Hopefully you have an automated deployment that enables you to quickly redeploy all VMs and containers with the latest version. That should help resolve some of the issues without the need for specific patches, but check the particulars above to verify. That includes your EKS hosts and containers running on them. You might think that is difficult to automate, but that is exactly what we showed students how to do in one of my AWS class labs. We automated deployment of EKS and the containers running on it. It is possible, and I’ve done it.

I’ve just finished a presentation on Developer Governance for IANS research and will be presenting that at their Dallas and Los Angelos forums this year. If you can’t make those check for an IANS event near you were someone else will be presenting and speaking to those slides. I cover topics like automated deployments. Additionally, you can schedule a call with me at IANS Research if you have questions about how to create automated, immutable deployments and secure deployment pipelines.

In addition to being able to quickly update containers, hopefully you are running in a zero-trust environment with proper networking controls and monitoring to help discover any attacks that may result from this vulnerability.

Finally, if you find all this patching to be complicated, consider cloud-managed services that handle some (not all) of the patching for you automatically. If you are using AWS Lambda, for example, the patching of the hosts would occur automatically under the hood. You still need to keep the code you run in your Lambda functions up to date. I was just talking to a client at IANS Research about that. Sometimes the size of your team drives you towards particular AWS services.

One other consideration here is that Palo Alto discovered this vulnerability, not Amazon. Palo Alto has a vested interest in finding vulnerabilities which they can incorporate into their cloud products and services that report security problems in cloud environments. Their product now reports this problem they discovered which can help customers quickly mitigate the issue.

What AWS does not have is a bug bounty program that pays other researchers with no such motivations to uncover and report bugs, like the other top cloud providers do. Amazon does. AWS does not. Check the scope.

Security issues discovered in the AWS IP Space (https://ip-ranges.amazonaws.com/ip-ranges.json) are not in scope for Amazon Vulnerability Research Program. As an infrastructure provider, AWS customers operate assets in this space. Discovering and testing against AWS and AWS customer assets is strictly out of scope for Amazon Vulnerability Research Program and against the AWS AUP (https://aws.amazon.com/aup/).

AWS just hopes that researchers will spend their free time finding and reporting bugs here.

Microsoft has a bug bounty program, though my experience with it recently wasn’t great. They suggested I talk to support instead of responding to the issue and what was causing it, if it was not in fact a security problem (which according to others it was). You know how I feel about contacting Microsoft support if you’ve been following along. I hope they can fix those issues but at least they have a bug bounty:

Google has a bug bounty program. I have not reported anything here but I know people who have been paid significant sums through this program:

If a company has a robust bug bounty program that pays well, that means more people are working to find vulnerabilities on the platform and will take the time to report and thoroughly explain them to receive payment for their time and effort. Luckily, Palo Alto had some external motivation in this case.

I hope that AWS will reconsider its stance on bug bounties in light of this issue. Some people report vulnerabilities on AWS for the fame and glory of talking about them at DefCon. How many other vulnerabilities were not reported or followed up on appropriately because AWS does’t have a bug bounty program? There’s no way to know for sure, but I wish they had one.

As you might have read in my last post, I’m an AWS Hero and I’ve loved using AWS for a long time. AWS has some of the best security engineers and in the world and I know some of them, but as this vulnerability shows — nobody’s perfect!

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2022

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab