Lambda Networking Best Practices
ACM.267 How A VPC might have protected your application from a Log4J vulnerability attack — or not if you’re using GitHub cloud
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Part of my series on Automating Cybersecurity Metrics. The Code.
🔒 Related Stories: Git Security | Application Security | Secure Code | AWS Security | Network Security | Lambda Security
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the last post I wrote about saving money on S3 bucket encryption because I was discussing it with a customer.
I’ve been jumping around a bit because I’m considering how I’m going to implement the next piece of the puzzle in my deployment pipeline for our static website and because I have other work to do at the same time.
I was watching a bunch of Lambda and serverless videos and ran across this presentation labeled “AWS Lambda networking best practices.” Although the video is very good up until the end I beg to differ on the “best practices” in the title. The video should instead be titled, “How to set up Lambda Networking to connect resources.” Leave off “best practices.” The video is very good in that it explains Lambda networking. However, the recommendations at the end open customers up to security risks.
I already told you and showed you why you need Lambda networking in this post. Without proper networking I can abuse your Lambda function.
I could potentially use a different DNS server which sends traffic to the wrong ip address for a DNS name. How is that useful? Potentially I could have your application redirect my request to the AWS metadata and extract credentials using DNS rebinding. I’ve explained what DNS rebinding is in various presentations.
I could point your code to a different AWS server to make requests to AWS service go to fake AWS services at a different IP address than the legitimate IP addresses.
I could point to my server to download malicious software from a rogue IP address instead of the software from your own internal service where the software is supposed to be coming from or some other trusted source.
If you invoke some rogue function or software in your Lambda function, the potential exists that an attacker can trigger a malicious downloaded to execute commands or download rogue code in your Lambda function. Malware that exists simply to load other malware is called a loader. You can read some articles about malware loaders here:
Alternatively, search Google for “malware loader” and start reading.
I could use DNS for DNS exfiltration of stoeln secrets stored in your environment variables in your Lambda function (which should be in AWS Secrets Manager instead.)
Unfortunately if my code has the ability to read variables in the function that are not properly scoped, it will be harder to stop the exfiltration at all. I could potentially also grab session tokens to make calls to other resources.
An unaware developer who doesn’t understand security and the implications might download code after the Lambda function has been deployed and injected untested, unverified, and potentially malicious code into a production pipeline. Too many times malicious or lookalike packages get inserted into deployment pipelines or developers pull them down from Internet repositories and the malware ends up carrying out attacks on the network where the code got put to use.
Malicious code someone got injected in the deployment of code during the SolarWinds breach — one of the biggest breaches of the US government to ate.
How networking protects your Lambda functions from Log4J style attacks
Let’s say you use Java in your Lambda functions and you’re using some code that is vulnerable to Log4J. I wrote about the log4j vulnerability that sent security teams around the world scrambling here:
The key point is that the code that handles the malicious line that get injected into your logs to trigger this problem is most likely going to try to reach out to some address on the Internet in the process.
Here’s a detailed analysis of actual log4j attacks:
Now, a reverse shell is going to be hard to obtain in a Lambda function unless somehow the attacker can get their code cached into some layer that is reused by many functions and has built some sort of re-runnable process. However, an attacker could more easily try to attempt an SSRF attack to obtain and return internal resources via this attack using the permissions granted to the Lambda function that the attacker might not otherwise be able to access. As you may recall Capital One was subject to an SSRF attack in the breach of their cloud environment.
How strict (zero-trust) networking can help
What if you have set up strict networking and only allow the malware to access what it is supposed to access? In that case, the attacker is out of luck if it tries to access the IP address shown in the above analysis of a real-world log4j attack. The network would bock access.
The challenge is when you have broadly opened up access to a cloud service with multiple customers that the attacker also can sign up for and use. If a cloud service does not offer a mechanism for you to specify both on the sending and receiving end who can access the service in your configuration, you’re out of luck.
This is the challenge I’m currently facing as I think through my GitHub clone implementation. I can lock down my GitHub account to only allow specific IP addresses to access that account. I can lock down my Lambda function to only allow it to access GitHub through a NAT with appropriate firewall rules.
What I cannot do is tell my Lambda function that it can ONLY access MY GitHub repository and not just any repository over at GitHub. And that’s why, when customers ask, the most secure option is to host your own copy of GitHub.
What if the log4J attack could somehow host some code on GitHub that when accessed would get pulled down an executed? If I have to allow all of GitHub, how can I stop that? With networking as it stands from GitHub cloud today, there’s no way to stop that. And by the way, log4j is just an example. So don’t think, “Oh, I’m not using log4j so we’re immune to this problem.”
I just wrote about how GitHub warns against all manner of code injection that could be used to insert code into a GitHub Action.
No matter what type of code you are using any kind of injection or vulnerability might allow an attacker to do something malicious. So you can ensure with 100% certainty that there is absolutely no way for an attacker to download code or inject code and that all of your developers are completely security conscious and will never do anything risky.
😊
Alternatively, you could implement proper networking to drastically limit the number of potential attacks an attacker can perform regardless of vulnerabilities in the code in some cases (not all.) We will never have perfect security but what I like to say is, “How many chances are you giving attackers? Let’s limit that.”
The GitHub in the cloud problem
So I’m back to this GitHub problem where I can’t specify that my code can ONLY go to one GitHub repository via network rules. But at least I can limit the outbound traffic to GitHub and not anywhere on the Internet. But my Lambda functions can access anything on GitHub and attackers could potentially host rogue code and try to somehow direct my application to the wrong repository if there’s any kind of vulnerability that facilitates this in my code or AWS Infrastructure.
It would be nice if I could set up a VPN connection with GitHub to ensure that my Lambda only has access to the VPN network which only allows traffic to GitHub. With a VPN it will have a specific IP range on the GitHub site and you could restrict your network to that VPN IP range associated with your GitHub account and no one else’s. I wrote about VPNs here and this is another variation on that theme.
Or perhaps GitHub could add an AWS Private Link option, but I’m still not sure that would limit to only my network on the Internet versus just keeping the traffic on the AWS network. It depends on how they implement that feature, if and when they do.
For now, I’ll combine the best networking I can create with GitHub cloud with some other security controls for a defense in depth approach. Stay tuned for more on that. If you are setting this up for an Enterprise you might want to consider hosted GitHub servers instead. I am not sure if you can still get all the security features with that approach. Something to look into.
The other option would be to skip GitHub altogether and use AWS CodeCommit in a completely private network. Hopefully AWS is working on some of the security features offered by GitHub and will be adding them to CodeCommit soon! #awswishlist
I walk through setting up CodeCommit here:
More on networking architecture with Lambda in mind:
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2023
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
