Troubleshooting Software Installs on AWS Private Networks

ACM.351 External and AWS repositories both slowing me down today

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: AWS Security | Network Security | S3

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I was having some issues deploying software and getting blocked from what I am trying to do today on AWS so just jotting it down in case you are also facing these issues.

Although I’ve already resolved networking issues related to installing AWS packages with pip, I’m getting networking issues again caused by trying to install git-remote-codecommit with pip:

So is this the same location where this was hosted before when I had this issue, has it moved, or is it just resolving to a different IP address? Not sure. Let’s see what the IP address is for that domain.

This further demonstrates the problem with hosting and obtaining packages from content delivery networks, or CDNs. The IPs are always changing and it makes network security very complicated. In fact, it’s nearly impossible to create decent firewall rules for these ever changing domains. That means you need a way to get the files into some kind of test environment and try to verify they are correct before pushing them into your development environment because you can’t trust the source. It’s not the company that made the software. Even if it is (AWS CloudFront) you might be using a CDN that is used by many other companies.

I tried to update my NACL rules but I hit the rules quota, so I had to make some ranges more broad than I would like. That’s another problem with this scenario. You have to keep opening and opening your network up until you have what I refer to as the Swiss cheese network. (Lots of holes.)

AWS should move this package above to some kind of local repository or get it set up for install through Yum. #awswishlist

Once I fixed that my Dockerfile built with the AWS git-remote-codecommit tool successfully.

Next I had issues installing Yum packages. That’s odd. This was working last night. But I had opened up some IP ranges I didn’t really want to open up at some point to get this going. Why wasn’t this all going over a private endpoint to S3?

I went to review my AWS S3 endpoint settings. I did have an issue. My VPC endpoint for AWS S3 had the wrong security group. That’s odd. I think this was working previously. Hmm.

To fix my next S3 endpoint issue, I inadvertently deleted the endpoint instead of just fixing the security group issue. When I redeployed the endpoint I was still getting public IP addresses for the particular domain name that was failing in my docker build:

cdn.amazonlinux.com

After deploying the endpoint I realized there’s a small box telling you the endpoint is changing and the information might not be correct. This box could be bolder and clearer — like in yellow and tell you that traffic will not be routed to this endpoint until the deployment is complete and this box goes away.

Once the deployment completed, I checked the IP addresses for the domain above.

Those are public IP addresses. That’s not helpful.

Then I realized I forgot to change the DNS setting to use private DNS. I fixed that and then I got different IP addresses, but they were still public:

But when I deployed my container, it worked. I didn’t dig into the details as to why. Did it actually use private IPs? A different domain? Or happened to resolve to IPs I had in my NACL? My NACL is already maxed out. I can’t easily add more so I wish AWS would get this traffic onto a private IP if it’s not going there already after my fix. Not going to research that now.

Update: Must have been random. Getting this again:

Ends with this:

And… now it works again.

Could be the domains randomly resolve to IPs I’ve allowed and then sometimes IPs I don’t. Can look into it more later but wish AWS would make it easier to create firewall rules for this. I would think the IPs should resolve to the private IP range with VPC endpoints? No?

The other thing I noticed is that there are separate install domains for Amazon Linux 2 and other versions. This makes networking even more complicated if you are using different operating systems in your environment.

This is what is slowing me down at the moment so just writing about it. Hopefully I’ll have the container done for my Lambda function shortly.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab