Sending Traffic From pfSense to a Site to Site VPN to a VGW to a NAT in AWS
ACM.470 Another thing Q told me I could do but not allowed
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Check out my series on Automating Cybersecurity Metrics. The Code.
🔒 Related Stories IAM | AWS Security | pfSense | Network Security
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the last post, upon Q’s blessing, I attempted to send traffic from a Load Balancer to a NAT. It didn’t work.
In this post, I again consulted Q regarding something I wanted to try: Sending data from a Site to Site VPN with a Virtual Private Gateway straight to a NAT.
While struggling with the network load balancer in the last post, I started to wonder what would happen if I just sent my traffic through the VPN to an Internet location. Would it just go through the VPN to the Internet via the NAT? Or do I still need something in the VPC to forward the traffic?
I should have known better having had transitive routing beat into my brain when I previously worked on proxies and NATs at Capital One — but that was pre-AWS NAT Gateway. Maybe something changed.
After a few tries, I got the question right (or so I thought) and got an answer that looked reasonable. According to Q I can do this.
First of all I checked to see if my AWS VPC Flow Logs that I set up in a prior post are working so I can monitor the traffic and they are. And there sure is a lot of noise coming from the Internet.
What I really wish is that AWS would let you create a rule to block traffic to two high ports as one rule could eliminate pretty much all of this, but that’s a tangential thought.
Verify the VPN is UP
Head over to AWS Site to Site VPN on the VPC Dashboard and check your VPN (created in a prior post).
The status is Up.
Head over to pfSense and choose Status > IPsec.
It shows that the connection is Established.
To see if I can reach the NAT, which for the moment is on 10.101.101.61, I simply made a request to that address in my web browser.
I can see logs for that request in my IPSec interface and they make their way to AWS successfully.
I can also see the traffic in AWS hitting the NAT in the VPC Flow Logs, but as you can see it is rejected.
I added a Phase 2 entry for all traffic.
I explained what those are in this post and how to get those change to kick in by resetting the connection:
Now I can see all my DNS traffic in the IPsec logs, so my routing rules are working properly as far as pfSense is concerned.
OK cool. But it’s not working. I can’t get to any websites via the NAT from my laptop.
I head over to AWS and I can’t find any traffic hitting the NAT in VPC Flow logs. So what is happening here? The traffic is allowed through the firewall on the pfSense side. I can’t see that it is getting to the NAT in AWS. Where is it going?
For testing purposes I set up a Phase 2 entry for the 65.0.0.0/8 range only for my website 2ndsightlab.com — because it currently resolves to that range. That way I can test that one website and everything else will still work. I can’t get to that website. But it’s trying to get there because the DNS request on my local network works.
So let’s think about this.
The traffic works for DNS. It makes a request and gets back the 65.x ip address.
Then it tries to send that address through the VPN.
On the local side it has an entry to send 65.0.0.0/8 through the VPN. I can see the traffic in the IPSEC logs with an address in that range and the traffic is accepted.
So I presume what happens next is that somewhere in AWS the traffic gets rejected, but I can’t see that anywhere in any logs.
- The VPN logs do not show any traffic — only IKE logs indicating the set up of the VPN connection between the two endpoints.
- I presume it is getting through the VPN into the VPC where the NAT exists. But the NAT never sees the traffic.
- I also have no logs for the Internet Gateway specifically so I can’t see if it is hitting the Internet Gateway or not.
If you go check out your network interfaces and your VPC Flow Logs you will see that there’s an elastic network interface (ENI) when you create a NAT and when you create a Transit Gateway, and any EC2 instance or Lambda function. What you won’t find here is an ENI for an Internet Gateway. You also will not find any logs specifically showing traffic hitting your Internet Gateway or a Virtual Private Gateway.
You will only get logs for ENIs in your VPC where you have configured VPC Flow Logs. That means, if traffic is going straight from the Virtual Private Gateway to the Internet Gateway, there’s absolutely no way to tell. Apparently you just have to keep digging until you finally find the nugget in the documentation that tells you what the problem is…
Well, here’s the problem which I found after probably digging for quite some time. This did not show up easily in an Internet search and I never search directly on the AWS site as Google searches work better to be honest.
You can’t route traffic through a NAT Gateway when traffic arrives over a hybrid connection (Site to Site VPN or Direct Connect) through a Virtual Private Gateway.
The answer Q should have given me.
And a solution that will work.
You can route traffic through a NAT Gateway when traffic arrives over a hybrid connection (Site to Site VPN or Direct Connect) through a transit gateway.
So I have to set up a transit gateway.
This, at the top, is misleading:
Alternatively, you can use a public NAT gateway to connect to other VPCs or your on-premises network.
Should that say…if you use a Transit Gateway?
The other thing I realized later is that there’s a routing problem. If you put the NAT in the private subnet with a route to the IGW for all Internet traffic, the traffic will try to go to the Internet Gateway instead of the NAT and get dropped due to transitive routing. So you need to set up your subnets like this:
Private subnet
- Route: 0.0.0.0/0 > NAT
Public subnet
- NAT
- IGW
- Route: 0.0.0.0/0 > IGW
So if someone wants to try this again and verify their routes are set up that way and see if it works feel free. The documentation could definitely be clearer with a chart of what is and is not possible. In addition, VGWs, VPNs, and IGWs need love (I mean logs) too. Or at least some sort of log somewhere showing the mystery dropped traffic or denied routes.
I’m done here and moving on to the Transit Gateway solution for now.
Which works, by the way.
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2024
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
