AWS Routing Logs and Error Messages #awswishlist
ACM.463 The routing mystery that would be much easier to solve with a “No Route To Host” error message
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Check out my series on Automating Cybersecurity Metrics. The Code.
🔒 Related Stories: Network Security | AWS Security | Cloud Architecture
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the last post I showed how I created a fake entry in VPC Flow Logs and what you can do about it. I’m wondering why this is even possible for AWS services. In any case, you need to disallow actions you don’t want to occur in your account.
I know AWS is trying to make networking easier with its visuals in the console but sometimes a plain old log is a bit easier to decipher. Especially when you have something complicated like I’m working on which involves on-premises routing, a VPN, a transit gateway, VPCs, a NAT and an Internet Gateway.
I wrote about the dreaded “No Route To Host” message which totally baffled me when I saw it in logs while working for a firewall vendor and trying to show how to automate the product configuration in AWS. The error was coming from the firewall not AWS. It was actually helpful because without that error message I’d have no explanation of what the problem was at all. But I had to get someone to explain it to me.
There is actually a better message vendors can provide, as explained below, if they really want to be more helpful.
I recently wrote about resolving that error message when you put a firewall or router in front of another firewall or router in this post. You have to add the routing to get the return packets back to the correct source.
You know what would make everyone’s lives easier in networking whether it be from AWS or any vendor who provides any sort of networking service with routing?
A routing log.
It should show you the packet that got sent and blocked, and whether it was blocked on the way out or the way back due to the no route to host problem or a firewall rule. Then explain precisely which rule blocked the packet — on which Interface in the firewall or which NACL or Security Group rule on AWS, for example.
At least with NACLs and Security Group rules you can see something got rejected in VPC Flow Logs.
What happens when you have a missing route in AWS? Nothing.
You get back no indication of what the problem is or how to fix it. It’s a mysterious, very difficult to troubleshoot problem. It is a problem that tripped me up repeatedly at Capital One when I was setting up networks for all the application teams there when we initially moved to the cloud.
Why isn’t this working? I know all the rules are correct!?
Oh yeah. My route table.
I recently was playing around with something that is not possible on AWS.
Q told me it was. More on that later.
I was trying to send packets from a network load balancer to a NAT just to see if I could. I’ll explain why later. But the problem is the target looks as if it is down because the NAT rejects ping packets. And there’s no logs or indication of that being the case. I missed it the first time I read the documentation. It’s buried in a sea of words.
Error logs, please. Show me that traffic is rejected instead of making me read the fine print and things I might not even understand the first time I read it. It’s only clear after I have the problem and go back and read it again and then I know what it’s talking about. Also, please organize the documentation for all these things with a table that lists what is and is not possible between AWS services and why. But really I just want helpful error messages.
OK moving on, do I even need a load balancer? I tried directing traffic from a VPN to a NAT.
Once again, Q told me I could do this. I later found in the fine print you can’t.
Is this a routing issue? I presume so. There are no logs. It just magically, invisibly doesn’t work.
OK so I read the documentation again. What are my options?
A transit gateway. Which is pricey for a small business or home user who is just trying to inspect network traffic.
So I get this all set up but here’s the strange thing.
I’m sending packets and my firewall says they are going into the VPN. My transit gateway attachment shows that the packets are coming in and the destination is CloudFront. I am trying to hit my own website:
For me, at the moment, it resolves to the following addresses:
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*DDq3lWoqh4M4sNJuu0RyaA.png)
Well, I don’t want to mess up everything I need to hit so I created a specific VPN configuration that sends traffic for 65.8.0.0/16 to the VPN.
I can see on my local firewall that traffic is going to the VPN. When it wasn’t working in the above scenarios, in AWS logs of any kind I would see…nothing.
I tried hitting the NAT gateway IP address on 443 since I know it will accept that traffic (even though I know it won’t work) because VPC Flow Logs will only show you traffic to a network interface in your VPC, not traffic on the “bridge to nowhere.”
At that point I can see that my VPN is working and sending traffic over to the NAT for the NAT IP address in the Flow Logs for the NAT elastic network interface (ENI). The logs or error message I need should be telling me that the ICMP traffic is getting rejected so I know what’s going on instead of having to guess.
Part of the problem here is that the VPN endpoint is not in a VPC (at least not my VPC — I don’t know what AWS uses on their side) so I can’t get Flow Logs for it. The VPN logs only show me the IKE traffic, not the traffic that is passing over the VPN, which would be most helpful. I would like to see Flow Logs like traffic for my VPN endpoint and No Route To Host messages. This almost leads me to want to set up my own device in AWS so I can get VPC flow logs for my VPN traffic. I’ve done that before with a pfSense in the cloud and used to demonstrate that in my cloud security classes. But that has it’s own set of challenges.
Next, I set up Transit Gateway. Now I can see traffic flowing through the transit gateway attachments. That’s a bit helpful but doesn’t help me immediately and quickly resolve my current problem. Transit Gateway logs should show No Route To Host messages.
So here’s the other really strange thing.
I can get to microsoft.com but not a CloudFront website — How is this even working?
I was trying to hit my own website fronted by CloudFront. I set up a route for that IP range to the VPN. It’s getting to the VPN. I can see that. However, I have no indication that my traffic is ever getting to my NAT which I presume is some kind of routing issue so I’m looking around. There is no traffic in my NAT logs showing that there was any attempt to reach that 65x range above.
I find it odd that it’s reporting CloudFront specifically. So I think hey, what if I try something not hosted by AWS. Will that work?
So I set up a new target for my VPN. Microsoft.com. I send everything for 20.0.0.0/8 through the VPN which is a Microsoft CIDR and what I get back when I dig microsoft.com.
And it works. ????
I can get to microsoft.com but not my site hosted on CloudFront.
So back to the logs. How is this working???
There’s no traffic hitting my NAT for that IP range. Yet somehow I’m able to access the microsoft.com website.
I double checked that there was no leak somehow to microsoft.com through some other interface on my local firewall. I only see traffic to my interface associated with the VPN, and through the IPSEC logs (see pfSense and my upcoming posts for details). I see no evidence of the 20x IPs leaving my local firewall or the WAN interface.
So here’s the problem. I can’t tell how that traffic is getting out to the Internet. It shows up in the transit gateway attachment logs but nowhere else. I have no idea how that is working. I would like to know.
More logs please.
Oh wait. The next morning the saga continues. Now it’s not working.
So I delete my VPN on the client side and re-create it the next day with the same routes. Now I can’t get to microsoft.com either.
What was going on with that? Why was it working and why is it not working now?
Inquiring minds want to know.
Recommending rules and routes
It also seems to me that if there is no route to host it would be pretty simple to recommend possible routes to fix the problem. I don’t want a complicated graphical GUI. I just want it to say in the logs where I’m looking at the traffic:
There is no route for x.x.x.x to reach x.x.x.x in available route table(s) #id(s).
Here are the potential routes you can add to fix this problem:
- vpc route table #id: route and transit gateway rt table #id: route
or
- vpc2 route table #id: route and transit gateway rt table #id: route
etc.
You could do the same thing with NACL and security group rules. When I perform a penetration test or a security assessment I basically do the opposite. I evaluate all the routes and propose a way to shore up the network by closing certain paths. I could easily add to this looking for ways to connect the dots between two network interfaces. Of course you’d also have to know all the tricky AWS rules I mentioned above which have nothing to do with normal networking.
Maybe AWS can use some of that there new fancy AI they are promoting to sort it out. 😊 But really that problem doesn’t require AI. That’s doable with basic logic. I’d do it if I had time.
Keep it simple: Tell me what the problem is and exactly how to fix it clearly, but in as few words as possible.
The problem with the visuals in the console
The problem with the visuals in the console is that they are only a partial picture. You can’t see all the connections, only a fraction of it when looking at a single VPC. If you try to include every single connection in a complex environment that might become overwhelming. The specific error and fix in a log is easier to parse.
That said, you know what would be super amazing and cool? A visual that allows me to trace a single packet or request to a specific IP address from the entry point at the VPN and watch it flow through all the routes and devices until the point where it got blocked with a big red X. How amazing would that be? Like your Uber driver coming your way only they hit a traffic jam and have to take an alternate route, but then they get blocked somewhere. You can see that all as the little car is coming your way in the app. Dare to dream.
But all I really need are basic error messages. I could have gotten it done in a fraction of the time if the error messages simply told me what the problem was and where the traffic stops and why.
And by the way, these problems apply to all networking products and services, not just AWS. Sometimes I think about getting funding to build a new firewall of some kind to make all this easier or more secure but it’s not really what I want to spend my time on. I wish someone else would do it. Like when I wished someone else would run my source control online which is what GitHub does and automate my server management which is what AWS does. So …who knows. It could happen.
I’ll hopefully be publishing what I’m working on soon.
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2024
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab
Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation
Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/0*H9Ew1KCl-29nZiPR.jpeg)