Cannot Restrict Lambda Actions Based on Private IP Addresses in a Service Control Policy

ACM.313 IPAddress and NotIPAddress conditions don’t work for Lambda on Private IP addresses in an AWS SCP

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: Lambda | Container Security | Application Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post I explained how to add network restrictions to Lambda functions within an AWS Organizational Unit (OU) to a specific public IP address.

In this post, I’ll finally show you how to restrict access to invoke Lambda functions in an account to a private network. Or so I thought. I’m actually going to show you how to restrict access to public IPs and that private IPs don’t work in policies like this. I’ll then show you how to address that in the next post.

Recall that I have a separate VPC where I’m running an EC2 instance. I have my Lambda functions in a private VPC. To allow my EC2 instance to access the Lambda service with a private VPC I need to add a VPC endpoint for Lambda to my EC2 VPC. I’m just going to do this manually for this test.

Try to invoke the lambda function again using the invoke.sh script from prior posts.

With the restrictions in place to the public IP address of the EC2 instance added in the last post the invocation fails. Why?

Network Endpoints send traffic over private routes

Once you added that all the traffic from that VPC takes the private route. In other words, after attaching the Lambda endpoint to my VPC, traffic to the Lambda service from my EC2 instance goes to the private IP addresses for Lambda from the private IP address of my EC2 instance.

You can verify that as follows.

Navigate to the VPC dashboard.

Click on Endpoints.

Hover over Network interfaces for your endpoint.

Navigate to CloudWatch and the flow logs for the developer VPC where you added the VPC endpoint for Lambda. Click on the endpoint logs for the network endpoint to view the traffic.

Look for the connection from the private IP address of your EC2 instance.

Try to find the same traffic in the logs for your EC2 instance based on the IP address for the network endpoint from the above logs:

You’ll find the corresponding traffic.

You can also see in the CloudTrail logs we set up for Lambda Data Events in a prior post that the IP address related to the Lambda Invoke function is now the private IP address of the EC2 instance.

Now I thought I could just head back over to my Service Control Policy and limit access to invoke Lambda functions in the Sandbox Account to the VPC CIDR range of the VPC where I deployed my EC2 instance. I added my local IP address as well so I can access the functions from the AWS console.

Now here’s where it gets interesting. I can now access the Lambda function from the console from my local IP which happens to be a public IP address.

But I am still getting errors when trying to invoke the function from my EC2 instance after adding the CIDR block for the VPC via the private IP range. Why?

Peering VPCs for private access between two AWS VPCs

I thought perhaps I needed to add VPC Peering for this to work.

I’m going to do that manually for the moment.

Head over the VPC dashboard. Click Peering connections on the left.

Click Create peering connection.

Give your peering connection a name.

Select the two VPCs you want to peer.

Click Create peering connection.

Get the peering connection ID from the details of the peering connection.

At the top of the screen select Actions > Accept request. Otherwise you might see a black hole when you try to add the route to your route tables.

Head over to the private route table for the NAT subnet.

Add a new route.

Enter the CIDR for the VPC to which this VPC is peering and select Peering connection for Target.

Select your peering connection ID. Click Save changes.

If you see a Black Hole here, check the step above related to accepting the peering request.

Repeat for the other VPC to allow traffic destined for the specified CIDR to route through the peering connection.

Even after peering the two VPCs the traffic gets denied by the SCP.

I tried adding all traffic on the Internet to the policy (0.0.0.0/0) and I still got the explicit deny in the policy.

Apparently this is not going to work.

SCP and notIPAddress or IPAddress condition doesn’t work with a private IP address

OK so I deleted the peering connection. I decided to change the logic to block IP addresses I specify instead of using the notIPAddress condition. By process of elimination I could figure out what IP address was causing my request to invoke the function to be blocked.

Once on Microsoft Azure, I had a request from my IP get blocked when a policy explicitly allowed my IP address. The blocked IP reported back to me was an internal Microsoft IP address. Somewhere along the way, there was a problem where a security check was not looking at the actual source IP or worse — something internal was performing a MITM attack. I wrote about those here.

Maybe something like that was going on and an IP other than my private IP on my EC2 instance was getting blocked somewhere along the way.

Here’s what I can do. I can deny IP addresses that match anything that is not the the IP ranges I want to allow using the methods in this post:

So here’s what I did. I methodically started adding IP ranges until I figured out what the range was where the blocked request came from. Or actually I didn’t but stay with me.

I started with this range:

I denied anything in those ranges:

My function call worked.

Next I tried this range:

Next I denied this range:

Then I added this range:

32.0.0.0/3

Then I added this range:

16.0.0.0/4

My function continued to work.

I ended up with these ranges, which should cover all IP addresses:

I can still invoke the function via the AWS CLI invoke command.

However, I cannot use the AWS console again now because my home IP is blocked.

Although I cannot find this documented anywhere, the IP condition does not seem to work with private IP addresses.

If you reject traffic unless it matches a particular IP address, the IP address can never access the function because the IP conditions never match a private IP address.

However, at least we can block all public IP addresses except the ones we want to allow. For that I can use my method again to calculate all the ranges before and after the IP address I want to allow as I did in the prior post above and block those ranges.

Let’s say the address you want to allow is 3.3.3.3.

Calculate all the CIDRs up to 3.3.3.2.

Calculate all the CIDRS from 3.3.3.4 to 255.255.255.255

Block all those ranges in the policy above and you will effectively only be allowing 3.3.3.4.

You can’t just disallow everything except 3.3.3.4 as I’ve shown above as that is blocking the private IP when the EC2 instance invokes the function.

Unfortunately, this a quite unwieldy, hacky work around to this problem. Secondly it does not help when trying to limit local traffic to a specific IP range.

After writing this post and searching around, I found a potential issue. I remembered that in an S3 bucket you have to use a VPCE restriction in the policy. But does it work with Lambda? I’ll test that in the next post.

There is one other thing I thought I could try — grabbing the IP address from within the Lambda function and using an allow list there. That’s not ideal. I also haven’t figured out how to do that with a custom runtime yet. I requested an update to the AWS documentation for that on the bash tutorial.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab