AWS Service Policy Failing to Properly Assess VPC Endpoint Requests

An AWS Service Control Policy That Worked Before Is Breaking — let’s find out why

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: Bugs | AWS Security | Secure Code

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In a prior post I wrote about how the SCP that evaluates regions broke. This SCP is in the AWS documentation.

Today I am testing a Lambda function that worked before and is now failing. Here’s the error I’m getting from CloudFormation when I try to deploy a Lambda function:

“Resource handler returned message: \”User: arn:aws:iam::464339214996:user/SandboxAdmin is not authorized to perform: lambda:GetFunction on resource: arn:aws:lambda:xxxxxx:xxxxxxxxxxxx:function:codecommittos3Lambda with an explicit deny in a service control policy (Service: Lambda, Status Code: 403, Request ID: 60e239da-6bef-4ab8-bd87–959278ba520a)\” (RequestToken: c80b1e8f-1403–6219-eb7d-d5ce6a07eed9, HandlerErrorCode: AccessDenied”

This SCP worked before when I used it to block requests that did not come specifically from my VPC Endpoint in the VPC and subnet where my EC2 instance exists.

I demonstrated in a prior post that with this service policy in place I can only invoke Lambda functions from the authorized private network:

The problem is that now I’m deploying a Lambda function and it’s telling me my user, SandboxAdmin, is blocked with an explicit deny from calling the GetFunction action in an SCP.

That’s not accurate. That user is not blocked. What is blocked, is whatever IP address is trying to make this call, so no matter whether this is right or wrong (I think it’s wrong) the error message needs to be corrected to accurately state the problem.

But let’s take a look at what is going on. I checked my VPC flow logs where I’m running a script to deploy a CloudFormation template. I don’t see anything blocked in the logs.

Let’s take a look at CloudTrail.

What is the source IP address in the request that is failing?

This is not helpful nor is it accurate. This should list the IP address of the user that is making the CloudFormation request. I actually wrote about a problem like this on Azure where my request was blocked but it reported an IP that was not my own. Now it’s happening on AWS. So sad to see that.

The request is ultimately triggered by a principal at a particular IP address and should log that address not CloudFormation. That CloudFormation address is used by any number of customers not only my account and private network.

I hope that AWS will address and fix this problem so I can properly limit requests by IP address and VPCEs for private networks when using CloudFormation. This is incredibly important given that CloudFormation can deploy — anything — in an account and a customer cannot put restrictions on CloudFormation, only on their own users and people making requests from the networks they control.

I am also not ok with that domain name. One of the major secruity problems on AWS has been an issue with DNS rebinding. I want to see the actual IP address at AWS that made the request. Perhaps there should a principal IP address and a service IP address, and then a separate domain name. Show me the IP so I kow it actually was the correct IP within the AWS network.

On top of all the above problems, I cannot put “cloudforamtion.amazonaws.com” in my Service Control Policy for the IP address. The console doesn’t allow this. This is all around not a good idea for logging IP addresses by CloudFormation.

This also does not work even though it’s listed as a global condition:

I see invokedBy in the policy but that fails saying it is not a global condition.

User agent is NOT safe to use. Anyone can easily change their user agent.

I don’t want to bypass this based on user name. The whole point of this is to use network security.

Other people are having this same problem:

AWS Please fix this soon. #awswishlist.

I’m calling this one a security *BUG*.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab