Why You Need CloudTrail Data Events for AWS Lambda Functions

ACM.310 How can you see which IP addresses invoked your Lambda function?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: Lambda | Container Security | Application Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post, I showed you how to reduce the noisy traffic hitting your Internet-facing hosts. If you haven’t looked there’s a lot of noisy traffic from rogue networks bouncing off them using two high ports. That’s not easy to block on AWS the way I showed you how to do that on a pfSense so the last post demonstrates an alternative.

In this post, I’m going to show you how to invoke Lambda functions without using the AWS console and instead using the AWS CLI. Up until now, we have been able to invoke our Lambda functions using the AWS console even though we put our function inside a VPC. If the function is deployed in a VPC, should it not only be accessible from within the VPC? Is that just because we’re using the AWS console? Let’s do some additional testing and investigate.

Calling a Lambda function with the AWS CLI

Another way we can trigger an AWS function is using the AWS CLI. Lambda has an invoke action that we can use to trigger the function call.

The only required properties are the function name and outfile.

Recall that I am running an EC2 instance in a different VPC than the one where I deployed my function. Recall that the function security group allows outbound access to GitHub IPs only. The requests route through the NAT as we tested two posts ago. Can I invoke this Lambda function from a separate VPC if the Lambda is deployed in a private VPC?

Let’s test it. Recall that dockertest is the function I’ve been working with over the last few posts.

aws lambda invoke --function-name dockertest --profile SandboxAdmin dockertest.txt

What’s in the dockerfile.txt output file?

cat dockertest.txt

The file contains the output of our function call. It worked. Why did it work if my EC2 instance is not in the same VPC as my function?

Looking for the IP address that called the function

Recall how I showed you how to look up the ENI ID and the IP address of the Lambda function network interface in a prior post.

Head over to the CloudWatch Log groups like I showed you and take a look at the logs for that ENI. Wait for the logs to display the most recent function call.

Notice what you do not see. You do not see the IP address that invoked the function call in these logs.

Navigate to the function logs in the Lamba console the way I showed you.

Click the most recent log link.

No IP address here either.

Where can we see the IP address that invoked the function?

Let’s check CloudTrail.

Looks like we can’t filter on the invoke event.

I tried searching via UserName but not seeing it.

ReadOnly = False doesn’t show the invocation because I can see all the events since I logged in.

Why you need CloudTrail Data Events

We are not seeing the invocation of the Lambda function because we have not enabled logging of data events in CloudTrail. From the documentation:

CloudTrail also logs data events. You can turn on data event logging so that you log an event every time Lambda functions are invoked. This helps you understand what identities are invoking the functions and the frequency of their invocations.

How much does logging cost for data events? That depends on how many events you log.

Well, to fix my CloudTrail logs to log data events I’d have to fix my organizational trail which I might do later. For now I’m just going to manually create a Lambda trail in my Sandbox account.

On the first tab, I give the trail and new KMS key name. I am using a new KMS key because my Sandbox key doesn’t grant permission to CloudTrail. I could fix that later if I wanted to use a single key to save money, or segregate the keys so that access to one key doesn’t grant access to everything encrypted in the account.

Scrolling down, I enable CloudWatch. Also enter a CloudWatch role name.

Note that this incurs CloudWatch costs.

I’m going to choose data events for Lambda.

Create the trail.

Head over to CloudWatch. Navigate to Log Groups.

Open the Log Group you created.

If you want to monitor for events without repeatedly hitting the refresh button you can click on Start Trailing.

What I noticed here is that it took a very long time for the logs to show up. That’s because the logs will dump after a certain time period or amount of data. You can read up on that in the CloudTrail documentation. But once the data events did show up, an IP is present in the logs. And although the Lambda function is in a VPC, an IP address from outside the VPC can invoke the Lambda function.

This should get you thinking about two things.

1. If you have a data breach, how are you going to know what IP address invoked a Lambda function if you haven’t turned on data events? The same is true for other services like S3 as I mentioned in prior posts in relation to the Capital One breach. Although the breach was not great, it could have been worse if they didn’t have data events turned on. The scope would have been bigger because they wouldn’t be able to prove which buckets, specifically, the attacker was able to access.
2. Our “private” Lambda function is not really private. Why not? Because it can be accessed via a public API that is accessible to anyone on the Internet who can reach AWS.

Recall how AWS services work. You make an API call to AWS and from there, the API call does something on the AWS platform. As long as you provide the right credentials and can reach the API via the Internet, it works.

Recall the photo I presented when using the AWS console. The request first goes to the AWS Console. Then from there a request is made to the Lambda function in AWS. You connect to the console over the Internet to make that call. Then the console, hosted on the AWS network by AWS, makes the call to invoke the function when you push a button.

When I look at the data events, I get data events from the AWS console showing my local home laptop IP address. I get data events from the CLI invoke action showing the IP address of my EC2 instance in the cloud.

What if we only want to allow access to our Lambda function from a private network? I’ll cover that in the next post.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab