S3 Access and Logging Pop Quiz

ACM.348 Can a user with s3:* read the objects in a bucket with the default policy? Where are logs when a user has cross account S3 access?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: AWS Security | S3 | Application Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post, I altered my generic application policy to pass in application specific actions and resources. I want to clone files from an AWS CodeCommit repository and copy them to an S3 bucket in a remote account.

I left off the part about copying to the remote S3 account because that requires some additional configuration.

Let’s think about IAM Roles and S3 bucket policies for a minute.

For this test I’m going to got create the following:

• An S3 bucket named bucket-mc-bucket-face with all the default bucket settings and upload a file to the bucket.
• I’m going to create a user account in the same account named s3localuser.
• I’m going to create a user in a remote account named s3remoteuser.

Can a user in the same account as an S3 bucket see the files in a bucket with a default bucket policy if you grant them s3:* access?

Attach the S3 full access policy to the local user.

Login as s3localuser. Can the user see the S3 bucket? yes.

By default, any user in the same account with S3 permissions can see the bucket with a default bucket policy.

What if you only specify a single bucket in the resources for that user?

Let’s say I grant a user explicit access to the bucket in the IAM policy.

Well, in order to list the bucket in the AWS console you have to allow the user use s3:ListAllMyBuckets. Otherwise when they open the console they will get an error on the screen above even if they have permission to get or push files to an individual bucket, they will not be able to access the buckets from the console.

Add the ability to list all the buckets to solve that problem.

Even as I was writing this I forgot the names of certain actions and had to go back and review it. There are separate actions for listing buckets (ListAllMyBuckets) and viewing the objects or files in a bucket (ListBucket).

Now I can see the bucket but some permissions for all aspects of S3 are missing.

I also cannot view the objects in the bucket even with GetObject permission for objects in the bucket.

We also need to add the ListBucket permission.

But that’s not exactly right either. See below.

What do we need to add besides an IAM policy to limit which buckets and resources a user in our AWS can see?

To restrict a user in a local account from accessing a bucket, you need to create a bucket policy that limits who can access the bucket. An S3 bucket is a resource policy. I wrote about the difference between IAM policies and resource policies here:

You can define exactly who can access a bucket in a bucket policy and those restrictions override anything granted to the user in the IAM policy.

Can you prevent the root user of an account from accessing a bucket?

Warning: It is possible to lock out everyone in an AWS account from accessing the content of buckets or modifying the policy in some cases. It depends who the owner of the account is and if you use something like allow everyone but this user with a DenyAll in your policy.

Make sure you understand who owns the buckets, which will vary based on when the bucket was created.

The post below suggests that if you get into this scenario, the root user of the account can delete the policy. However, recall that we limited the actions of the root user in an organization in an earlier post. You may need to alter your SCP to restore root user permissions to allow the user to perform this action.

Obviously logging in as the root user is not something you want to do in the first place, so be careful not to lock everyone out of a bucket when altering bucket policies.

Can you grant a user the ability to view buckets in another account in the AWS Console?

No. A user can only access resources in the specific account where it exists. However, you can allow a user to assume a cross-account role to access resources in another account in the AWS console. I wrote about cross-account IAM roles here for an AWS Organization and how to use them:

Note that you can also switch accounts using AWS SSO (IAM Identity Center). I just noticed that in one of my accounts, the screen changed when creating a user suggesting you can create one in IAM Identity Center or with IAM. I wrote some articles about why I still prefer IAM in these posts and how to fix the problems with AWS SSO if you choose to use it:

Some of the things I’m showing you how to do in the posts I’m writing were not even possible in AWS SSO last time I tried it. It doesn’t work for my use cases.

I also explored using Okta, which I will explore further in the future.

Note that Okta support had yet another data breach of their support team recently. The problem had to do with screen captures that the support team asked for and customers shared with the support person. Any time support teams ask me to do a screen share I say no for a number of reasons — not just the issue in this breach. I feel that support teams should not be using that method and especially for an identity product as there are too many attack points.

Also, the solution I presented in my Okta stories separates the people who create the users from the people who grant the permissions. I also use separation of duties within Okta itself. If you do that access to an Okta session token alone will be of limited use to an attacker.

Those are some of the ways you can grant a user access to the console to view S3 buckets in multiple accounts.

If you grant a user in a remote account access to that bucket via s3:* permissions and add the bucket to the resources in their policy can they get the contents of that bucket using an AWS Access Key?

Here’s what I mean to be clear. I created the s3remoteuser in a different AWS account than the one where the bucket exists with no console access since that won’t help us access the bucket as just explained.

I gave that user s3:GetObject permissions on the bucket-mc-bucket-face S3 bucket.

I create an access key and secret key for the user.

Note that I always require MFA with long-lived credentials when handling sensitive information or actions, but I am going to delete these credentials before this blog post gets published and this user only has access to a bucket with an image in one of my blog posts currently once I have this working.

Create an AWS Profile.

Next I am going to use the s3 cp command. Initially I was going to use the GetObject command but I think s3 sync will be a bit simpler and more efficient. Sync should only copy over the files which are different so less gets and puts.

First I try to run this command:

aws s3 sync s3://bucket-mc-bucket-face . --profile s3remoteuser

Note that s3 bucket URIs start with s3:// followed by the bucket name.

Remember that I am running this command from a private network with limited outbound network access.

I get this error but it takes a very long time to execute.

fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Let’s add that action to our policy. Note that his command falls under the s3api. Why there are two separate service name categories I don’t know. Perhaps to make policies easier to create by giving people specific s3 actions vs s3api actions.

When I look in the AWS console, I only see the option for s3, not s3api for services and I do not see this list-objects-v2 command as an option.

You also won’t find it on the S3 IAM actions page.

What’s going on here? Well, the AWS IAM documentation and UI does not align with what’s going on here. Under the hood, S3 bucket calls ListObjectsV2. Seems like the error message needs to be fixed in this case.

Note: s3:ListBucket is the name of the permission that allows a user to list the objects in a bucket. ListObjectsV2 is the name of the API call that lists the objects in a bucket.

I’m going to manually add this action and here’s why. I have to use a slightly different URI.

I’m still getting the error.

But I notice that the command takes far too long to return a response. So here’s my next question.

What happens when you try to access an S3 bucket in another account when you have a VPC Endpoint in the subnet where the host trying to reach the remote bucket exists?

Recall that the EC2 instance on which I’m running my commands is in a subnet that has both an Internet Gateway and VPC Endpoints. I added an S3 VPC Endpoint to my configuration.

Let’s take a look at the logs for my EC2 instance. It’s trying to reach this remote bucket. Where is it sending the requests?

I actually don’t see any traffic at all related to these requests on private IP addresses.

I do see requests to public IP addresses when I reverse the DNS for one of the IPs.

Cross-account requests do not appear to remain on the private network without additional configuration. Or so I thought…

You could try using a gateway endpoint for S3 or other types of private networking but I’m not going to get into all that here. Gateway endpoints use an entry in your route table instead of a network interface in your subnet so in theory everything would be forced to the correct endpoint.

Later, when I successfully got the command working, I only saw traffic traversing private IP addresses. Did the S3 endpoint just take a very long time to deploy or was something broken and then fixed? I don’t know.

The other weird thing is that I was getting network errors in the console when trying to update the bucket policy with a user that didn’t have permissions to do that. I switched to the root user (I know bad) and then I was able to update the policy and didn’t get the network errors anymore. Not sure if that was a temporary glitch or really is related to a user who didn’t have appropriate permission.

What do we need to add besides an IAM policy to let a remote user access an S3 bucket in an account?

To allow a remote user to access a bucket in another account, we need to modify the remote bucket policy to allow that user to access the bucket.

I still get an error on ListBucket (ListObjectsV2).

Where can I view the error in the logs for cross account bucket access — in the bucket account or the remote account?

Remember to turn on the error column in your CloudTrail event logs to make your life easier.

First let’s check the account with the S3 bucket for errors.

This error is a bit strange. The public block access policy is not found? And that appears to be an internal AWS user agent. I get this error for the user logged into the console. Weird, but not what I’m looking for. (And I hope my public access block policy is working.)

I don’t see any events in this log for the remote user. Let’s head over to CloudTrail logs in the account where the remote user exists.

Well, not seeing the error there either.

Hmm.

Where are the failed access logs?

What caused the ListObjectsV2 to fail?

I’ll give you a hint. I already showed you above. The tiniest little syntax error causes this problem. And here’s what makes it harder to solve:

• The S3 Bucket UI tells you something is wrong with the policy but not exactly what. At least it doesn’t allow you to create a policy that doesn’t work.
• The AWS IAM console will let you create an invalid policy.
• The AWS CLI gives an inaccurate error message about ListObjectsV2 when it should say s3:ListBucket (which allows you to list the objects in the bucket using the ListObjectsV2 API).
• There are no additional error logs in AWS CloudTrail as far as I can tell.

Here’s the problem.

If you look at the policy above, I had a slash after the bucket name.

To make things even more inconsistent, you have to include the slash at the end of the bucket when you use the aws sync command:

What can I see when I change the policy for the local user as I just did for the remote user?

Now the user can see the contents of the bucket.

What if I change ListAllMyBuckets to only allow a specific resource?

Recall that we had to add ListAllMyBuckets for the local S3 user to view the buckets in the console. What if I only allow access to a specific bucket like this?

The user cannot see any buckets.

So perhaps the ARN is wrong and we need to remove the slash and asterisk?

No. Still doesn’t work.

What if we add a slash?

Nope.

You have to grant a user access to see all your buckets in the console. That is unfortunate.

What if you don’t want the user to see some buckets?

You can segregate them to different accounts in your organization.

Will you see CloudTrail Logs for a successful action?

Well, I can’t find them. I looked in the current region and us-east-1. I searched for the access key id. I searched for actions on S3 object resources. I am not seeing any record of this event.

The S3 GetObject logs appear in the S3 Access logs. This is something additional you have to enable.

I wrote about why you need those type of logs in a post on CloudTrail Data Events for AWS Lambda Functions which serve a similar purpose.

Here’s how you can set up an S3 Access Log policy:

In the console:

You can also enable S3 object access events in CloudTrail.

Under your bucket properties:

That takes you to CloudTrail. Click on the trail you want to modify.

Note if you have an organizational CloudTrail you’ll need to modify this is the account where you created the organizational trail.

More on S3 Logging options:

Why would you use a cross-account role where a user assumes a role to access the bucket versus the direct access I show above?

You can’t view the remote console with the method I just demonstrated with a remote user. If you want the user to be able to login and use the console to view the remote account in the AWS Console, you’ll need to use one of the other options I mentioned above.

I’ve seen someone write in an AWS resource that it is easier to manage cross-account roles. I’m not sure if that is exactly true. It will depend on your use case.

Remember that any local users with full S3 access can read any bucket contents that are not encrypted with a key they do not have permission to use or do not have a bucket policy limiting who can access the contents.

Remember that for console access you need to give full access to view all the bucket names in the account.

Does this seem complicated and hard to remember all the details?

It is.

I’ve been writing S3 bucket policies for over 10 years and I still can’t keep it all straight. I had to look things up while writing this post.

That’s why I recommend using CloudFormation micro-templates — a term I made up and the methodology I have been using throughout this series:

And figure out how to create a single template or limited number of correctly configured S3 bucket templates that work for your organization. I started providing an S3 bucket template in this post which I will be building on for my Lambda function that copies files to an S3 bucket.

When people can’t figure out how to configure access properly — that’s when they get frustrated and start opening everything publicly because they can’t figure out how to do it correctly. Make it easier. Use common templates for S3 bucket deployments, but make sure they work for all your use cases or people will push back against using them.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab