Permission denied on S3 path: s3://aws-controltower-logs-xxxxxxxx.json.gz. This query ran against the “default” database, unless qualified by the query.

AWS Control Tower with CloudTrail Logs and Athena Not Working

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: Bugs | AWS Security | Secure Code

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I wrote about problems trying to use AWS Athena with AWS CloudTrail when you have set up your account with AWS Control Tower once before. This is my second attempt to get this working.

Here’s the real problem. When you try to use Athena with CloudTrail it should work by default if you have access to view the logs anyway or there should be some convenient UI functionality to make this work. Instead I’m getting errors with no clear resolution as to how to fix the problem from the documentation I’m finding. I’m making hacker-like attempts to resolve the problem but so far I have not.

Basically when I try to run a query using AWS Athena I get this error:

Permission denied on S3 path: s3://aws-controltower-logs-xxxxxxxx/xxxxxxxxx/AWSLogs/xxxxxxxxx/CloudTrail/us-east-1/2022/02/03/xxxxxxxxxx_CloudTrail_us-east-xxxxx.json.gz. This query ran against the “default” database, unless qualified by the query. Please post the error message on our forum  or contact customer support  with Query Id: xxxxxxx

About the end of that error message: 1. ) I don’t want to pay for support where there is no documentation and there seems to be a bug here. 2.) I don’t want to post private account information in forums.

Let’s see if I can figure this out myself.

Why can’t I access the S3 bucket ? Is it because I’m using AWS Control Tower?

Back to the documentation in search of an answer. At first I looked at the information for Control Tower but then I landed here:

It looks like the CloudTrail bucket by default has limited permissions. I don’t recall when I taught how to use Athena in my cloud security class before if I had to manually go create permissions to allow Athena to access the bucket. I would have to go back and look since my brain has been on a million other things including an Azure deep dive class (because someone asked). But here’s the thing:

Why can’t the UI just tell me I don’t have permission to access the S3 bucket and ask me if I want to add it and step me through it if I’m an administrator the account? Also, if I already have access to view the events in the UI, why can’t I have read-only access to the CloudTrail bucket as part of those permissions?

Alright, enough wishful thinking. We need to get this working. Read the documentation.

By default, Amazon S3 buckets and objects are private. Only the resource owner (the AWS account that created the bucket) can access the bucket and objects it contains.

If you are using AWS Control Tower, that would be the Control Tower Administrator account that created this account and it’s CloudTrail configuration.

The resource owner can grant access permissions to other resources and users by writing an access policy.

So is this Access policy a Resource Policy or an IAM Policy? I presume it is a resource policy because the user already has full permission to access S3.

Next the documentation points to a link to create an organizational trail on the command line. But here’s the thing. This was all created with CloudFormation templates by Control Tower. So that command line information isn’t helpful. It seems like somehow I have to edit the CloudFormation Templates Control Tower uses to create this bucket.

Also, admittedly I scanned the documentation but I don’t see anything about adding a user to allow S3 bucket access for Athena. Maybe we can find that somewhere else.

That search leads me back to the Athena documentation which does not address this issue and it points me back to the CloudTrail documentation.

Am I just overthinking this? What if I just update the S3 bucket policy? Of course that puts my whole Control Tower configuration out of whack and I’ll probably get drift errors but at least I can see if I can access the bucket:

AWS let me save the changes.

That’s not a recommended approach, by the way. It’s just that I can’t find any documentation on how else to fix this and it’s taking too long.

Back to CloudTrail and “Create Athena Table.” Right here it should let you select your existing table but it doesn’t. I could go over to Athena and try to find my table but I just click through the Wizard.

It tells me a bucket exists and now I get a button to go to Athena here:

Again with the permission error, but I probably had my S3 path wrong.

I’ll copy the path out of this error message and paste it into my bucket policy.

Well now that I’m actually looking at the bucket ARN, it’s in another account. Now I have to figure out where that is. Back to the SSO dashboard. Ah, yes of course. The Log Archive account. But that’s where I was testing before. Let’s check out that bucket policy.

Ok now I remember I manually set up a temporary CloudTrail log in the test account that’s why CloudFormation didn’t exist for that S3 bucket. I found the proper bucket and the CloudFormation stack that deployed it in the Log Archive account.

So the question is, how can I edit the templates Control Tower used for deployment and redeploy just this bucket?

I found this page with the words “customize” and “Control Tower” in it.

It links to this template on GitHub:

Holy moley. All that is deployed in one single template? That is the opposite of how I deploy anything with CloudFormation. It’s like the monolithic architectures we tried to get away from with micro-services. I just want to edit one bucket policy and I have to risk messing up that whole stack? And I don’t want to figure out that whole stack right now as that is not anywhere near my objective.

When I deploy CloudFormation, as I’ve been doing in my latest blog series, I create simple templates for each resource so I can easily re-deploy it. I can test and re-deploy each component independently.

Well, I’m not going to risk redeploying all of that Control Tower functionality right now if I can even do that. I’m reconsidering my use of Control Tower at this moment. Maybe there’s a way to use the code but decouple it and deploy it myself. Not sure. Not the task at hand.

For now, let’s see if I can edit the S3 bucket policy manually because I’m not trying to write a series on Control Tower. I’m trying to write about batch jobs. And right now I just want to query the CloudTrail logs so I can show how to create a zero trust IAM Policy.

Basically I’m going to see if I can insert my policy into the S3 bucket short of a better option. You probably can’t do that in a well-governed organization, by the way.

Oh of course. Access denied. That’s good actually. That’s what Control Tower should do.

By the way, I already tried accessing Athena using the Power User in the Control Tower Log Archive account and that user didn’t have access either.

At this point I think I have the following options:

1. Spend a lifetime trying to figure out if and how I can customize the Control Tower CloudFormation to re-deploy this bucket policy.
2. Set up a second CloudTrail in my test account and pay double for CloudTrail logs.
3. See if I can access the logs with administrator access in the log archive account.
4. Try to query CloudTrail with something besides Athena.

Number three seems like the easiest even though it’s not a good solution at all for a number of reasons:

• I have to switch back and forth from my test account to the Log Archive account to test my changes.
• If this was a big organization you don’t want to grant people admin access to the log archive account so they can query CloudFormation logs with Athena.
• In fact, you probably wouldn’t want to grant the people writing policies access to the Log Archive account at all.

I’ve spent far too long on this already. This is a problem for another day. Let’s see if giving myself admin access for now works.

And…it takes me like 20 tries to figure out how to add the permissions as always because AWS SSO is so confusing. The UI and the relationships between things is so disconnected but I jumped around back and forth to see different things and figured it out.

Yes, I know, it would be easier and better to just automate creating SSO users I just haven’t had time yet, I don't have a huge organization, and I am just testing.

Refresh the SSO page and login to the AWS Log Archive account as admin. Go to CloudTrail. Repeat the steps to create an AWS Athena Table.

Now the thing is, querying this trail is going to be a bit more complicated because it’s not just a single account. It’s the entire organization.

And….I still cannot access the CloudTrail bucket. I still get the permission error on the S3 bucket even though I’m an admin. Looking at the policy I don’t see how the permissions will work with Athena, and even as and administrator I can’t change them probably due to a Service Control Policy (SCP) in Control Tower — which is good — except that I need to be able to change it to make this work if that is the issue.

I think my next approach will be to write AWS CLI scripts to access CloudFormation as this is taking way too much time.

It looks like I can query CloudTrail with CloudTrail Lake.

But of course there is a cost associated with that:

Compare that to Athena:

Looks like it depends on the amount of data. For me I have a low amount of data and I think Lake will be cheaper. But now I have a question. Am I paying for Athena for all these queries that failed? I have to figure out how to delete all that now.

I went in and deleted the Athena tables and removed the location setting. I hope that is enough to stop any charges.

I know Athena and CloudTrail was on the AWS security exam at some point (based on reviews by others I haven’t taken it) but I wonder if organizations are actually using it if they use Control Tower.

Seems like this should all be a bit more intuitive and maybe just work?

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2022

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab