Create an Organization CloudTrail with CloudFormation

ACM. 202 Automating deployment of a trail to monitor events across all AWS accounts

Part of my series on Automating Cybersecurity Metrics. The Code.

Free Content on Jobs in Cybersecurity | Sign up for the Email List

In the last post in this series, we added permission for CloudTrail to use the KMS key we created for that purpose — or so I thought. Nothing is ever simple with KMS. In that post, I attempted to make this code work for any service.

Now we should be able to deploy a CloudTrail for our Organization. Just a reminder that I covered the steps to enable an AWS Organizations cloud trail here:

Here’s CloudFormation for a CloudTrail trail. As you can see there’s an option for IsOrganizationalTrail. Let’s look at all the properties and see what we need.

CloudWatchLogsLogGroupArn and CloudWatchLogsRoleArn (No): We would use these values if we wanted to send logs to CloudWatch.

EnableLogFileValidation (Yes, value=true): AWS creates a hash of every file it delivers to validate the integrity of the logs.

I do not see that this costs extra on the CloudTrail pricing page:

However, you will pay for additional gets and puts and S3 data storage for the additional hashes, I presume. I’m going to go ahead and turn this on and then evaluate the cost after taking a look at it.

EventSelectors (No / Not Required): Log only selected events. Note that by default all management events are logged and no data events are logged (like accessing an S3 bucket).

I’m going to revisit this later, because it is possible to create multiple trails for an organization. I would like to have different trails for different types of logs.

What I would really like to be able to choose events to exclude like this:

IncludeGlobalServiceEvents (Yes, value=true): Yes, we want IAM logs and events for another global services. I can’t imagine anyone would want to turn this off, but you might want to create a separate trail for IAM vs. other types of events.

InsightSelectors (No / Not Required): A list of Insight types. The current allowed values are: ApiCallRateInsight and ApiErrorRateInsight. These are valuable insights, however, I wonder if these are highlighted in AWS GuardDuty. This finding below may cover the same information and since I’m going to enable GuardDuty next I’m going to skip this one.

IsLogging (true / required): used to turn CloudTrail on and off.

IsMultiRegionTrail (true): We want to log for all AWS regions.

IsOrganizationTrail: (true!)

KMSKeyId (Yes): our KMS key id from the output of our Key stack.

S3BucketName (Yes): our CloudFormation bucket.

S3KeyPrefix (No): We did not set a prefix for the logs in the bucket.

SnsTopicName (Not Yet): Used or notifications. Not adding at this time.

Tags (No): No tags.

TrailName (Yes): We’ll name our trail and include the organization Prefix like this [Prefix]-Org-CloudTrail.

CloudTrail CloudFormation Template

Functions

Deployment Script

Errors

Ah…yes. Here we go. The S3 bucket policy is invalid. S3 buckets and policy are notoriously difficult to create properly, which is why I avoided them up to this point. Now we have to figure out what is wrong here

Recall that I created the bucket policy here, merging what was in the documentation and what was in my policy for the organization CloudTrail created by AWS Control Tower. Some discrepancies existed.

I’m going to head over to S3 and click on my bucket, Permissions, and then compare the bucket policy to the two policies in my prior post to try to find what’s missing.

One thing I know right away was in the Organizational CloudTrail policy but not in the AWS documentation policy was ListBucket. Add that, but we don’t need AWS config yet.

That did not solve the problem.

The other thing I see missing from the AWS Organization CloudTrail bucket policy is the ability to get the Bucket ACL. AWS says they are disabling ACLs in recent announcements, but as I showed you in a prior post that’s not exactly accurate.

Let’s try adding this statement and run deploy again.

No worky.

The other thing I did was temporarily set the ARN to * in the source arn condtion:

Maybe that’s not good. Back to my conditional policy statements, let’s try to set the condition to no value if the value is not set.

I’ll add a CloudFormation condition to determine if the CloudTrail ARN is provided or not.

I changed the condition in the bucket policy statement (a different type of condition) to this:

The other thing I noticed was that Control Tower uses the following format for the PutObject ARN:

The documentation uses this format:

In the end I checked the log bucket and here’s the format for an organization so we can match it (presuming you did not add prefixes to the CloudTrail or S3 configuration).

[bucket arn]/AWSLogs/[Organization Id]/[Account Id]/*

Note the folder has multiple account IDs so we’d need an asterisk rather than a specific account. I ended up using this.

[bucket arn]/AWSLogs/*

I could make it more specific but CloudTrail should be the only thing to have access to write to this bucket in the end. It is up to AWS to make sure CloudTrail logs properly, though we will restrict it to the appropriate bucket.

KMS Key Permissions

Looking at the error message it now says I don’t have bucket permissions *or* KMS key permissions set correctly. I’m going to go with KMS because KMS error messages are always weird and usually not as helpful as they could be. Since the error message changed, I am guessing KMS is the problem now.

It would be much more helpful if CloudFormation would tell you specifically which one it is and where the problem lies.

Aha. Over in my key policy in the AWS console:

How did CloudFormation even let me deploy this? With all the restrictions going on it seems like this should be an obvious catch with an obvious error message like the one reported here. The statement doesn’t have an action.

I also realized I wasn’t passing in “cloudtrail” for the CloudFormation ServiceParam for the key deployment. I think that was initially by design but now that I know CloudTrail needs access or it won’t deploy I’ll go ahead and pass that in.

I will have to make it work without a specific ARN because I can’t deploy without one apparently.

What I did next was manually delete the KMS condition out of frustration.

Then I got a new error:

Hmm. So turning on all the organizational features wasn’t enough? I don’t recall the documentation telling us to do that. How did I miss it? Here’s what it says:

I’m currently running these scripts in a sandbox account I created using the credentials of the OrgRoot user from the management account. Is that not good enough?

Ah, here’s what I missed, I think:

After you enable all features, you must configure Organizations to trust CloudTrail as a trusted service.

I’m going to add that to my organization_functions.sh file:

and call it from my deployment script for the organization:

Next, I confirm that yes, I can deploy the organizational CloudTrail if I remove the condition from the key, essentially allowing CloudTrail to encrypt or decrypt anything with it in the account. That’s not what I want but now I know exactly where the problem lies.

Also, now that I have the CloudTrail deployed I can take a look at the ARN and match the policy to the ARN. It looks like this for an organizational CloudTrail (with no prefix):

arn:aws:cloudtrail:[region]:[accountnumber]:trail/[orgprefix]-org-trail

Alright back to our Key policy. I thought I saw the problem. I had the bucket ARN where I thought the CloudTrail ARN should be.

Additionally, we don’t really need to restrict logging to a specific CloudTrail ARN.

So I changed the condition to this:

But it doesn’t work with the account in there even though the ARN in the console only has the account ID of the management account wherever I check it.

And after all that searching around, I find the default CloudTrail policy in the documentation and this message. Yes I know.

So let’s look at what the default policy contains.

Cross-account decryption? Hmm.

I wonder why the EC2 service needs to create an alias. My trail works without this.

Here’s the conditon that was giving me grief that I removed:

 "StringLike": {"kms:EncryptionContext:aws:cloudtrail:arn": "arn:aws:cloudtrail:*:account-id:trail/*"}

AWS recommends not using this for an organizational CloudTrail:

"StringEquals": {"kms:CallerAccount": "account-id"},

They allow CloudTrail to describe the key which doesn’t seem to be necessary.

It appears this statement below allows CloudTrail to encrypt the logs.

   {
     "Sid": "Allow CloudTrail to encrypt logs",
     "Effect": "Allow",
     "Principal": {"Service": ["cloudtrail.amazonaws.com"]},
     "Action": "kms:GenerateDataKey*",
     "Resource": "arn:aws:kms:Region:account_ID:key/key_ID",
     "Condition": {"StringLike": {"kms:EncryptionContext:aws:cloudtrail:arn": "arn:aws:cloudtrail:*:account-id:trail/*"}}
   },

Now this is interesting because it does *not* appear that you have to grant encrypt and decrypt permissions in this case to allow CloudTrail to encrypt the logs. The only permission here is GenerateDataKey. In the past, I could not get encryption to work until I added Decryption permissions. Or at least that’s what CloudTrail told me to do. Hmmm. I want to revisit all that now. Was it just a misleading CloudTrail log from KMS, or are there instances where encrypt and decrypt are required?

Anyway, let’s try to see if this works. I’ll change the policy to the above one more time.

No.

No matter what kind of condition I add, CloudFormation complains that CloudTrail will not have enough permissions. And this is taking waaaaaaay too long so for now I removed the conditions and deploy the key policy with this statement that allows CloudTrail to encrypt logs with this key:

That is definitely not what I want. So I’ll have to take a look at the logs and see if I can figure out how to create a key policy that is more restrictive, but works. It could be that CloudFormation is throwing an incorrect error message again.

The whole KMS key policy is too long to post but you can check it out in the repo here:

Remember this key is designed to deploy every KMS key I need in my organization (so far). I wouldn’t do this unless you have one very small team who really knows what they are doing managing this policy.

Also note that I did not need to create a multi-region key — only a multi-region trail. You might want multi-region keys if you are backing up the data to another region or need to access it if a particular region is down.

I’ll possibly revisit multi-region keys later.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab