When Drift Detection Fails

ACM.447 Be aware of when Drift Detection helps you, when it doesn’t — and why policy documents should be first class citizens

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Part of my series on Automating Cybersecurity Metrics. The Code.

🔒 Related Stories: AWS Security | Application Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post I wrote about a firewall for CloudShell as I’ve been showing how you might use a container to configure your initial AWS account and the risks that come with that approach. Unfortunately it’s a chicken and egg problem when you start up your first AWS account. You have to start somewhere.

In this post I’m going to tell you about a few issues I found with drift detection in a few places.

Drift detection is super helpful. I wrote about here:

But it’s not foolproof. Understand when it works and when it doesn’t — and run your own tests to make sure it’s working if you are relying on it.

If you are going to use Drift Detection you need to be aware of what it does and does not support. Here’s the complete list of what is supported by Drift Detection:

Terraform caching for drift detection

I haven’t used Terraform’s drift detection. It sounds very helpful in a multi-cloud environment.

However, I was reading that it uses some type of caching to speed up the process. Caching can be fraught with problems as reported over and over by type security researchers and has been used in many data breaches (Check out DNS Cache Poisoning, for example and James Kettle’s research on web cache poisoning which I’ve mentioned more than once on this blog).

How long does it take for the cache to get updated and report something is out of sync? That’s problem number one. Could someone take an action, change a resource, and then put it back the way it was so the cached drift detection never detects the problem?

Another problem with caching si that, in general, is simply a constant target of attack. There are many ways to perform cache poisoning on all sorts of caches. So what type of cache poisoning might you perform against the Terraform cache so it doesn’t detect a change? I don’t know but someone should be looking into that.

Perhaps you can turn caching off and only use real-time updates to make sure your Terraform drift detection is not subject to any of those potential issues. In the end, that was James Kettle’s advice on web cache poisoning — if you don’t need it, don’t use it because it’s complicated and often exploited.

I’m also curious what “syncing” means exactly but I didn’t look into it further.

You’ll also want to test that it is actually working if you are depending on it and not subject to issues like the ones I found in AWS CloudFormation drift detection below. Any drift detection system likely has things it doesn’t detect so test it out.

What would be great is if all companies have bug bounties to report undocumented problems with their drift detection system so they can fix them. Hint, hint.

I do not see that Terraform has a bug bounty and so far neither does AWS. I keep hoping.

#awswishlist

CloudFormation doesn’t recognize changes to KMS key policies

For this example, I haven’t tested drift detection specifically but I know that CloudFormation itself doesn’t redeploy a resource when there is a change in a policy. This threw me for a loop for quite some time. Not sure if it has been fixed yet.

When you delete a user that is in a KMS key policy, AWS for some unknown reason alters your key policy instead of warning you that the user exists in a policy and you should change that first and then possibly allow you to override the warning. But you get no warning.

And why does AWS ever alter customer policies, by the way? I think that should not be happening personally. #awswishlist

This also happens, if I remember correctly, if that user exists in IAM role policies but I’m not sure if CloudFormation also fails to recognize those changes.

When you finally realize what’s going on after a bunch of obscure messages and that there’s a logical user ID in your policy instead of an ARN, you think, “Well I’ll just update my key policy and redeploy my CloudFormation stack, right?”

Nope. CloudFormation does not recognize that the key policy no longer matches what was deployed and does not redeploy it. I got around this by feeding a timestamp into a parameter and using the parameter in an output — but that means you’re redeploying the key every time whether you need to or not if you always pass in a timestamp.

Does that mean AWS Drift detection doesn’t work if your KMS policy changes? I haven’t tested that out yet. If you’re depending on that you might want to do so.

AWS::IAM::Policy does not support drift detection

Are you using AWS::IAM::Policy in CloudFormation? This policy does not support drift detection. Use the alternate and more specific policies: GroupPolicy, RolePolicy, and UserPolicy.

Drift detection doesn’t detect inline role policies

This is another one I haven’t verified but you should probably look into if you’re counting on drift detection. At some point it did not detect inline policies added to a role.

Drift Detection Doesn’t Work on JSON Strings in Some (ALL?) Cases

While deploying an organization resource policy I ran across this message:

What’s a JSON string versus a JSON object?

A string is written like this in your CloudFormation template:

JSON:

 "Properties": {
        "Content": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"AllowDescribeOrganization\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::111122223333:root\"},\"Action\":[\"organizations:DescribeOrganization\"],\"Resource\":\"*\"}]}"
  }

YAML:
 Content: >-
        {"Version":"2012-10-17","Statement":[{"Sid":"AllowDescribeOrganization","Effect":"Allow","Principal":{"AWS":"arn:aws:iam::111122223333:root"},"Action":["organizations:DescribeOrganization"],"Resource":"*"}]}

An object is written like this:

JSON:

"Statement": [
            {
              "Sid": "AllowDescribeOrganization",
              "Effect": "Allow",
              "Principal": {
                "AWS": "arn:aws:iam::111122223333:root"
              },
              "Action": [
                "organizations:DescribeOrganization"
              ],
              "Resource": "*"
            }
          ]

YAML:

        Statement:
          - Sid: AllowDescribeOrganization
            Effect: Allow
            Principal:
              AWS: 'arn:aws:iam::111122223333:root'
            Action:
              - 'organizations:DescribeOrganization'
            Resource: '*'

I would venture to guess that the above statement applies to more than just Organization Resource Policies.

The real problem: Policy Documents and Statements are not first-class citizens in CloudFormation

The real problem here is that policy documents and statements are not first class citizens in CloudFormation. They are a blob of text you pass in that cannot be easily verified.

Some other teams have started to do. Here’s a property called “config” and you just pass in a JSON blob. (Ahem, AWS Step Functions.) That’s not CloudFormation and it seems like someone didn’t want to take the time to implement things correctly. I wonder how well Drift Detection works on all the variations in which Step Function configuration can be written but haven’t checked.

CloudFormation has a set of properties with verifiable types and a structured format that can be validated by things like AWS Drift Detection due to their consistent structure. I wrote about the importance of data types here:

I wrote about how CloudFormation helps security here — but it doesn’t if it’s just a big blob of unverifiable text or requires inconsistent verification methods:

Instead of having a CloudFormation resource like:

Policy:
  Name:
  Content:

It should be:

Policy:
  Name:
  Content:
    Statement:
      Principle:
      Action:
      AllowDeny:
      Resources:
      Conditions:
          ...condition properties here...

This would make it so much easier to verify, write, and generate policies than it is today. It would likely make drift detection easier as well for any type of AWS policies.

#AWSWISHLIST

Pretty please. Yeah, I know it’s not easy. Maybe a new alternate set of policies that use this new approach and an underlying infrastructure that implements it for all of them using the principle of abstraction.

I think all kinds of obscure, time wasting issues when implementing CloudFormation and IAM policies would be resolved by this one single change. Drift detection would likely work more consistently as well.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2024

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab