If You Delete a Resource Created By CloudFormation Will It Be Recreated If You Redeploy the Stack?

ACM.412 Taking a closer look at what happens when you delete resources deployed by CloudFormation stacks

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: AWS Organizations | IAM | Deploying a Static Website

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post I added environment names to my AWS Naming convention.

In this post I’m going to dig into another problem I had while trying to rearrange my Organizational Units created by a CloudFormation template.

How well do you really know what happens when you add, remove and change CloudFormation resources. Have you tested that? You would expect AWS to test all of this and make sure it’s working properly but what is “properly”?

Sometimes “By Design” might not equal to what you expect that design might be. In other cases, there may be a glitch here and there and for critical resources you should not just assume things are working as expected. You should verify the implementation at each step.

Here are a few examples of things you should be aware of and verifying in your own environment.

In my last post, I explained how I deployed a resource (my nonprod organizational unit) incorrectly and tried to fix it.

Unfortunately, CloudFormation did not do what I expected. I thought the easiest thing to do at this point would be to delete the resources and start over.

Deletion Policies

Typically I delete resources from the CloudFormation console by deleting the associated stack. Here’s where you can have your first issue.

A CloudFormation stack can have a Deletion Policy assigned to it that retains the resource even though you delete the CloudFormation stack. This may or may not be what you are expecting. An AWS Organization, for example, retains the Organization resource by default when you delete the CloudFormation stack but you can override that. In general it didn’t work right when I tried to override it so I don’t recommend doing that.

If you do delete a stack you needed and the resource exists and you don’t want to redeploy it to get it back into CloudFormation what can you do? I create an import of an AWS Organization into a CloudFormation stack here:

Some other resources won’t delete when you delete a stack like Users and KMS Keys which have a minimum 7 day waiting period when you schedule their deletion. Keep all these things in mind when deleting stacks. Test resource deletion as well as creation before using resources in production environments or trying to delete production resources. It’s a good idea to test creating and deleting your entire stack for applications and resources you create so you are aware of and plan for all these nuances.

Can you manually delete a resource and redeploy an existing stack to recreate it?

When I was fixing this problem I manually deleted my resources and tried to rerun the script that creates all my OUs in parallel again.

What do you think happened?

Nothing.

CloudFormation compares the template your provided to the template you last deployed — not the underlying resources. It won’t deploy anything if there is a discrepancy between what exists in your account and what is in the template and the template hasn’t changed.

So you could be looking at this whole list of lovely CloudFormation stacks and nothing exists behind them!

In order for me to redeploy my OUs I could do one of two things:

1. I can delete the stacks altogether. This may or may not be possible if you have dependencies on those stacks and things can get into a very mixed up state sometimes when you do that.
2. If you have dependencies and want to force a stack to redeploy I’ve done that by passing a timestamp as a parameter and then outputting the timestamp in the outputs. That looks like a change to CloudFormation and it redeploys the stack. The problem with that is that if you change your timestamp every time you’re constantly redeploying a stack that may not have actually changed.

I have this commented out right now but apparently I had to force a user update at some point so I added this parameter:

Then I write it to the outputs:

CloudFormation sees the new timestamp as a change and will update the stack even if nothing else in the template changed.

Problems with DriftDetection

You can use DriftDetection to try to find discrepancies between the configuration defined in your CloudFormation stack and what exists in your account.

I want to look into some additional automation around this later but one way to detect drift is to click on a single stack in the console and then Detect drift.

I can see that I have an IAM organizational unit stack in my list of CloudFormation stacks:

I can also see the organizational unit exists:

I’m going to leave the stack intact but delete the nonprod-projects organizational unit.

Now return to the CloudFormation stacks and refresh the screen. The stack still exists. Now use Drift Detection on that stack.

Here’s what I get:

It doesn’t tell me the resource is missing, it just gives me an error. That error message is not very helpful. You can find the root cause in CloudTrail as explained in other posts.

The root cause is that I don’t have permission to detect drift with this user on these particular stacks which belong to the root-orgadmin user.

Well that led me down a rabbit hole that took like a day to resolve between policy issues and the AWS console locking me out momentarily. I granted my user permissions and had to resolve some unrelated errors. I’m not exactly sure how the policy got into the state it was in but this seems to be working.

I tried it again and did not get the error. Drift detection shows that it has been initiated. When it completes you can choose View drift results from the action menu.

It doesn’t alert me when it’s complete but when I view drift results I see that it reports the resource got deleted.

That tells you to be aware that just because you see a stack doesn’t mean the stack exists.

Additionally, what is in the stack might be different than what you first deployed and that would be reported above as well — most of the time.

I just ran across this warning about policies and drift detection when implementing an AWS Organizations delegation resource policy:

This warning is on the CloudFormation resource documentation page:

It was not clear to me at first what their distinction was between a string and an object but if you scroll down you’ll find it.

JSON Object:

[YAML truncated — see the page]

JSON String:

So this got me wondering — is the Organizations resource policy the only one that has this issue? Policies are crucial for security and governance and you’ll want to know if they change. If you are counting on drift detection you may want to evaluate this for any type of policy you create. Also report issues to support and submit feature requests to fix any issues to your account management representative for your organization. AWS is good about fixing issues when a customer requests it!

Well, hopefully those few notes will raise awareness of issues you might face when using AWS CloudFormation when you’re expecting your stacks to match what’s in your account. Don’t assume — check it regularly!

I hope to write more about drift detection in the future. For now, I need to move on.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab