How do I convince my security team to let me stop using CloudFormation
ACM.283 What problem are you really trying to solve, what problems does your solution cause, and are there any alternatives?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Check out my series on Automating Cybersecurity Metrics | Code.
🔒 Related Stories: Container Security | AWS Security | Application Security
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the last post before a hiatus I wrote about deploying a container registry on AWS with AWS CloudFormation.
After that, I was off to Seattle to meet with AWS teams to talk about new products and services and hang out with other AWS Heroes. So, I’m of course skipping around to cover a few things from that event but I’ll get back to containers shortly. And, it’s all related.
If you don’t know what an AWS Hero is you can find out more about that here:
I haven’t been able to attend this event for a couple of years but this year was fortunate enough to do so. It’s a chance to get together and learn about what’s coming down the pipeline for AWS products and provide feedback on those products and new features. A lot of what was discussed is not something I can make public but I can address some general topics and themes I heard at the event.
At these types of events, it’s always nice to be able to provide input, but I also always learn things by hearing not only what AWS has to say but also others working in the industry trying to solve various problems. I like hearing what other challenges people are facing and thinking about how to address them. I can also learn things from other heroes who have more experience in different areas. I love the questions people ask me that get me thinking about things in new ways, from a new angle, or get my thoughts going down a different path I haven’t thought of before.
What problem are you trying to solve?
When you’re trying to solve a problem, it’s always good to step back and determine if you’re solving the right problem. Not only did I ask someone that question, I literally heard one one of my fellow heroes say the same thing to another person a few minutes after I said it. It’s a great question to revisit often when you’re trying to solve a problem.
Instead of just saying what you want to do — I want to bypass all security controls like I heard one person say :-D — express the root problem. I get it. You want to go fast. Taking time to explain to the other person what you are doing slows you down.
At the same time, consider the problem the other person is trying to solve who is “blocking” you and the problems you might cause them with your solution. The two sides need to come together to form a solution that works as best it can for all parties involved.
That’s what the person I was talking to is trying to do. He’s doing the right thing by reaching out to the person on the other side of the problem to figure out what what he wants to do is a problem for them and how to fix it. He has a problem to solve but so does the person who’s passing down a requirement to use CloudFormation and he’s trying to understand why that requirement exists.
The question “What problem are you trying to solve?” reminds me of the Five Whys. When something happens that you are trying to prevent in the future, you ask why it happened. But typically the first answer isn’t the final answer. You ask why again after the first answer to dive deeper, and then ask why to the answer to that question, and so on, until you get to the root of the problem.
I was also thinking at the event, “Don’t try to produce a solution that is looking for a problem.” That’s a related but also relevant issue when people are trying to produce solutions for customers. Make sure you are not using technology for technology’s sake if there is a simpler way to solve the problem.
So is there a better way to manage deployments besides CloudFormation? Like AI? Just kidding.
How can we convince Security Teams to let us not use CloudFormation?
So here’s the scenario. Someone asked me, how do we convince security teams to let us not use CloudFormation? I said “What’s the problem?” and the answer was “We don’t want to use CloudFormation.” But that’s not the actual problem. The problem is something to do with the way CloudFormation works that is making your job hard in some way.
As it turns out, for this person, the problem was speed. We want to deploy faster. By going directly to the APIs we can deploy faster than if we use CloudFormation. CloudFormation is slow.
We want to deploy faster. << The actual problem.
There’s another underlying problem with why a lot of people don’t like CloudFormation. They don’t want to learn YAML or JSON and use something they already know. If you are struggling to learn CloudFormation, check out this post.
OK, so there are a number of solutions to that problem. One is to stop using CloudFormation, but then you have to address all the problems CloudFormation solves. It may solve your particular problem, but what problems does it introduce for a company that has to meet compliance standards to a.) prevent data breaches and/or b.) meet compliance regulations enforced on the company by external regulators => to achieve a, prevent data breaches.
There are some other alternative solutions which I wrote about in my next post:
But let’s address what a solution that does not use CloudFormation would need to do in order to truly replace the functionality CloudFormation provides that helps organizations achieve security and governance objectives as I’ve shown repeatedly in my current blog series.
Requirements versus How To
When I speak to security teams I tell them, don’t tell people how to do things or what technology people have to use. Instead, specify your security requirements. I used to tell business people the same thing. Don’t tell me how to build your system and what the screen needs to look like. Tell me what your business problem is and can likely solve that problem faster and more cost-effectively than what you are proposing I should do to solve the problem.
The same is true for security teams telling developers what they need to do to meet your security objectives. Don’t tell them, “You must use CloudFormation.” Tell them, here are the requirements for a deployment system. Use a mechanism that meets these requirements. Let them figure out how to do it.
My requirements for an alternate solution for CloudFormation would include the following:
- Everyone in the company needs to deploy the code in a way that produces a minimal set of logs to review all deployment activity. The security team is generally small and has limited resources for monitoring so keep it simple. A larger security team may be able to support more than one deployment method, but the more complete but at the same time the fewer logs the security team has to evaluate, the more secure the company will be. When using CloudFormation, you get events related to each stack and CloudTrail events. If using something like Jenkins, that deployment system will also produce deployment logs. GitHub also produces logs indicating who pulled, pushed, etc. Any new mechanism for deployments should not create additional log files and locations to monitor but any deployment actions taken should be in the logs.
- There is generally a single team that manages the deployment system to keep it secure. Not all companies do this, but those that have a good handle on governance will do this. Those that have policies and mechanisms in place to prevent misconfigurations that lead to data breaches and breaches of the deployment systems themselves will generally have a single team for deployments. Your proposed deployment mechanism is should be able to be managed by that same deployment pipeline and not create additional work for that team that they cannot handle and still keep systems secure. That integrity of that pipeline is maintained by a team that understands the crucial nature of that pipeline in how it may contribute to a supply chain attack or a data breach. (See my posts on the Solar Winds Breach for why that matters.) They will generally ensure the deployment systems are configured securely, including log integrity, zero trust policies, and patching systems regularly.
- Does your company have systems in place for scanning and evaluating code for errors, misconfigurations, vulnerabilities, and malware? How will they monitor your new deployment mechanism for similar problems? Some companies use linters and secrets monitoring software. They may also have monitoring for CVEs in their pipeline and a mechanism for tracking SBOM (Software Bill of Materials.) Does your new deployment mechanism work with all existing scanning and code analysis checks? This will vary by company.
- Security teams should be able to see a list of all the resources deployed in each account in one place like CloudFormation provides. The team doesn’t want to be looking in six different places to try to figure out all the things deployed in an account. Side note: It would be great if this list were available for the entire company in AWS Organizations, or in a single OU, or for a single account. #awswishlist
- Within the list of all the resources deployed in an account, security teams need to be able to click a link and see the code that deployed the resources like you can when you click the link to view a CloudFormation template. I put a link to the GitHub repo file in my CloudFormation templates. That way a security team can go to the file and check the history to see if the code used to deploy the resource matches what is in the repo, who changed it last, and what changes were made over time.
- Security teams need to be able to monitor for drift detection the way you can in CloudFormation. They need to easily and quickly see if something that has been deployed does not match what the code indicates got deployed. (See my posts on the Solar Winds Breach for why that matters.)
- In the list of deployments, the security team needs to be able to get a list of the resources deployed by a stack and click on a link to navigate to that resource the way you can in the CloudFormation console. That makes it easy to evaluate resources deployed in a stack.
- The parameters passed into the code need to be available as they are in CloudFormation.
- The outputs for each stack should include any IDs or ARNs. https://readmedium.com/simplifying-id-lookups-from-cloudformation-outputs-85186160a597
- The deployment mechanism needs to be able to enforce governance when resources are created by having all teams use standard code “templates” that abide by company policies. As long as people use that template they are deploying things that are in compliance with the rules the company has established internally or that that the company must follow to be compliant with external regulations (PCI, HIPAA, NIST, SOC2, ISO27001, etc.)
- The code used to deploy the resource has to produce the same results every time.
- Executable code should be separated from data, parameters, or metadata that configures a resource. In other words, configuration should be separate from executable code. Or, the control plane (the deployment system) should be separate from the data plane (the code that describes the thing you are trying to deploy using the deployment system.)
- The new deployment mechanism should abide by the DRY principle so it does not produce unnecessary repetitive code. I’ve been showing how to use a single, compliant template for a KMS key across an organization, for example.
- Security teams need to be able to to easy see who, what, where, and when related to deployment activity. The way I have been deploying CloudFormation stacks in my latest blog series I can see by looking at the stack name: which role deployed the resource, the resource that exists in the stack, and the resource name. Security teams need to be able to enforce that naming convention in any alternate mechanism of deployment.
- The way I designed my code, each role can only deploy and modify their own stacks so that also should be supported.
- The ability to search by resource type, principal that deployed the stack or name. Based on my naming convention I can do that in the CloudFormation search bar. I can simply type “S3” in the search bar and see all the stacks that deploy an S3 bucket. I can also type “IAM” and see all the stacks deployed by my IAM role. I can search on the name I gave the resource as well.
- The ability to enforce a single deployment mechanism so all the deployments are handled in a consistent manner and people are not bypassing that mechanism with manual CLI commands or clicking around in the console. In a prior company, I gave developers a sandbox and access to read only accounts in a development account that mimicked production where resources were deployed via CloudFormation. All deployments were easy to find in those accounts because they only occurred via CloudFormation without an exception. How can you replicate that if not using CloudFormation so that security teams and others can easily identify all deployments in an AWS account?
- A way to search for all actions related to resource deployments using a single search term in CloudTrail. If it’s not “CloudFormation” how will I find all deployment events in CloudTrail logs with a single search term?
- The solution needs to support termination protection for a stack as can be done in CloudFormation.
- Provide a list of deployment events including the error message that caused a problem with a stack for each stack and each deployment attempt (successful or failed). That may be relevant in the case of a data breach where an attacker gets into an account and is trying to delete or modify stacks, a rogue insider is doing something they should not, or a developer is making a mistake that causes problems which need to be investigated.
I will probably think of more later, but that’s a decent start. CloudFormation provides a lot of security benefits that could possibly be replicated, but it is not a simple feat to do so.
Is ditching CloudFormation the only solution to the problem?
When you think back to the original problem, ditching CloudFormation is one potential solution to the problem faced by one person or group. However, that solution may cause a lot of problems for another group of people trying to do their job in the organization. And although slower deployments is a hassle, it may result in saving the company millions of dollars in the event of a data breach.
But perhaps there is some other solution to the problem, or at least a way to improve the situation for the developers who want to move faster.
Feedback to Amazon and the AWS Wish List
One solution would be to make CloudFormation work faster. When I suggested that, this person said, “Oh they always say it’s going to be faster but it never happens.” Well, perhaps these concerns need to be addressed more emphatically and by more people. Perhaps if more people submit this request to the AWS Wish List, it will happen.
I’ve written about this before but you can submit a wish to the AWS Wish List by sending a tweet (and yes I am still calling it a tweet) with the hash tag #awswishlist.
If you don’t like Twitter then maybe you can tweet one time requesting that this list be adjusted to also work with Mastodon or GitHub or to be able to add to it from the AWS console, or that all feedback submitted on the website goes into the AWS Wishlist if you check a box or something like that.
In any case, you have a way to submit this request and get a lot of other people to like your tweet and reinforce that this really is a big problem. If it is, a lot of other people will likely back you up.
Leveraging CloudFormation differently
Perhaps there are mechanisms for leveraging CloudFormation differently to get around the inherent delays in deploying a large stack. While at the AWS Heroes event I explained to AWS and others that I do not create monolithic stacks to deploy my AWS resources. If you have been following along in this blog series, you see that I deploy a single resource in a single stack. This has some pros and cons.
One of the pros is that I can speed up deployments when testing individual resources. One of the cons is that I lose the dependency management provided by CloudFormation and supposedly faster deployments through parallel deployments of resources with no dependencies. But in reality, I can troubleshoot faster with my own alternate method using single templates for individual resources. I just need to understand the dependencies if I choose to do that.
I’ve already shown how I use single templates and why throughout this series — often for governance and to reduce code using the DRY principle mentioned above. I’m going to summarize and reiterate that in a future post as it came up a lot while I was at the AWS Heroes event. However, I’m also going to show you how I make my CloudFormation deployments faster. It may not be as fast as calling an API, but it is faster that redeploying a monolithic stack over and over again. I’ve also though about some tweaks that might help me speed up deployments even more while talking through my approach with the CloudFormation team.
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2023
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab