How a CloudFormation Template Can Secure Your S3 Buckets

ACM.193 Secure templates depend on a secure deployment pipeline

Part of my series on Automating Cybersecurity Metrics and stories on AWS S3 Buckets. The Code.

Free Content on Jobs in Cybersecurity | Sign up for the Email List

In the last post we created a generic AWS S3 bucket CloudFormation template. I hard coded the template to block public access and did not set any ACLs.

AWS is actually changing these default settings to help customers with AWS S3 bucket security this month. They are going to block public access by default. WOOT!

So in theory, I didn’t even need to add the public access block in that template I could have simply left it off. But since I don’t know exactly when that change goes into effect and to be very sure it is set correctly, I am explicitly defining and hard-coding that configuration in the template.

How the generic template helps prevent mistakes

By having everyone in your organization use a defined template to create a particular type of resource, you will help ensure they do not make mistakes in their configuration.

If everyone in your organization uses a template that blocks S3 access and does not change that setting — then you will never have a public S3 bucket!

And you don’t need one.

There are still a few ways a developer could expose data via an S3 bucket policy but we’ll address that later.

So in theory, you create a securely defined template which everyone uses and you never have a public S3 bucket, right? There are a few caveats which you’ll need to address.

Permission to change the template

If the people who can deploy the bucket have permission to change the template, they can simply alter the public block property and deploy that version instead.

Permission to make changes in the AWS console

If people can make changes in the AWS console they can override the changes in the template.

Bottlenecks may foil your best intentions

I always told my DevOps team:

If people are complaining about the deployment pipeline — either they don’t understand why we did something because we did not properly explain it to them, or we’re doing it wrong.

You need to make sure your security rules, guardrails, templates, and frameworks do not block people from doing their jobs.

I haven’t added every possible property to the S3 bucket template yet. A deployment may come along and someone may need to alter the template to support a new configuration or use a new bucket feature added by AWS. You need to make sure you can make that change in a timely and controlled manner.

Otherwise, people will complain. And when people complain the fate of your secure deployment process rests in the fate of executives who may or may not understand why it exists. Keep reading for more on that point.

If you do not have the resources to keep up with the needs of the business when maintaining central control of S3 bucket configurations, you may need to consider an alternate strategy. But if you can, it is the safest approach.

Testing and rollout strategy

When you make a change to your S3 bucket template, it could affect other deployments using that template if it a shared template. When a team needs to redeploy their application, your change could break their deployment even though it works fine in other scenarios.

If your whole organization uses a single template and you make an error when you change the template that creates a production outage or blocks all developers from doing their work, you’re going to have a bad day.

Now before you say, this is why we need separate templates— consider AWS. You deploy your assets on a cloud platform that is constantly changing. With each change they need to make sure that change does not break things for existing customers. This requires testing in advance of deployments — yes, for infrastructure too, not just applications!

Also, AWS uses rolling updates to push out changes. They don’t push the change out to all customers at once. If you make a change perhaps you test it first, then push it out to a few teams at a time to make sure it doesn’t break things, until you finally roll it out to the whole organization.

Consider how you will version your template in case people need to keep using an old version because your new version breaks something. Consider how people will be able to use templates from prior branches in source control — and how you will block prior versions that have security issues.

Centralized deployment systems

Design a deployment process that allows developers to deploy an S3 bucket that allows developers to deploy a bucket by passing in parameters without changing the underlying template.

Perhaps you have a job that developers can run, passing in some parameters via a script and the job uses the parameters an the script to deploy an S3 bucket.

Incorporate checks into your deployment system to ensure developers are following your rules for S3 buckets.

You need executive support to build a secure deployment pipeline

Building a secure pipeline is complex, takes time, and costs money.

If you do not have executive support to build a secure deployment pipeline that is properly architected — you likely won’t be successful.

Even if you do build it, if they don’t support it, it will likely be dismantled once it blocks someone who complains loudly enough.

Ensure that you spend time educating executives why they need a secure pipeline. That is in part, why I wrote my book at the bottom of this post. If executives do not see the value in terms of preventing security incidents, data breaches, reducing overall risk, faster deployments, and saving time on security reviews, then they might not support this approach.

Unfortunately, even if you do explain the risk to executives, sometimes they simply do not listen, care, or have competing objectives.

I was able to architect a secure deployment pipeline at one company and worked with my DevOps team to implement it only because I had complete and total support from my boss that hired me. I only took the job because he promised I could build things the way I wanted. I wanted to build a secure cloud deployment process based on a white paper I wrote for SANS Institute.

Unfortunately, other executives within the company did not fully support the effort. Although we got it done, it was challenging. Some executives didn’t want to take the time to do the things we were doing. I was concerned with rushing to move a product to the cloud that was not designed to be exposed to the Internet instead of residing in a private network (a firewall management interface). They did not seem to understand the purpose of the security controls in the deployment pipeline.

Hopefully now they do, in light of the Solar Winds breach.

In addition, the company’s own security devices were compromised through its update process. I remember trying to explain how an insecure deployment process could infect our security product. I will leave out the reactions I experienced here. My prediction, unfortunately, came to fruition. Malware was able to infiltrate the process and deploy rogue code.

It uses the infected devices’ legitimate firmware update channels to maintain access to the compromised devices by injecting malicious code and deploying repacked firmware images.

Here are a few other articles on the incident.

You need executive support to build a secure pipeline and maintain the security controls, as designed.

Rogue executives, developers, and business system owners

My top DevOps engineer that worked with me at that company went to another company and deployed such a system for them as well. He told me a story about how some assets got exposed because an executive had an intern working outside their deployment pipeline.

The executive told the intern to build and deploy something outside the standard process. The intern, seeking favor from the executive or possibly because the person felt they had no choice, deployed things as directed. It resulted in a security incident.

If you have that scenario in your organization and use of a secure deployment pipeline is not enforced in all cases — people will bypass your it and you possibly expose systems and data. I’ve seen it too many times.

At another company I was in a training session where an executive at a bank said to a room full of developers, “Security told us not to do this but we’re going to do it anyway.” The attitudes of some executives at that company (not all) speak for themselves — one of the largest cloud breaches in history.

If you have executives like this in your organization and people working outside the bounds of the security controls you define or constantly rage against the controls in place, your security team doesn’t have a fighting chance.

Invest in a secure deployment pipeline

If companies want to avoid a security breach, bite the bullet. Architect a secure deployment pipeline. Enforce the rules from the top down — including all executives. If you need help with that, those are the types of calls I answer through IANS Research.

I’ve done it. Sometimes things that sound good in theory don’t work in practice.

I can also provide training on the topic to developers or executives through my company — 2nd Sight Lab. I provide training to organizations, not individuals.

Reactionary controls vs deployment governance

There are other approaches to securing S3 buckets — reactionary controls and policy as code. Both have been used successfully and unsuccessfully.

I’m going to use the Capital One breach as an example, but there are many, many companies with similar problems — or worse. Capital One had some good security practices while I was there and some areas for improvement. Some very smart people worked with me at that company. But security is hard, as I always say. We can use the information from this incident not to blame or point fingers, but to learn how to improve our own security.

Controls were very reactionary when I worked at Capital One. I don’t know if and how that changed after I left. But I spoke to various people who worked there during the breach and apparently the SOC did get a message about a large data transfer, but for whatever reason, notification of the breach came from an external source. So those reactionary controls were not super helpful.

On the other hand, some development teams at Capital One took the step to implement a customer managed encryption key (CMEK) for their buckets. The keys were restricted to only allow their applications to use the key. When the attacker got access to *all* the buckets, those using CMEK’s were not accessible to the role the attacker used.

I showed you how to assign a key to an S3 bucket in the last post. Our generic KMS key template in this series allows you to pass in the roles that are allowed to use the key. Now, you will pay for each key if you assign an individual key to each application. But perhaps the cost of those keys is worth it. You will need to compare the costs to the potential cost if the data is exposed.

What if Capital One had enforced secure deployments through a centralized pipeline? Capital One characterized the root cause of this breach as a “misconfiguration.” Could the misconfiguration have been prevented with a secure deployment process that prevented it from reaching production? (Whether or not it really was a misconfiguration depends on who you talk to as I’ve heard different stories from people with whom I used to work.)

What if Capital One enforced the use of a per-app CMEK through the deployment pipeline with zero trust permissions? Let’s say they used the template I’ve designed and each team created a key specific to their application? The attacker would not have been able to access any data in the S3 buckets, because the attack came by way of a web application firewall (a virtual appliance, not the AWS WAF) and a role on an EC2 instance that allowed access to every AWS S3 bucket — but not every AWS KMS key.

Note that default AWS encryption did not help. Unless policies changed, Capital One enforced “encryption” on all S3 buckets. Also AWS encrypts everything by default now. Encryption doesn’t help you if the attacker has permission to decrypt the files or accesses a running system where the data is encrypted. But in the case of the Capital One breach the attacker was retrieving data stored in S3 buckets at rest and presumably had permission to decrypt files enrypted with default AWS encyrption.

AWS has some new tools for governance since I worked at either of those companies. I’ve already explained what SCPs are. You can also use Config to alert on or roll back unwanted changes. You can use these tools to write policies as code. We’ll continue to look at some of those controls as well for prevention as well.

Although we can block action through policies or react to them after the fact using policies or AWS Config, providing a secure deployment process to allow developers to quickly deploy a secure S3 bucket seems like a more user-friendly approach. Developers won’t make mistakes that need to be rolled back with potentially obscure or confusing error messages. They simply set the values for their bucket and deploy it. They don’t have the option to change controls they shouldn’t.

Now that we’ve looked at how a generic S3 bucket template helps us keep S3 buckets secure, let’s test out the template and use it to deploy S3 buckets for our AWS Organization CloudTrail log.

In the next post I test out the generic CloudFormation S3 bucket template for a server access logs bucket, create deployment scripts, standard naming convention, and other nifty things.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab