Letting Terraform Handle The Complexity of Multi-Cloud Deployments
Multicloud.9 You’re not actually reducing the complexity, you’re letting someone else handle it — understand the implications
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
⚙️ Check out my series on Automating Cybersecurity Metrics. The Code.
🔒 Related Stories: Multi-Cloud Security | AWS Security | Azure Security | GCP & Google Security.
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the last post, I wrote about abstraction for multi-cloud security. Abstracting out the commonalities can help us reduce complexity and lines of code. That only works when there’s enough in common for that to make sense.
As I explained, under the hood, all three clouds implement deployments to their platforms in very different ways. I showed you how you might write some code to hide some of that complexity, implement security controls, and make your life easier. Only doing all that isn’t really simple.
In this post we’ll take a look at Terraform and see if and how it can help us with all that complexity. Terraform offers a way to deploy resources to multiple clouds using a consistent mechanism and template definitions for each type of resource you can deploy in your cloud environments.
Install the Terraform CLI
To use Terraform, write and test deployment code, the first thing developers generally do is install a Terraform client to execute the commands. This page shows you how to deploy the CLI.
Evaluating the security of the Terraform CLI
The first thing you will want to understand is the security of that CLI. One basic thing you can check is how many CVEs exist for that tool. I wrote about CVEs here:
As proceed to look this up, I will tell you that I have no idea what I will find because I don’t use Terraform on a regular basis. I haven’t looked at this since about 2018. I used to show people how to use it in class labs for the reasons we are discussing. Let’s see what we find. There are not too many CVEs listed, which is good, but there was a critical vulnerability earlier this year:
You can look at the details to understand if and how it might affect your organization.
In any case, if you choose to use Terraform, you will want to understand how the vendor will prevent problems like this in the future, look at how quickly this was fixed, and make sure you keep any software for Terraform deployments up to date.
If you are using developer workstations in the cloud, it’s easier to force refreshes and updates and keep that software away from malware installed by phishing attacks (to a degree). On my class and penetration testing systems I have a way to automatically deploy new machine images that updates all the software on them for each penetration test, class, or as needed.
If you are allowing people to deploy software on their local machines you need to include the Terraform CLI along with all the other tools on the developer workstation.
You also need to make sure any server side software used in deployments is kept up to date.
Monitoring CVEs is only one aspect of securing software development and deployment processes but it is a start. A lot of the concepts that apply to git and GitHub also apply to Terraform and any other deployment product you use.
Not only does Hashicorp have a CLI they offer some enterprise and cloud sofware as well, so you’ll want to get for any vulnerabilities in those products too.
Does Hashicorp have a bug bounty?
No, they do not. That means anyone unscrupulous who finds a vulnerability might be motivated to take it elsewhere. Or they may simply not take the time to flesh out the problem and explain it to Hashicorp because they need to earn a living elsewhere. That may also be a reason why the number of CVEs reported is low.
This tool can deploy things in your cloud environment. It has a lot of access — potentially to all your major cloud platforms. You may want to understand and research how they assess their products and get a copy of penetration test reports carried out by external companies to overcome the lack of exposure to a broad range of security researchers.
Bug bounties can require a lot of time and effort on the part of the company that needs to implement them, so they may be using other means to secure their products. Small companies especially may not have the resources to implement a bug bounty. Is Hashicorp a small company? It’s not a huge company like AWS. Here’s what the Crunchbase has to say:
Authenticating to your cloud environment to run commands
How does Terraform work? You write code and you send commands to the Hashicorp CLI to deploy things in your cloud environment, similar to the methods I showed you for AWS, Azure, and GCP.
Terraform is essentially overlaying and calling all those commands on your behalf as I explained in the last post. If you want Terraform to deploy things to you, you need to provide credentials to it to perform the deployment actions in my prior post. Here’s what that looks like:
AWS:
GCP:
Azure:
As you can see, if you use Terraform to “make everything easier” you still have to manage credentials, access, and IAM policies (permissions) for any user or application that leverages terraform to deploy resources in your cloud accounts. You would need to do that either way so it’s not any better or worse than managing things in each account separately.
Writing code to deploy resources
One nice thing for developers who use Terraform is that you can use a consistent language to deploy resources. Here is a block of code written in JSON. You can also use a Terraform-specific language.
This configuration written in JSON tells Terraform to deploy a VM in your AWS account. If you know anything about AWS you will notice that the configuration items and terminology below is very specific to AWS. You can’t simply use the same template to deploy a VM with the same configuration to AWS, Azure, and GCP.
You’re still going to need to understand the underlying configuration types, parameters, and acceptable values for each individual cloud provider. I already explained that in last year’s IANS presentation and showed some code that I used to provide in my multi-cloud security class.
I don’t teach that multi-cloud class anymore. It’s just not feasible or something I can do in good faith in a short period of time, as I already explained in a prior post in this series. It is more valuable to teach the fundamentals of security and give people the tools to properly evaluate things themselves than try to explain every always-changing aspect of every cloud provider or give them the false impression that one class can make them an expert on all three, as hopefully you understand by this point in the series.
As you can see the complexities of deploying to AWS, Azure or GCP cannot be masked and merged into a single common format. Just like when SQL came out — the magical standard by which all databases would be queried — you still can’t simply take your code from one SQL-compliant database and drop it in another. There are variations. Each cloud provider offers different services in different ways and there’s not a one-to-one mapping of all the configuration values you can set, even with something as basic as a virtual machine —one of the initial core components on any infrastructure as a service (IAAS) cloud platform.
One benefit of using Terraform
So if you can’t write one single template and use it to deploy to every cloud provider, what is the benefit of Terraform? Well, I’ve shown you already that you may choose to use an industry standard language like JSON to implement the above templates. That way you can hire people who already know JSON, but may have to learn the details of deployments with Terraform if they haven’t done that before. Instead of having people using YAML and Bicep and JINJA, everyone is using the same language in deployments. Though the configuration templates may vary from cloud to cloud at least you can use some standard JSON linters to find errors.
I say this but I know some developers will cry who love JSON or YAML or Bicep or JINJA or Terraform native configurations. I get that. When I took over a team at a company, we had a meeting as a team to discuss the pros and cons of different approaches and land on two programming languages everyone would agree to use. That way at least we didn’t have people using 50 different languages for a small team with limited resources. Every company will have to decide for themselves what they can realistically secure and support.
I wrote about choosing programming languages here. I should link to this post when considering which templating language to use for cloud deployments.
Terraform Open Source Controversy
Just recently some controversy erupted when Terraform changed their license in an effort to try to make money off what they had built, I suppose. There are many different opinions and I’m not going to get into all of that but I kind of like this title.
How does this affect you? You’ll need to understand the license. Ask your legal team. My understanding is that you can still use it for free but not if you’re using it in some product you’re reselling. In that case, pay Hashicorp a cut for using the product they spent years developing.
Now a bunch of people stirred up a lot of drama over this decision and when off and created something called OpenTerraform.
Honestly, I don’t understand two things:
- If you’ve been giving something away for free don’t get mad if someone else is using it to make money. That was your choice. Rather than throw a fit and maliciously change the software in some way as some people have done in the past, Terraform changed their license which is completely valid. They have the right to get paid for what they built.
- Open source software does not equal free source. It just means you can see the code, unless the license for it says it is free.
If you are using a free product, you get what you pay for. There are certain risks associated with free products. In fact, someone from Hashicorp came all the way to Savannah to discuss the problems with trying to keep free software up to date and secure with me. If they could get some money for supporting the use of the open source code, then that would be easier to do right?
If you are using open source code without a paid and direct license agreement you face certain risks, of which you need to be aware and handle appropriately. Using open source code is valid, but you need to use it with your eyes open.
So before you stomp off and say, “Well then, I’m going to use the free OpenTF Foundation version! Hmmfph.” Make sure you have a process to evaluate everyone who is contributing to that code and the changes they are making to ensure there’s not another Log4J scenario in that code base.
Make sure you don’t just say “Look! They encrypted the state file!” Although that is a good thing, make sure you are evaluating all the other code changes each time you deploy as well. This is a very powerful tool with a lot of access to your cloud environments and a significant attack vector for anyone who can insert a line of code with a vulnerability into the project — intentionally or maliciously.
If you use the Terraform version from Hashicorp, at least you have vendor support to some degree. Though once again — read the license and consult a lawyer to understand how much support and what recourse you have if there is a vulnerability or weakness in the software.
Also be aware that Microsoft currently offers Terraform configuration files in their documentation. Be aware of whether they continue to provide Terraform or if they provide the OpenTF version in the future and understand how that might impact your organization.
Terraform deployment server options
There are two different ways to deploy the Terraform deployment engine. You hopefully do not have developers running commands on local laptops to make changes in production. Right?! You are running a Terraform server side option that deploys your configuration files after they have been locally tested.
One is a cloud option and one is a self-hosted option.
Terraform Cloud
Terraform Enterprise
All the risks and evaluations I applied to hosted or self-hosted GitHub apply to these options. I’m not going to repeat all that here but will refer you to the considerations posed in my GitHub posts.
You could also create your own software deployment pipeline in some other way to deploy the scripts — something I’m working on in my related series to deploy resources. The same concepts used to deploy AWS resources using AWS scripts and templates would apply to terraform such as segregation of duties, MFA, deployment integrity, network access, and credentials. I’m working through the use of containers for deployments at the moment.
If you run Terraform Enterprise, make sure you understand and are able to run the system and configure all the components securely. Terraform provides information on the architecture in their documentation.
Terraform for Governance
Can Terraform help with our governance dilemma? Can you create all the governance resources I covered in the prior posts with Terraform? Let’s check that out. Sometimes Terraform provides configuration files for deployments before the cloud vendor does, though under the hood they are likely using the alternate methods of deployment I showed you when a templating option is not available from the cloud provider.
We can check the resource lists Terraform supports for each cloud provider:
Clicking view more you can see many other options:
AWS on Terraform
You’ll find a list of AWS resources you can deploy using Terraform here:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs
The coverage for AWS governance resources we covered is pretty good.
They even offer a configuration for an organization. Though that comes with a few warnings.
I find the enforcement of allowing services through the AWS console a bit odd. I’ve shown how to configure a lot of things for AWS organizations without that and if you have to use the console it’s not as easy to track your configuration to your deployment. Maybe I have not yet hit the problem child that is the underlying reason for the above warning yet.
As you can guess, if you want to use new features for this or anything else you can deploy with Terraform you’ll need to wait until Terraform supports it. You’ll also want to review what commands are run and the resulting configuration of any new or critical resource.
Take a look at the information that needs to be provided to Terraform and understand where that is stored. Consider how securely the information is stored. I see in this example for an AWS organization you can import the organization ID:
Azure on Terraform
You’ll find a list of Azure resources you can deploy using Terraform here:
https://registry.terraform.io/providers/hashicorp/azurerm/latest
I see a way to manage a tenant configuration but not to create a tenant. You need to provide pretty hefty permissions to perform this action. You would want to avoid giving this role to any Terraform deployment just because you need it for this resource. That would obviously be dangerous.
You can deploy management groups — with a caveat.
It looks like Hashicorp has done the heavy lifting to allow you to deploy a subscription. As I explained in the last post there will be three different ways to deploy it for the three different licensing options, and if you don’t have one of those, you’re out of luck.
Terraform provides a lot of options to manage Azure Policies. I didn’t look into the details but I am not sure why there are so many options here. It seems like a policy should be a universal thing you can deploy and apply in a single template in the spirit of abstraction but I’ll leave that as an exercise for the reader. 😊
I don’t see anything for configuration of an Azure Initiative.
GCP on Terraform
You’ll find a list of Google Cloud Platform (GCP) resources you can deploy using Terraform here:
https://registry.terraform.io/providers/hashicorp/google/latest/docs
I don’t see a way to create a GCP organization but that is created by default now anyway as explained in a prior post.
Terraform can create a GCP folder:
Terraform can create a GCP Project:
Terraform can create an GCP organizational policy constraint:
Terraform can also configure other types of organizational policies and settings. I didn’t cover all of these due to the time involved, but as mentioned in my GCP post, Google can be opinionated about defining the settings you can control for your organization across the board. This has some pros and cons.
Secure by default?
Now you know what Hashicorp can deploy. However, what if you only set a few required properties. Are the templates giving you resources that are secure by default? Is everything being deployed the way you specify? Are there any security gaps?
For example, you might want to check the version of the AWS Metadata deployed by your Terraform configuration files and make sure it is version 2.0. Although the CIS benchmarks only lists that as a medium risk issue, I mark it as high due to the potential damage that configuration and has caused. It’s still up to you to make sure your deployment configurations are secure and follow security best practices.
Should you use Terraform?
Are you using Terraform for deployments? Do you want to discuss the pros and cons? Come join me at the IANS Security event in Atlanta in October 2023. Sorry no vendors in my session, but contact IANS if you’re a vendor and want to present or sponsor this or a similar event.
In the next post I’ll show you another way Terraform may be able to help you keep track of cloud deployments and configurations. Bear in mind that some have complained about the time it took Terraform to implement some security features so I am not saying Terraform is your only option or recommending the product per se. I’m just pointing out what it can and cannot do and the value it offers in providing a common layer to interface with your cloud environments. Please perform your own independent security assessment.
Follow for updates.
Teri Radichel | © 2nd Sight Lab 2023
About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight LabNeed Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for PresentationFollow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab