Summary

The article proposes a configuration-based approach using Terraform resource factories and YaML/JSON configurations to enable non-Terraform-enabled teams to contribute to IaC codebase and parcel out the repetitive creation of specific resources.

Abstract

As Terraform codebases grow, managing and enabling contributions can be challenging. Traditional approaches like naive resource definition and wrapping logic in Terraform modules with variables have limitations. The article proposes a descriptive approach using Terraform resource factories and YaML/JSON configurations, which enables non-Terraform-enabled teams to contribute and simplifies the management of large and complex infrastructures.

Opinions

The article highlights the limitations of traditional approaches, such as naive resource definition and using Terraform modules with variables.
The proposed descriptive approach using Terraform resource factories and YaML/JSON configurations is battle-tested and suitable for large and complex codebases.
The article emphasizes the benefits of using YaML over JSON and CSV, as it is easier to parse for humans and allows for comments and nested, complex structures.
The article suggests implementing GitOps strategies for reviewing commits for correctness and compliance and applying infrastructure updates.
The article recommends trying out an AI service, ZAI.chat, as a cost-

Resource Factories: A descriptive approach to Terraform

“Wolfsburg — Volkswagen Assembly Line” by roger4336 is licensed under CC BY-SA 2.0

Preamble

As your Terraform codebase grows in size and reach, you might find yourself in one of these common scenarios:

enabling contributions from specific teams (e.g. NetOps to create new subnets or firewall rules, or the Logging and Monitoring team to configure new alerts) can be difficult since Terraform knowledge is not widespread across teams
the repetitive creation of resources of a given kind (e.g. firewall rules, subnets, projects) is scattered throughout your codebase, making it complex to get a clear picture of their structure
sticking to a monolithic, end-to-end approach for a large infrastructure (i.e. describing your whole infrastructure in a large module/state) makes for a slower, less maintainable codebase, which doesn’t mirror the different speeds at which your infra evolves (think core infrastructure vs firewall rules)

This article proposes a configuration-based approach that enables parceling out the repetitive creation of specific resources in distinct repositories and enabling non-Terraform-enabled teams to contribute to your IaC codebase.

Traditional approaches

Let’s see an example of how we would handle the creation of multiple resources of the same kind using solutions within the traditional spectrum.

Naive definition using the provider’s built-in resources. Doesn’t solve the challenge of enabling different teams to contribute to the codebase, can creep into long and hardly readable code-walls when resource count gets into the tens or the hundreds and is generally unsuitable for production usage.

Wrap logic in a Terraform module and leverage Terraform variables. This approach allows for the separation of code and configuration, allowing non-core teams to contribute code to a specific repository, and providing them a much easier interface to work with (configuration vs Terraform code). However as resource count grows, your variables rapidly become harder to parse and make sense of, and a single typo can lead to hard-to-troubleshoot mistakes.

Descriptive approach

Wrap logic in a Terraform resource factory, and leverage YaML/JSON configurations, one per resource. This is a battle-tested approach we are using on large and complex codebases.

A resource factory is an opinionated, purpose-built module that

implements specific requirements and best practices (e.g. “always enable PGA for Google Cloud VPC subnets”, or “only allow using Google Cloud regions europe-west1 and europe-west3”)
codifies business logics and policies (e.g. labels and naming conventions)
standardizes, automates and centralizes the repetitive creation of resources of a given kind

Our approach is based on modules implementing the factory logic using Terraform code and a set of directories having a well-defined, semantic structure, holding the configuration for the resources in YaML syntax.

Terraform natively supports YaML, JSON and CSV parsing — a few observations:

YaML is easier to parse for a human, and allows for comments and nested, complex structures
JSON and CSV can’t include comments, which can be used to document configurations, but are often useful to bridge from other systems in automated pipelines
JSON is more verbose (reads: longer) and harder to parse for a human
CSV isn’t often expressive enough (e.g. doesn’t allow for nested structures)

Tl;dr, if you’re editing files by hand, YaML is probably your best choice.

With that in mind, let’s see an example of this approach using thesubnets module available in the Cloud Foundation Fabric in the Resource Factories section.

The module is conceived so that it can be used by simply pointing it to our configuration folder

module "subnets" {
  source        = "./cloud-foundation-fabric/factories/subnets"
  config_folder = "./subnets"
}

… uses a fixed data directory structure

└── subnets             # Configuration folder entry point
  ├── project-ada       # Project ID the VPC belongs to 
  │ ├── vpc-alpha 
  │ │ ├── subnet-a.yaml # Subnet name 
  │ │ └── subnet-b.yaml 
  │ └── vpc-beta 
  │   └── subnet-c.yaml 
  └── project-bob
    └── vpc-gamma
      └── subnet-d.yaml

…and YaML files for the actual configuration, with one file per subnet

# subnets/project-ada/vpc-alpha/subnet-a.yaml 
region: europe-west1 
description: Frontend 
ip_cidr_range: 10.0.0.0/24 
secondary_ip_ranges: 
  secondary-range-a: 192.168.0.0/24
  secondary-range-b: 192.168.128.0/24

# subnets/project-ada/vpc-alpha/subnet-b.yaml 
region: europe-west1 
description: Backend 
ip_cidr_range: 10.0.1.0/24 
# Assign roles/compute.networkUser 
iam_groups: ["[email protected]"]

The folder structure supplies many of the required attributes for each subnet (project id, VPC name, subnet name), reducing the potential for mistakes, and making the network structure explicit and easy to navigate even from a repository browser view, while the YaML file structure is essential and easy to understand and maintain for non-terraform-savvy operators, enabling them to contribute to your IaC codebase.

GitOps strategies can then be implemented to review every suitable commit for correctness (e.g. YaML syntax, terraform plan outputs) and compliance (e.g. allow only certain users/groups to commit under given folders), and to finally terraform apply the resulting infrastructure updates.

This is just an example of how resource factories can be handy when managing large and complex infrastructures. Besides subnets, our code repository has a growing number of other factories (hierarchical firewall policies, VPC firewall rules) you can take inspiration from, or directly grab and adapt to your needs.