Terraform Provisioning EKS Cluster with Additional Node Groups

Terraform is a popular provisioning tool for building, updating, and versioning infrastructure safely and efficiently. It is also well-known technology of Infrastructure-as-Code (IaC).

A common scenario in the work of cloud engineers is to provision and update Kubernetes clusters. So, the following points are covered in this post with Terraform.

• Provisioning an EKS cluster
• Provisioning an additional node group in the EKS cluster
• Tainting nodes in the node group

Provisioning an EKS cluster

Terraform provides a good tutorial about how to providsion an EKS cluster: https://learn.hashicorp.com/tutorials/terraform/eks

Therefore, the github repo is good enough for start point.

Clone the Repo

Clone the repo of learn-terraform-provision-eks-cluster to local machine.

> git clone git@github.com:hashicorp/learn-terraform-provision-eks-cluster.git

Update AWS Info

In this repo, all the config values is preset and you have to put the right values per your AWS account information.

VPC

In the vpc.tf file, update region, provider and subnets with your preferred values.

If you want to provision an EKS cluster by assuming an IAM role, then specify the role_arn in the provider section.

Here is my case:

provider "aws" {
  region = "us-east-1"
  assume_role {
    role_arn = "arn:aws:iam::xxxx4338xxxx:role/ClusterAdminRole"
  }
}

In the vpc module, update CIDR and subnets for the EKS VPC.

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.66.0"

name                 = "rocky-dev-eks-vpc"
  cidr                 = "10.17.40.0/22"
  azs                  = data.aws_availability_zones.available.names
  private_subnets      = ["10.17.41.0/24", "10.17.42.0/24", "10.17.43.0/24"]
  public_subnets       = ["10.17.40.64/26", "10.17.40.128/26", "10.17.40.192/26"]
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
  }

public_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = "1"
  }

private_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = "1"
  }
}

Provisioning an additional node group in the EKS cluster

Node group is a set of EC2 instances with the same type. An EKS cluster may contains multiple node groups with different instance types.

In this tutorial repo, all EKS-related settings are included in the file eks-cluster.tf. By default, it contains two node groups.

worker_groups = [
    {
      name                          = "worker-group-1"
      instance_type                 = "t2.small"
      additional_userdata           = "echo foo bar"
      asg_desired_capacity          = 2
      additional_security_group_ids = [aws_security_group.worker_group_mgmt_one.id]
    },
    {
      name                          = "worker-group-2"
      instance_type                 = "t2.medium"
      additional_userdata           = "echo foo bar"
      additional_security_group_ids = [aws_security_group.worker_group_mgmt_two.id]
      asg_desired_capacity          = 1
    },
  ]

These two node groups contains different instance types, one uses t2.small and the other uses t2.medium.

Ideally, we hope Kubernetes schedules resource-consuming Pods to the nodes with powerful CPU or memory and schedule other Pods to the normal nodes. However, the node group setting above is not enough since Kubernetes has no idea about the strategy.

Therefore, Kubernetes treats those nodes in the both node groups as same and schedules Pods onto them with no difference.

In order to achieve the scheduling strategy mentioned above, tainting nodes is needed.

Tainting nodes in the node group

Taint is a Kubernetes concept which works together with toleration.

Taint

When a node is tainted, it repels all Pods from being scheduled onto it by Kubernetes except for those Pods with corresponding tolerations.

Toleration

Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints.

Node Group with Tainted Nodes

Imagine this, there is a group of powerful nodes and all of them are tainted. So, the Pods without tolerations cannot be scheduled onto them.

For those resource-consuming Pods, we can put tolerations of the corresponding taint, and then they can only be scheduled onto those tainted powerful node in the group.

For more details about taints and tolerations, please refer to https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/.

Taints in Terraform

How can we implement tainting nodes with Terraform? It is a bit tricky by providing kubelet arguments like:

--register-with-taints app=worker-3:NoSchedule

Then, a taint is added to the node spec:

taints:
  - effect: NoSchedule
    key: app
    value: worker-3

For the Pods they want to be scheduled onto those tainted node, they should include the following tolerations in their spec:

tolerations:
- key: "app"
  operator: "Equal"
  value: "worker-3"
  effect: "NoSchedule"

And here is my definition of node groups in the eks-cluster.tf :

worker_groups = [
    {
      name                          = "worker-group-1"
      instance_type                 = "t2.small"
      additional_userdata           = "echo foo bar"
      asg_desired_capacity          = 2
      additional_security_group_ids = [aws_security_group.worker_group_mgmt_one.id]
    },
    {
      name                          = "worker-group-3"
      instance_type                 = "t2.large"
      additional_userdata           = "echo foo bar"
      additional_security_group_ids = [aws_security_group.worker_group_mgmt_three.id]
      asg_desired_capacity          = 1
      kubelet_extra_args            = "--node-labels=node_group=worker-group-3 --register-with-taints app=worker-3:NoSchedule"
    },
  ]

Apply EKS Cluster with Terraform

Apply Resource

Once everything is ready, run the following commands to apply the EKS cluster:

> terraform init

> terraform apply

Generate Kubectl Configuration

If you run terraform with your AWS user account, it would be easy to retrieve the access credentials for your cluster and automatically configure kubectl.

> aws eks --region $(terraform output -raw region) update-kubeconfig --name $(terraform output -raw cluster_name)

However, if you run terraform with assuming a AWS Role like me, it needs an extra step for configuring kubectl.

• Add the assuming role to AWS config file ($HOME/.aws/config)

[profile dev-terraform]
role_arn = arn:aws:iam::xxxx4338xxxx:role/ClusterAdminRole
source_profile = default
output = json
region = us-east-1

Append the section above to the file.

• Run update-kubeconfig CLI to configure kubectl.

> aws eks --region $(terraform output -raw region) update-kubeconfig --name $(terraform output -raw cluster_name) --profile dev-terraform

Connect to the EKS Cluster

Now, your default Kubernetes config is pointing to the EKS cluster just created. Let’s run a kubectl command directly.

> kubectl get node
NAME                           STATUS   ROLES    AGE     VERSION
ip-10-17-41-172.ec2.internal   Ready    <none>   7d20h   v1.20.4-eks-6b7464
ip-10-17-42-55.ec2.internal    Ready    <none>   7d20h   v1.20.4-eks-6b7464
ip-10-17-42-81.ec2.internal    Ready    <none>   7d20h   v1.20.4-eks-6b7464