CODEX

Setup Kubernetes Cluster Multi-tenancy with AWS EKS

0. What is multi-tenancy?

A multi-tenant cluster is shared by multiple users and/or workloads which are referred to as “tenants”. The operators of multi-tenant clusters must isolate tenants from each other to minimize the damage that a compromised or malicious tenant can do to the cluster and other tenants. Also, cluster resources must be fairly allocated among tenants.

There are many articles discussing multi-tenancy on Kubernetes clusters. Typically, Kubernetes Namespace is used for setting up multi-tenancy in Kubernetes clusters. The biggest benefit here is that Namespace is a native scope or border for most Kubernetes resources, so it would be easy for Cloud Ops to manage and organize the cluster.

For example, I want to create a namespace for testing team so they are able to access their resources via Kubectl in the Kubernetes cluster including Deployments, Services, Pods, etc.

Let’s assign the Namespace testing-services as the tenant namespace which is assigned to testing team, so they can access the Kubernetes resources in this Namespace.

The follow steps focus on setting up the tenant environment in the Kubernetes cluster managed by AWS, known as AWS EKS.

1. Setup Namespace

The namespace is created as a normal space and it would be used for hosting the tenant resources. Here is the yaml file for testing-services namespace.

Apply the yaml file and check if the namespace created successfully.

$ kubectl get namespace
NAME                      STATUS   AGE
...
default                   Active   39m
kube-system               Active   39m
testing-services          Active   16s

2. Setup RBAC Rules

In order to limit the access only for the specific namespace in the Kubernetes cluster, a Role needs to be created to define what operations are allowed to be executed. Then, a RoleBinding is also needed to connect the Role and the Kubernetes User who operates the resource indeed.

Setup Role

In this post, I created a Role of testing-services-admin-role which has fully permission to operate the resources hosted in the namespace of testing-services.

Setup RoleBinding

In the same yaml file, a RoleBinding is also created under the namespace testing-services. It connects the Kubernetes User testing-services-admin-user to the Role testing-services-admin-role that means the Role is assigned to the user. Therefore, the User testing-services-admin-user has the permission defined by the Role to access the Namespace testing-services.

Then, what is the Kubernetes User testing-services-admin-user? Where does it get defined? It would be introduced in the latter section.

3. Setup Tenant Admin Role in AWS

In order to manage permission easily, we create an AWS IAM Role for the testing team with the permission to access the testing-services Namespace.

Please note: this AWS IAM Role should be created in the same account with the AWS Kubernetes cluster (EKS) in case the team members are hosted in another AWS account. If your team members locate in the same account with your AWS EKS, then feel free to ignore this.

AWS IAM Role: TestingServicesTenantAdmin

An AWS IAM Role TestingServicesTenantAdmin is created in the same account of the Kubernetes (EKS) cluster. It is used as the Kubernete User testing-services-admin-user.

An IAM Role is not enough and we also need to provide a limitation for it: a Policy. That is the attached policy: EKSReadOnly.

Attached Policy: EKSReadOnly

This policy allows the Role TestingServicesTenantAdmin to be assumed and the read operations to the EKS clusters.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::account:role/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "eks:DescribeCluster",
                "eks:ListClusters"
            ],
            "Resource": "*"
        }
    ]
}

4. Group Permission to Assume Tenant Admin Role

Putting team members into an IAM Group is a convenient way to manage permissions for a team. Once we have the Group for our testing team, I would add a group policy to allow it to assume the IAM Role TestingServicesTenantAdmin and then all the group member are able to assume that Role as well.

IAM Group: TestingServicesTeam

It is simple to create a Group from AWS IAM console page by providing a Group name and leaving attached policy empty.

Let me create an IAM Group TestingServicesTeam for testing service team and add myself to the members.

IAM Group Policy

A Group Policy needs to be attached to the Group TestingServicesTeam as Inline Policies.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt_id",
      "Effect": "Allow",
      "Action": [
        "sts:AssumeRole"
      ],
      "Resource": [
        "arn:aws:iam::account_id:role/TestingServicesTenantAdmin"
      ]
    }
  ]
}

5. Map Tenant Admin Role to K8s Cluster User

Now, the settings are ready on both Kubernetes cluster side and AWS IAM side.

The settings on Kubernetes side:

• Kubernetes Namespace testing-services for the tenant
• Kubernetes Role testing-services-admin-role for the tenant admin permission
• Kubernetes RoleBinding for binding the Kubernetes Role to the Kubernetes User testing-services-admin-user

The setting on AWS IAM side:

• IAM Role TestingServicesTenantAdmin for the tenant admin permission
• IAM Group TestingServicesTeam for the testing services team

Now, we missed the last connection — between the Kubernetes User testing-services-admin-user and IAM Role TestingServicesTenantAdmin.

In AWS, the users or IAM Roles for EKS cluster are managed in aws-auth ConfigMap in the kube-system Namespace.

Update ConfigMap aws-auth

To grant additional AWS users or roles the ability to interact with your cluster, you must edit the aws-auth ConfigMap within Kubernetes.

Run the following command to check the current aws-auth ConfigMap:

kubectl describe configmap -n kube-system aws-auth

In the Data section, there should have a map: mapRoles.

The connection between the Kubernetes User testing-services-admin-user and IAM Role TestingServicesTenantAdmin can be added here, like:

- rolearn: arn:aws:iam::account_id:role/TestingServicesTenantAdmin
  username: testing-services-admin-user

More details could be found here.

6. Setup Local K8s Config

The last step is to enable testing services team members to be able to run kubectl command against the tenant in the Kubernetes cluster on their local machines. I would assume team members are using MacOS.

First, kubectl should have been installed on their local environment.

aws-iam-authenticator

aws-iam-authenticator is needed and should be installed if not:

brew install aws-iam-authenticator

Update Local AWS Configuration File: $HOME/.aws/config

Add a new profile section to $HOME/.aws/config:

[profile testing-services-tenant] role_arn = arn:aws:iam::account_id:role/TestingServicesTenantAdmin
region = your-region
source_profile = default

Update Local Kubernetes Config File

aws eks --profile testing-services-tenant update-kubeconfig --name your-eks-cluster-name --role-arn arn:aws:iam::account_id:role/TestingServicesTenantAdmin

7. Verify with Kubectl Commands

The simple way to verify the team members able to access the tenant Namespace testing-services is to run a simple kubectl command:

kubectl get pods -n testing-services

If everything goes well, a normal output message shows:

No resources found in testing-services namespace.

It means no Pods found in the Namespace testing-services since it was just setup.

8. End

The most important part is to assign proper permission to the tenant team members, which depends on the real situation.

Another challenge is to make the entire setting process automated. In this post, all steps are manual for demo purpose but it is not a good practice.