How to use secrets from Azure Key Vault in Azure Kubernetes Service

Secure way of using secrets from Azure Key Vault in AKS cluster

Managing secrets and using secrets in the Kubernetes environment is a very important security aspect. It is very essential for DevOps engineers to properly define an ecosystem for handling secrets during application deployment.

In this article, we will discuss a secure way of using secrets that are stored in Azure Key Vault into your Azure Kubernetes Cluster(AKS). We will talk about different approaches but will primarily focus on the Secret Store CSI Driver.

The Scenario: You have to deploy an application that needs to connect to database.. hmm, so you need DB user name and password and it has to be securely stored.

Standard Approach

We can inject sensitive data such as usernames and passwords into pods using the secret object. Let’s take an example, we have db_username as iamdbadmin db_passwrod as supersecret. First, we need to convert these variables into base64 encoded format.

echo -n 'iamdbadmin' | base64
echo -n 'supersecret' | base64

Then we need to create a configuration file to create a secret

cat <<EOF> db-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-secret
data:
  username: aWFtZGJhZG1pbg==
  password: c3VwZXJzZWNyZXQ=
EOF

Create the secret

kubectl apply -f db-secret.yaml

Now we can create a resource, which is using secret values in env

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  containers:
  - name: nginx-container
    image: nginx
    env:
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: db-secret
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-secret
          key: password

Or we can use envFrom to use all the secret’s data in container environment variables.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  containers:
  - name: nginx-container
    image: nginx
    envFrom:
    - secretRef:
        name: db-secret

Well, this is simple! But project scenarios are not so simple. Secrets are stored in some sort of a digital vault, and then the DevOps team has to take care of making it available for the applications. To implement all the security best practices we should use Pod identity to access secrets.

Security best practice approach

Securing private data is a top priority for DevSecOps practitioners. Azure Key Vault(AKV) is a very good solution to store keys, secrets, and certificates. Once we store secrets in AKV we also need a proper mechanism to use them in our applications.

If we are using Azure DevOps then we can create a secret object in the release stage. We need to create a service connection that can connect with AKV to pull the secrets during pipeline execution.

In this article, we will discuss another approach that can directly get secret content from AKV and mount in the Pods.

Azure Key Vault Provider for Secrets Store CSI Driver

Azure Key Vault provider for Secret Store CSI Driver allows us to get secrets from AKV and mounts them in the Pods or sync them in the secret object.

1. Installation: It is very important to use the recommended Kubernetes version ( v1.16.0+) otherwise this driver will not work.

We will do helm deployment using the following commands

helm repo add secrets-store-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts

helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver -n azure-akv

Verify this installation using kubectl get po -n azure-akv . We should see pods for secret store CSI driver

2. Setup Azure Provider using kubectl apply

kubectl apply -f https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/deployment/provider-azure-installer.yaml --namespace azure-akv

Verify usingkubectl get po -n azure-akv , we should see the following output

3. Create an Azure Key Vault Resource — We can use the Azure portal to create AKV. Please refer to this tutorial to use the Azure portal for AKV. We name this resource as db-keys

4. Create SecretProviderClass object — This custom resource will provide the required parameters for Secret Store CSI Driver. Update resourceGroup, subscriptionId and tenantId in azure-kv-provider.yaml .

5. Provide Identity to access KeyVault — there are 4 modes for accessing key vault

• Service Principal
• Pod Identity
• VMSS User Assigned Managed Identity
• VMSS System Assigned Managed Identity

In this article, we will use the 1st option of using the Service Principal.

If we have the Service Principle that can access KeyVault then we don’t need to create a new SP. We just have to make sure the SP has permission to read secrets and keys. We can use the following commands to create SP if we don’t have one already.

# Create SP
az ad sp create-for-rbac --skip-assignment --name $SP_NAME

# Get client id of SP
SP_CLIENT_ID=$(az ad sp show --id $SP_NAME --query appId -o tsv)

# Role Assignment for AKV
az role assignment create --role Reader --assignee $SP_CLIENT_ID --scope /subscriptions/$SUBID/resourcegroups/$KEYVAULT_RESOURCE_GROUP/providers/Microsoft.KeyVault/vaults/$KEYVAULT_NAME

# Set permissions to read secrets
az keyvault set-policy -n $KEYVAULT_NAME --secret-permissions get --spn $SP_CLIENT_ID

Now we can generate a secret with SP_CLIENT_ID and SP_CLIENT_SECRET This secret is used by pods to get secret from AKV

kubectl create secret generic akv-creds -n azure-akv --from-literal clientid=$SP_CLIENT_ID --from-literal clientsecret=$SP_CLIENT_SECRET

6. Deploying the sample application — Finally, we can deploy a sample application to verify the setup.

kubectl apply -f sample-app.yaml

Wait for the pod to start, once it is running we can verify mounted content at the specified path.

#Show secrets
kubectl exec -it nginx-secrets-store ls /mnt/secrets-store/

# Verify secrets mentioned in provider class
kubectl exec -it nginx-secrets-store cat /mnt/secrets-store/db-username

kubectl exec -it nginx-secrets-store cat /mnt/secrets-store/db-password

What Next?

There are few good examples in the repository. You can try out these examples to understand this plugin in a bit more detail.

Also, go through Azure’s best practice recommendation for Pod security.

Stay tuned for my next blog on using Pod identity for accessing KeyValut. Please share your comments or questions in the feedback section.

Subscribe to FAUN topics and get your weekly curated email of the must-read tech stories, news, and tutorials 🗞️

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇