How to deploy applications in Kubernetes via CI/CD pipeline

How to automate the process of deploying new applications to a Kubernetes cluster using GitLab CI/CD pipelines. In this blog post, you will learn how to deploy new applications to your Kubernetes cluster using a yaml template file.

Environment

We use AWS as our cloud platform. In AWS, we have deployed an EKS cluster to run our containerized applications and a container registry (ECR) to store Docker containers. For version control, we use Git as GitLab server hosted on an AWS EC2 instance.

Deployment process

The usual deployment process for an application already deployed in Kubernetes looks like this: The developer adds a feature to their application and pushes those changes directly to the application repository. Once the repository detects the changes, it automatically starts a predefined pipeline where it builds the application, creates a Docker container, and updates the image for the Kubernetes deployment.

This happens automatically and works well to update a given application already deployed in Kubernetes.

However, there is one case that is not automated at the moment: When a new application is created by a developer, the pipeline fails at the stage where it tries to update the Kubernetes deployment. Why? Because the pipeline assumes that a deployment for the application already exists in Kubernetes, and it simply tries to update the image used by the deployment.

So what needs to be done to deploy a new application to the Kubernetes cluster? At this point, the developer must contact the DevOps team and ask them to create a new deployment for their application. Specifically, the DevOps developer creates a deployment, a service, and an ingress resource for the application within the Kubernetes cluster. When creating these resources, the application developer must provide some information, such as the port that the application uses, the name of the application, the URL where the application should be accessible, and the image used by the application.

The resources in Kubernetes are almost the same for all applications. They only differ in the points that are provided by the application developer. So why not enable the application developer to deploy new applications to Kubernetes himself?

Automate the process of deploying a new application

The first thing to do is to create a yaml template file that defines all the resources needed for an application deployment. All data that need to be changed because they vary with each application are replaced with placeholders. In my case it was name, namespace and port. I replaced them with placeholders like {NAME}, {NAMESPACE}, {PORT} and {IMAGE} .

Here are the main parts of the yaml template file:

apiVersion: apps/v1
  kind: Deployment
  metadata:
    ...
    name: template-ms
    namespace: eco-staging
  spec:
    replicas: 2
    revisionHistoryLimit: 10
    strategy:
      rollingUpdate:
        maxSurge: 25%
        maxUnavailable: 25%
      type: RollingUpdate
    template:
      ...
      spec:
        ...
        containers:
          ...
          image: {IMAGE}
          imagePullPolicy: IfNotPresent
          ...
          name: {NAME}
          ports:
          - containerPort: {PORT}
            name: {NAME}
            protocol: TCP
          ...
  ---
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
    name: {NAME}-api
    namespace: {NAMESPACE}
  spec:
    rules:
    ...
        - backend:
            service:
              name: {NAME}
              port:
                number: {PORT}
          path: /{NAME}
          pathType: ImplementationSpecific
    ...
  ---
  apiVersion: v1
  kind: Service
  metadata:
    labels:
      app: {NAME}
    name: {NAME}
    namespace: {NAMESPACE}
  spec:
    ports:
    - name: {NAME}
      port: {PORT}
      protocol: TCP
      targetPort: {PORT}
    selector:
      app: {NAME}
    sessionAffinity: None
    type: ClusterIP

After this Yaml template file is created, we need to make it accessible to the pipeline. And there are some requirements for that:

There is one pipeline for all applications All applications use the same pipeline. This keeps maintenance simple. The goal is to keep this architecture and not create a dedicated pipeline for each application.
The yaml template file should be stored in a central place It would be easy to add this yaml template file to any repository to apply it to Kubernetes. But it is never a good idea to repeat code over and over again. Therefore, we need a central place to store the yaml template file that is easily accessible to each application pipeline.
The yaml template file should not be modified by any application or pipeline run The yaml template file itself should only be used as a reference and should not be changed.

With these requirements and the given environment in mind, it is best to upload this yaml template file as a configmap into the Kubernetes cluster. Use this kubectl command to create a Kubernetes configmap from the yaml template file:

kubectl -n <namespace-name> create configmap template-app --from-file=template=template.yaml

This command puts the contents of the template.yaml file into a configmap. To access the contents from your Kubernetes cluster, simply type:

kubectl -n <namespace-name> get configmap template-app -o jsonpath='{.data.template}'

The output should be the same as the contents of the template.yaml file.

Create a CI/CD pipeline

Now that the template.yaml is stored inside your Kubernetes cluster and reachable for every app pipeline start to create a stage that will create a deployment for the app.

stages:
    - build
    - package
    - deploy_first
    - deploy

...

deploy_first:
  image: $KUBECTL_IMAGE
  stage: deploy_first
  rules:
    - if: $FIRST_DEPLOY == "True"
  before_script:
    - curl -o /tmp/aws-iam-authenticator <aws-iam-authenticator-url>
    - export PATH=$PATH:/tmp && chmod +x /tmp/aws-iam-authenticator
    - echo ${KUBE_CONFIG} | base64 -di > $KUBECONFIG
  script:
    - kubectl --kubeconfig $KUBECONFIG -n $K8S_NAMESPACE get cm template-app -o jsonpath='{.data.template}' > /tmp/deploy.yaml
    - sed -i "s/{NAME}/$NAME/g" /tmp/deploy.yaml
    - sed -i "s/{PORT}/$PORT/g" /tmp/deploy.yaml
    - sed -i "s/{NAMESPACE}/$K8S_STAGE_NAMESPACE/g" /tmp/deploy.yaml
    - sed -i "s/{IMAGE}/$REPOSITORY_URL\/$IMAGE:$IMAGE_TAG/g" /tmp/deploy.yaml
    - cat /tmp/deploy.yaml
    - kubectl --kubeconfig $KUBECONFIG -n $K8S_NAMESPACE apply -f /tmp/deploy.yaml

deploy:
  image: $KUBECTL_IMAGE
  stage: deploy
  rules:
    - if: $FIRST_DEPLOY != "True"
  before_script:
    - curl -o /tmp/aws-iam-authenticator <aws-iam-authenticator-url>
    - export PATH=$PATH:/tmp && chmod +x /tmp/aws-iam-authenticator
    - echo ${KUBE_CONFIG} | base64 -di > $KUBECONFIG
  script:
    - kubectl --kubeconfig $KUBECONFIG -n $K8S_NAMESPACE deployment/$K8S_DEPLOY set image $IMAGE=$REPOSITORY_URL/$IMAGE:$IMAGE_TAG
    - kubectl --kubeconfig $KUBECONFIG -n $K8S_NAMESPACE rollout status deployment/$K8S_DEPLOY

The pipeline contains 4 stages:

build This stages builds the application.
package In this phase, a Docker container is created that contains the created application and is used to run the application within the Kubernetes cluster.
deploy_first This phase is what this article is about. Here, the template.yaml is used to create a deployment for the application.
deploy In this phase, an existing deployment is assumed and the image used by the container is updated with the newly created image.

The deploy_first stage

Some explainations about the deploy_first stage:

$KUBECTL_IMAGE: Is a Docker container with kubectl pre-installed.

To connect to the Kubernetes cluster in AWS, authentication is required. This is basically what the before_script does. It authenticates against AWS and connects to the Kubernetes cluster so that the pipeline can execute kubectl commands.

$FIRST_DEPLOY is a environment variable that has to be set by the application developer. Whenever he deploys the application for the first time, he sets it to True and then the deploy_first phase is executed. Otherwise this step is skipped.

The script section pulls the template.yaml from the previously created configmap and saves it to /tmp/deploy.yaml. Than it uses sed to replace the placeholders with the actual values. Once all the placeholders are replaced, the file is applied to the Kubernetes cluster.

The deploy stage

If the deployment already exists and does not need to be recreated, the pipeline can be run without the $FIRST_DEPLOY variable. Then, the deploy_first stage is skipped and only the deploy stage is executed. In this phase, only the deployment container image is updated.

Conclusion

With this solution, you can automate the deployment of your applications to a Kubernetes cluster. This saves a lot of time for DevOps. With a central yaml template file, this solution is also easy to maintain.