Kubernetes CI/CD with CircleCI and ArgoCD

An end-to-end setup for declaratively deploying applications to Kubernetes using GitOps principles.

One of the top DevOps trends in recent years has been the adoption of GitOps to manage Infrastructure as Code (IaC) and cloud-native applications on Kubernetes. GitOps was first introduced by WeaveWorks as a way to declaratively configure and manage infrastructure and application via Git. While GitOps is an opinionated approach to the CI/CD process, but the underlying ideas behind IaC and the convergence properties of Kubernetes mesh well with GitOps. In this post, we will look at a demo setup of an end-to-end CI/CD pipeline and discuss some important considerations based on your team size, roles, and needs.

Note: This guide uses CircleCI and ArgoCD, but the tools can be interchanged to establish a GitOps-driven CI/CD pipeline (e.g. JenkinsX, Spinnaker)

End-to-End CI/CD Architecture

At a high-level, the CI/CD pipeline will cover the following steps:

1. Developer commits code to a Git repo (e.g. Github/Bitbucket/Gitlab).
2. CircleCI runs through the CI workflow steps (i.e. security audit, linting, testing, etc).
3. A Docker container is created and pushed to a container registry (e.g. Docker Hub, GCR, ECR).
4. Kubernetes manifests are generated and committed to a separate repo that ArgoCD monitors.
5. ArgoCD detects changes and deploys the new application/infrastructure to the configured Kubernetes clusters.

CI Considerations

To keep this guide generic to apply to other CI tools, instead of focusing on CircleCI-specific syntax, I’ll lay out some important considerations for the CI pipeline to accommodate GitOps CD operations. For a concrete example of using CircleCI, you can follow this excellent guide by David-VTUK.

The first consideration for your CI pipeline is how to separate the source/application code and infrastructure. ArgoCD recommends keeping two separate repos: one for app code and another for manifests. The benefits of using separate Git repositories are:

1. There is a clear separation between application and configuration code. This prevents the entire CI build from running, which could take a long time, when you wish to only modify the manifests (e.g. adding a new env variable or changing a configmap).
2. If you have a separate Production Engineering/SRE team responsible for deployments, you can separate access and have a cleaner audit log.
3. Some applications may be deployed as a single unit or have dependent components (e.g. ELK stack, Prometheus + Grafana). A separate configuration repo allows multiple application repos to push changes independently, but allow the CD tool to monitor a single repo and deploy all the components at the same time.

The exact point of separation, however, depends on the makeup of your team and the tools used to package Kubernetes manifests (e.g. Helm, Kustomize, Ksonnet, etc). At Leverege, we don’t have a dedicated SRE team in charge of deployments. Every developer is responsible for creating the Helm templates using an internal tool to bootstrap the manifests based on local config files. In our case, CircleCI builds the Docker image, packages the Helm file, and pushes it to our private ChartMuseum instance. Another mono-repo stores values.yaml files per environment/cluster and triggers the deployment process. A larger team with dedicated SREs may elect to track config files in dedicated repositories per team or product with Helm files also living in that repo.

Finally, even though one of the goals of GitOps is immutability, it is important to leave some freedom for imperative changes. On Kubernetes, this usually has to do with dynamic values (i.e. number of replicas for Horizontal Autoscaler or resource requests). For example, if you are relying on the Vertical Pod Autoscaler to dynamically change CPU/Memory requests and limits, don’t track those fields via Git or applications will constantly be out of sync. This is similar to ignoring the desired node count in Terraform to allow the Cluster Autoscaler to respond to Kubernetes workloads.

Setting Up ArgoCD

ArgoCD is a popular cloud-native CD tool used by Intuit, New Relic, NVIDIA, and more. With Flux CD from WeaveWorks joining forces with ArgoCD last year, ArgoCD became a great tool for users who don’t need all the robust features of Spinnaker.

Installation

Installing ArgoCD is very simple. Use the manifests on Github or the Helm chart maintained by the community:

$ kubectl create namespace argocd 
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

On GKE, an additional admin role is required to create new cluster roles:

$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user="$(gcloud config get-value account)"

To interact with the ArgoCD server, you need to first download the CLI (use brew or download directly from GitHub):

$ brew install argocd

Initial Setup

ArgoCD server exposes both gRPC (for CLI) and HTTPS (for UI). To access the server, you can port-forward or configure ArgoCD with an ingress:

$ kubectl port-forward svc/argocd-server -n argocd 8080:443

Since we use Traefik with Cert Manager, we deployed the following IngressRoute and also added the insecure flag on the argocd-server deployment (IMPORTANT: if you use Traefik, note the initial pod name of the argocd-server before updating the deployment since that’s the default password):

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: argocd-server
  namespace: argocd
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`argocd.example.com`)
      priority: 10
      services:
        - name: argocd-server
          port: 80
    - kind: Rule
      match: Host(`argocd.example.com`) && Headers(`Content-Type`, `application/grpc`)
      priority: 11
      services:
        - name: argocd-server
          port: 80
          scheme: h2c
  tls:
    secretName: <my-wildcard-cert>

Note: ArgoCD also supports ingress-nginx and AWS ALB/ELB

Once the server is exposed via Ingress or forwarded to localhost, grab the autogenerated password:

$ kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2

And login with username admin (if your ingress doesn’t support HTTPS/gRPC, add the following flag --grpc-web ):

$ argocd login <server>

Finally update the default password via:

$ argocd account update-password

Adding a Cluster

Since ArgoCD is most likely installed on a utility cluster (at Leverege, it runs on the same cluster that hosts Vault, Chart Museum, and Prometheus/Grafana), ArgoCD first needs access to external clusters. ArgoCD installs a service account (argocd-manager) into kube-system namespace and binds to an admin ClusterRole:

$ argocd cluster add <context-name> 

For internal clusters, the Kubernetes API server address can be set to https://kubernetes.default.svc without having to bind the service account. By repeating this step, you can add all the clusters that ArgoCD will have access to deploy. Currently, multi-cluster deployment (i.e. deploying the same application to multiple clusters in one pipeline) is not yet supported but is in active development under ApplicationSet CRD.

Connect to a Git Repo

Unless the repo containing the manifests is public, ArgoCD needs to first authenticate with the private repository.

Generate the token for your Git hosting service:

• Github
• Gitlab
• Bitbucket

Then add the repo via CLI:

$ argocd repo add https://github.com/argoproj/argocd-example-apps --username <username> --password <token>

Or via UI: Settings/Repositories > Connect Repo using HTTPS

Create an ArgoCD Application

Finally, we can deploy our application. If you don’t have an application or Git repo ready for test, you can choose from ArgoCD’s example apps:

• Plain YAML
• Ksonnet
• Helm
• Jsonnet
• Kustomize

To start, from the UI, click on New App :

Give the app a name, select the Default project, and choose a sync policy (either automatic or manual). For production clusters, you can also create a separate project (via Settings > Projects ) to limit access to cluster resources and namespaces.

Next, configure the Source with Git repository URL and path as well as the Destination to choose the cluster and namespace to deploy into.

Once the application is configured, the application is ready to be synced (deployed). From the UI, click on Sync and ArgoCD will apply the changes to the cluster.

When the sync is successful, the application will change its status from OutOfSync to Healthy :

If you prefer a video walkthrough, you can watch a quick 10 min demo (between 1:04 and 9:57):

Other Considerations

Even though CircleCI and ArgoCD provide a ton of functionality for your CI/CD pipeline, as with any software, there are a few other considerations to discuss with your team as usage grows:

1. Don’t rely on deployment failure to validate the manifests. At the very least, test the configuration code during the CI process using linting tools (e.g. helm lint ). Developers can also manually validate the correct values using tools such as helm template .
2. Secret management is tricky when it comes to GitOps. ArgoCD works with most popular solutions (e.g. Bitnami Sealed Secrets, Godaddy Kubernetes External Secrets, External Secrets Operator, Hashicorp Vault, Banzai Cloud Bank-Vaults, Helm Secrets, Kustomize secret generator plugins, aws-secret-operator, KSOPS) with some community-driven examples listed on this Github discussion.
3. As the number of apps grows, it becomes a challenge to bootstrap a new cluster. ArgoCD presents a solution via apps of apps pattern where one ArgoCD application encompasses child apps. If there’s a logical grouping, this pattern works great. If not, until multi-cluster deployment is supported, you’ll have to write some scripts to programmatically create pipelines via the CLI.
4. Finally, don’t forget to back up ArgoCD data. Even though all the configuration is checked into Git, it’ll be easier to import from a backup then run through all the pipelines to recreate if ArgoCD goes down.