Using CoreDNS on GKE

Overcoming kube-dns restrictions on GKE via stub domains

While CoreDNS reached GA for Kubernetes since v1.11 back in 2018, Google Kubernetes Engine (GKE) users are stuck with kube-dns even in rapid channels for v1.17. You can easily install CoreDNS using the migration tool on existing Kubernetes clusters, but GKE won’t let you delete kube-dns so queries end up randomly hitting one or the other.

In most cases, using kube-dns is not an issue, and according to the CoreDNS benchmark tests, kube-dns actually performed about 10% better for internal names. The need for using CoreDNS arises when configuring dnsConfigs or hostAliases on pods doesn’t suit your needs. For example, you may want to create an alias for an internal service to a FQDN to help with migration, mapping calls to an internally hosted service, or to resolve hostname certificate issues for redirected calls. All of these issues are simply solved when using CoreDNS and most of the StackOverflow answers related to managing custom DNS entries point to CoreDNS plugins.

So how do we make use of CoreDNS on GKE?

CoreDNS on GKE

To illustrate, let’s look at a scenario where we have a new scheduling service called scheduler . Inside the cluster, you can access the endpoint by calling its ClusterIP: scheduler.default.svc.cluster.local . Externally, you have set up an Ingress protected with TLS certificates bound to the hostname: scheduler.example.com . The goal is to use the same name to resolve the service, avoiding multiple configurations, and stopping internal calls from traversing the internet and using the external IP. We can accomplish this by using the rewrite plugin on CoreDNS:

.:53 {
    errors
    health {
        lameduck 5s
    }
    ready
    rewrite name scheduler.example.com scheduler.default.svc.cluster.local
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }
    prometheus 0.0.0.0:9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

Now the calls to scheduler.example.com inside the cluster will be modified to route to the ClusterIP. To see this in action, let’s install CoreDNS and configure our Corefile .

Installing CoreDNS

Assuming you have Helm installed, you can install CoreDNS with ClusterIP defined:

$ helm install coredns --namespace=kube-system stable/coredns --set service.clusterIP=<pod.ip> 

Now to edit the Corefile , you can use the GKE Configuration UI or issue the edit command:

$ kubectl edit configmap coredns-coredns -n kube-system

Add in the rewrite lines and reload CoreDNS gracefully:

$ export POD_NAME=$(kubectl get pods -n kube-system -l "k8s-app=coredns" -o jsonpath="{.items[0].metadata.name}")

$ kubectl exec -n kube-system $POD_NAME -- kill -SIGUSR1 1

Configure kube-dns

Next, we need to configure kube-dnsto use CoreDNS when responding to our scheduler service calls:

$ kubectl edit configmap kube-dns -n kube-system

The ConfigMap for kube-dns exposes two configuration options:

• stubDomains: map of DNS suffix keys to DNS IPs
• upstreamNameservers: additional nameservers

For our use case, we are going to take advantage of stubDomains to redirect our calls to the scheduler service to CoreDNS.

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
data:
  stubDomains: |
    {"example.com": ["<CoreDNS IP>"]}

NOTE: We specified the ClusterIP for CoreDNS via the Helm chart so that we don’t have to update the IP if the CoreDNS pod dies and gets a new IP assigned.

Finally, we need to restart kube-dnspods to reload the ConfigMap:

$ kubectl get pods -n kube-system -l "k8s-app=kube-dns"

// delete each kube-dns pods to recreate

Validate DNS Calls

After kube-dns pods restart, we can spin up a pod with dnstools to validate our custom DNS entries:

$ kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools

// Let's make sure, existing DNS entries are fine
# host kubernetes
kubernetes.default.svc.cluster.local has address 192.168.0.1

# nslookup scheduler.example.com
Name:      scheduler.example.com
Address 1: 192.168.9.247 scheduler.default.svc.cluster.local

We can see scheduler.example.com being resolved to our internal scheduler endpoint. To support other endpoints, we can extend the rewrite plugin with regex:

rewrite name regex (.*).example.com {1}.svc.cluster.local

Reload CoreDNS gracefully and now all calls to *.example.com resolves to the corresponding internal service.

Final Words

Until GKE adopts CoreDNS as the default, managed DNS service, using the above workaround will allow us to use CoreDNS as the cluster DNS service addon. As a side note, if you need an external DNS service (e.g. cluster federation), you can modify the CoreDNS Helm chart to set isClusterService to false.

Resources:

• Custom DNS Entries for Kubernetes
• CoreDNS