CoreDNS

Adventures in Kubernetes

“For Want of a Nail”

I have a — let’s not go there — need to alias a Kubernetes Service to a fully-qualified domain name. The hack to get this done took me on a learning opportunity with CoreDNS. CoreDNS is an alternative to kube-dns (link).

Problem

I need to temporarily alias [release-name]-[chart-name]-orderer to orderer.example.com to progress (resolve!?) a Helm Chart for Hyperledger Fabric on Kubernetes.

One Solution?

Success is defined to be that, when Pods call orderer.example.com, the name is resolved to the aforementioned Service. It is possible to configure the cluster’s kube-dns with so-called “stub domains” (configured with a ConfigMap) such that, DNS requests for (in this case) *.example.com are resolved by a DNS service that I control. One based on CoreDNS.

CoreDNS

Let’s test it locally.

I’m going to retain CoreDNS’s example of example.org (dot-org) as a known-working configuration and will add my ownexample.com (dot-com):

NB The magic happens in line #13 where orderer is aliased to [[RELEASE-NAME]]-[[CHART-NAME]]-orderer. Do not forget the terminating “.”. I will replace RELEASE-NAME and CHART-NAME with actual values.

We need a configuration file for CoreDNS too. By default this is called Corefile:

NB this defines 2 DNS domains (example.com and example.org). Both comprise a pipeline of plugins beginning with file. The value after file is a parameter and references the above e.g. example.com file.

Let’s run the Docker image:

docker run \
--interactive \
--tty \
--publish=1053:1053/tcp \
--publish=1053:1053/udp \
--volume=$PWD:/tmp \
coredns/coredns \
  -conf /tmp/Corefile \
  -dns.port 1053

NB The double-publish on port 1053 is because DNS uses both TCP and UDP. I’m assuming the above files are in the current (${PWD}) directory. This directory is mapped to /tmp in the container. The container will find Corefile in this directory and it references example.org and example.com that should be in the same (/tmp) directory.

This should output something similar to:

example.com.:1053
example.org.:1053
2018/08/22 16:48:32 [INFO] CoreDNS-1.2.0
2018/08/22 16:48:32 [INFO] linux/amd64, go1.10.3, 2e322f6
CoreDNS-1.2.0
linux/amd64, go1.10.3, 2e322f6

From another terminal try:

nslookup orderer.example.com -port 1053 localhost

And this should produce:

Server:  localhost
Address: ::1#1053

orderer.example.com canonical name = x-hyperledger-fabric-orderer.
** server can't find x-hyperledger-fabric-orderer: REFUSED

NB This is correctly wrong ;-) CoreDNS has aliased orderer.example.com to x-hyperledger-fabric-orderer. That’s correct. There is currently nothing on that endpoint, so we receive a “can’t find” error (correctly).

CoreDNS’ documentation uses dig instead of nslookup…. your preference:

dig @localhost -p 1053 a orderer.example.com

Container-Optimized OS (COS)

If you have a container and you’d like to run it on Google Cloud Platform, Kubernetes is a good, default choice. In this case, I’m going to run CoreDNS on a COS instance in the same project as my Kubernetes cluster.

To make life straightforward, after provisioning a COS instance, I’m going to run the CoreDNS command manually. COS provides a read-writable filesystem on /tmp which is why I chose this directory previously.

Before running the container, let’s copy the CoreDNS configuration files to the instance:

PROJECT=[[YOUR-PROJECT]]
INSTANCE=[[YOUR-INSTANCE]]

for FILE in Corefile example.org example.com
do
  gcloud compute scp \
  ${FILE} \
  ${INSTANCE}:/tmp \
  --project=${PROJECT}
done

NB COS uses systemd-resolved and COS defaults to DNSStubListener=udp . To run CoreDNS you’ll need to disable this which you can do (temporarily) with sudo systemctl stop systemd-resolved.

Then gcloud compute ssh and run the Docker command as before but (1) change the local directory to /tmp and (2) change the DNS port to its default (53):

docker run \
--interactive \
--tty \
--publish=53:53/tcp \
--publish=53:53/udp \
--volume=/tmp:/tmp \
coredns/coredns \
  -conf /tmp/Corefile \
  -dns.port 53

Issue #1: I would prefer to not open firewall ports unncessarily. However, I’ve been unable to connect to the CoreDNS instance from the Kubernetes cluster using its internal IP. Investigating. The CoreDNS’ instance’s internal IP is available:

DNS=$(\
  gcloud compute instances describe ${INSTANCE} \
  --project=${PROJECT} \
  --format="value(networkInterfaces[0].networkIP)"\
) && echo ${DNS}

The public IP (which works) is available:

DNS=$(\
  gcloud compute instances describe ${INSTANCE} \
  --project=${PROJECT} \
  --format="value(networkInterfaces[0].accessConfigs[0].natIP)"\
) && echo ${DNS}

For the public IP, you’ll need to punch a hole in the firewall. I’ll leave this to your discretion; this opens the instance’s ports to the internet:

gcloud compute instances add-tags ${INSTANCE} \
--tags=coredns \
--project=${PROJECT}

gcloud compute firewall-rules create temp-test-coredns \
--action=ALLOW \
--rules=tcp:53,udp:53 \
--target-tags=coredns \
--project=${PROJECT}

And then, from another instance in the cluster:

nslookup orderer.example.com ${DNS}

Server:  10.138.0.5
Address: 10.138.0.5#53

orderer.example.com canonical name = x-hyperledger-fabric-orderer.

Or, in my case, because I’m running from another COS instance:

docker run \
--net=host \
busybox \
  nslookup orderer.example.com ${DNS}

Server:  10.138.0.5
Address: 10.138.0.5:53

orderer.example.com canonical name = x-hyperledger-fabric-orderer

*** Can't find orderer.example.com: No answer

NB busybox’s nslookup has slightly different behavior. It reports the correct alias but confirms it’s unable to find orderer.example.com (rather than x-hyperledger-fabric-orderer).

Kubernetes

All working well to this point, let’s revise Kubernetes’ kube-dns configuration by applying a ConfigMap that defines the DNS server:

NB You must replace ${DNS} with its value. The result must be enclosed in quotes. The manifest explicitly references the kube-system namespace.

Then:

kubectl apply --filename=kube-dns.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
configmap/kube-dns configured

NB Ignore the “Warning”.

And, let’s test it!

Testing

Issues #2: My attempt to resolve orderer.example.com to the Kubernetes Service name is not working. I do not know why. Investigating.

This (CNAME) does not work:

Instead, I determined the Service’s IP address and reconfigured the CoreDNS example.com file to A(lias) to that:

ORDERER=$(\
  kubectl get services \
  --selector=component=orderer \
  --namespace=${NAMESPACE} \
  --output=jsonpath="{.items[0].spec.clusterIP}"\
) && echo ${ORDERER}

And then:

NB You must replace ${ORDERER} with its value.

Now, from a Pod within the cluster, nslookup will resolve orderer.example.com:

kubectl run test \
--namespace=${NAMESPACE} \
--image=busybox \
--stdin \
--tty
If you don't see a command prompt, try pressing enter.
/ # nslookup orderer.example.com
Name:      orderer.example.com
Address 1: 10.27.253.203 x-hyperledger-fabric-orderer.ytterbium.svc.cluster.local
/ #

Bam!

Outstanding Issues

• Why am I unable to use internal IPs?
• How (!) can I write the example.com to resolve to CNAME (or SRV?)?

Conclusion

It’s (much easier than I made it look) to run your own DNS service using CoreDNS.

If you wish to complement a Kubernetes cluster’s kube-dns with a DNS service that you control that’s easy too.

While I achieved my goal and and am now able to test my Fabric on Helm deployment with the Pods able to resolve orderer.example.com to Kubernetes Service names (and prove a debugging point), the journey was longer than I would have liked :-)

That’s all!