Spring Boot | Spring Cloud Stream | Minikube | Kubernetes

Running in Minikube (Kubernetes) a Kafka Producer and Consumer that uses Spring Cloud Stream

Step-by-step guide on how to run News Producer and Consumer in Minikube (Kubernetes)

In this article, we will demonstrate how to locally set up Minikube (Kubernetes) on our machines and deploy, within it, instances of two Spring Boot Kafka applications, News Producer and News Consumer. Additionally, we’ll show you how to deploy Kafka in Minikube, which both News Producer and Consumer depends on.

You can find the complete code and implementation in the article linked below. Feel free to follow the steps explained in the article and get started.

This is one article of a series of articles where we are using the News Producer and News Consumer as base applications. Throughout these articles, we’ll cover topics such as implementing unit tests, implementing end-to-end tests and deployment to Kubernetes.

So, let’s get started!

Prerequisites

To follow along with this guide, beside Java 17+ and Docker, please ensure that you have the Minikube, Kubectl, Helm and VirtualBox installed on your machine.

Updating News Producer and Consumer Apps

When we use Kubernetes to control our apps, it takes care of keeping the pods healthy. For example, it makes sure our apps only start handling requests when they’re ready. If a pod’s main process stops working, Kubernetes restarts the container.

To do this job, Kubernetes uses two probes: readiness and liveness. The readiness probe checks if an app is ready to get requests. The liveness probe checks if a pod is still going. This helps decide when to restart a container.

Now, let’s add the Actuator tool and enable the liveness and readiness probes.

Add the Actuator Dependency

In the pom.xml file of both News Producer and News Consumer, let’s include the Actuator dependency by adding the following code (highlighted in bold):

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" ...>
    ...
    <dependencies>
        ...
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        ...
    </dependencies>
    ...
</project>

It was added because it provides built-in endpoints for checking the application’s health, including specific ones for liveness and readiness probes.

Enable the Liveness and Readiness Probes

In the application.properties of both News Producer and Consumer, let’s enable the liveness and readiness probes by adding the following line:

management.endpoint.health.probes.enabled=true  

Creating YAML Deployment files

Create k8s folder

Inside the news-app root folder, let’s create a folder called k8s.

Create News Producer YAML Deployment file

Inside the k8s folder, let’s create a file named news-producer.yaml and include the following content:

apiVersion: v1
kind: Service
metadata:
  name: news-producer
  labels:
    app: news-producer
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 8080
      targetPort: 8080
  selector:
    app: news-producer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: news-producer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: news-producer
  template:
    metadata:
      labels:
        app: news-producer
    spec:
      containers:
        - name: news-producer
          image: news-producer:0.0.1-SNAPSHOT
          imagePullPolicy: Never
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: KAFKA_HOST
              value: kafka.dev.svc.cluster.local
            - name: KAFKA_PORT
              value: "9092"
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: http
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: http

This YAML file sets up a News Producer service and deployment. The service is a LoadBalancer, exposing the port 8080 for external access. The deployment creates a single pod with the news-producer:0.0.1-SNAPSHOT image and configuration, connecting to a Kafka message broker. The configuration includes liveness and readiness probes.

Create News Consumer YAML Deployment file

Inside the k8s folder, let’s create a file named news-consumer.yaml and include the content below:

apiVersion: v1
kind: Service
metadata:
  name: news-consumer
  labels:
    app: news-consumer
spec:
  type: ClusterIP
  ports:
    - port: 9080
  selector:
    app: news-consumer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: news-consumer
spec:
  replicas: 3
  selector:
    matchLabels:
      app: news-consumer
  template:
    metadata:
      labels:
        app: news-consumer
    spec:
      containers:
        - name: news-consumer
          image: news-consumer:0.0.1-SNAPSHOT
          imagePullPolicy: Never
          ports:
            - name: http
              containerPort: 9080
          env:
            - name: KAFKA_HOST
              value: kafka.dev.svc.cluster.local
            - name: KAFKA_PORT
              value: "9092"
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: http
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: http

This YAML file sets up a News Consumer service and deployment. The service makes the application accessible within the cluster via ClusterIP. The deployment creates three pods with the news-consumer:0.0.1-SNAPSHOT image and configuration, connecting to a Kafka message broker. The configuration includes liveness and readiness probes.

Installing Bitnami Helm Repo

In order to run Kafka, we will use the Bitnami Helm Charts. So, in a terminal, let’s add the bitnami repo by running the following command:

helm repo add bitnami https://charts.bitnami.com/bitnami

To delete the bitnami repo, run: helm repo remove bitnami

In order to check the repo list, run:

helm repo list

Finally, run the command below to get the latest information about charts from the chart repositories:

helm repo update

Start Environment

In a terminal, make sure you are in the news-app root folder.

Start Minikube

Run the following command to start Minikube:

minikube start --memory=8192 --cpus=2 --vm-driver=virtualbox

Create dev namespace

Once Minikube is up and running, let’s create a namespace called dev. For it, run the command below:

kubectl create namespace dev

To delete the dev namespace, run: kubectl delete namespace dev

Deploy Kafka

Let’s deploy Kafka. For it, make sure you are in news-app root folder. The, run the following command:

helm -n dev install kafka bitnami/kafka --set listeners.client.protocol=PLAINTEXT

To delete the Kafka installation, run: helm uninstall -n dev kafka

Watch Kafka Deployment

We can watch the deployment by running the following command:

kubectl -n dev get pods -w

To stop watching, press: Ctrl+C

Please be patient as Kafka Docker image will be pulled before we can proceed further. After a few seconds, the Kafka instances should be up and running. To confirm this, you can use the following command:

kubectl -n dev get pods

You should see something like:

NAME                 READY   STATUS    RESTARTS   AGE
kafka-controller-0   1/1     Running   0          4m34s
kafka-controller-1   1/1     Running   0          4m34s
kafka-controller-2   1/1     Running   0          4m34s

Building News App Docker images

In a terminal, let’s execute the following command to sets up our local Docker environment to use Minikube’s Docker daemon:

eval $(minikube docker-env)

Navigate to the news-procucer folder. Run the command below to build the News Producer Docker image:

./mvnw clean spring-boot:build-image -DskipTests

Let’s build the News Consumer Docker image. For it, navigate to news-consumer folder. Once there, run the following command:

./mvnw clean spring-boot:build-image -DskipTests

After the completion of the commands above, the News Producer and Consumer Docker images will be available within Minikube. You can verify this by executing the following command:

docker images

Let’s switch back to our local host Docker daemon by running the command below:

eval $(minikube docker-env -u)

Deploying News Apps

In a terminal, ensure that you are in the news-app root folder. Then, execute the following command to deploy the News Producer:

kubectl -n dev apply -f k8s/news-producer.yaml

To delete the news-producer deployment, run: kubectl -n dev delete -f k8s/news-producer.yaml

Now, let’s deploy the News Consumer by running the command below:

kubectl -n dev apply -f k8s/news-consumer.yaml

To delete the news-consumer deployment, run: kubectl -n dev delete -f k8s/news-consumer.yaml

After a few seconds, the single News Producer instance and three News Consumer instances should be up and running. To confirm this, you can use the following command:

kubectl -n dev get pods

You should see something like:

NAME                             READY   STATUS    RESTARTS   AGE
kafka-controller-0               1/1     Running   0          13m
kafka-controller-1               1/1     Running   0          13m
kafka-controller-2               1/1     Running   0          13m
news-consumer-69cd94fb64-4p2pk   1/1     Running   0          15s
news-consumer-69cd94fb64-k6nd4   1/1     Running   0          15s
news-consumer-69cd94fb64-rm8v7   1/1     Running   0          15s
news-producer-5bc75c69c9-88vfs   1/1     Running   0          23s

Sending News

In a terminal, let’s start monitoring the logs of the three News Consumer instances:

kubectl -n dev logs -l app=news-consumer -f

In another terminal, run the following command to get the NEWS_PRODUCER_HOST_PORT:

NEWS_PRODUCER_HOST_PORT="$(minikube ip):$(kubectl get services -n dev news-producer -o go-template='{{(index .spec.ports 0).nodePort}}')"
echo $NEWS_PRODUCER_HOST_PORT

In the same terminal where NEWS_PRODUCER_HOST_PORT was obtained, let’s send one news by executing the command below:

curl -i -X POST $NEWS_PRODUCER_HOST_PORT/api/news \
  -H 'Content-Type: application/json' \
  -d '{"title":"Article about Spring Cloud Stream and Kafka"}'

One of the News Consumer instances should listen to the message sent and log it. Here is what was logged during my execution:

INFO 1 --- [container-0-C-1] c.e.newsconsumer.listener.NewsListener   : Received News! "Article about Spring Cloud Stream and Kafka" created on '2023-10-28T21:11:05.605584947Z'

Shutdown

In a terminal, run the following command to delete completely Minikube:

minikube delete

Conclusion

In this article, we configured Minikube for local Kubernetes development. Before deploying the News Apps, we added the Actuator dependency and enabled their liveness and readiness probes.

In Minikube, we have deployed Kafka, one instance of News Producer and three instance of News Consumer. At the end, we sent a news using the News Producer endpoints, and checked whether the message was successfully consumed by News Consumer application.

Additional Readings

Support and Engagement

If you enjoyed this article and would like to show your support, please consider taking the following actions:

• 👏 Engage by clapping, highlighting, and replying to my story. I’ll be happy to answer any of your questions;
• 🌐 Share my story on Social Media;
• 🔔 Follow me on: Medium | LinkedIn | Twitter | GitHub;
• ✉️ Subscribe to my newsletter, so you don’t miss out on my latest posts.