Overview of UI Monitoring Tools for Apache Kafka Clusters
What are the best monitoring tools for Apache Kafka?
With the increasing demand around Kafka Cluster monitoring and management, a number of open-source and commercial graphical tools have reached the market, offering a variety of administration and monitoring functionalities.
tl;dr: What are the best monitoring tools for Apache Kafka?
- Confluent Control Centre
- Lenses
- Datadog Kafka Dashboard
- Cloudera Manager
- Yahoo Kafka Manager
- KafDrop
- LinkedIn Burrow
- Kafka Tool
Confluent Control Centre
Confluent is the company founded by the original creators of Apache Kafka. Confluent Enterprise, is a -more complete- Kafka distribution for production environments. The commercial licence of Confluent Platform comes with Confluent Control Centre which is a management system for Apache Kafka that enables cluster monitoring and management from a User Interface.
Confluent Control Center delivers understanding and insight about the inner workings of the Apache Kafka clusters and the data that flows through them. Control Center gives the administrator monitoring and management capabilities through curated dashboards, so that they can deliver optimal performance and meet SLAs for their Apache Kafka clusters.
Lenses
Lenses (ex Landoop) is a company that offers enterprise features and monitoring tools for Kafka Clusters. More precisely, it enhances Kafka with User Interface, streaming SQL engine and Cluster monitoring. It also enables faster monitoring of Kafka data pipelines by providing SQL and Connector visibility into your data flows.
Lenses works with any Kafka distribution, delivers high quality enterprise features and monitoring, SQL for ALL and self-serviced real-time data access and flows on Kubernetes.
The company also offers Lenses Box, which is a free all-in-one docker that can serve a single broker for up to 25M messages. Note that Lenses Box is recommended for development environments.
Furthermore, Lenses also offers Kafka Topics UI, which is a web tool for managing Kafka Topics.
Datadog Kafka Dashboard
Kafka Dashboard by Datadog is a comprehensive Kafka Dashboard that displays key metrics for Kafka Brokers, Producers, Consumers and Apache Zookeeper. Kafka deployments often rely on external software which is not part of the Kafka, like Apache Zookeeper. Datadog enables a comprehensive monitoring on all the layers of your deployment, including software components in your data pipeline which are not part of Kafka as such.
Save time on setup and visualize your Kafka data in minutes with Datadog’s out-of-the-box dashboard.
Cloudera Manager
Kafka in Cloudera Manager is clearly a less rich monitoring tool compared to Confluent, Lenses and Datadog. However, it is very convenient for companies that are already customers of Cloudera and need their monitoring mechanisms under the same platform.
Want to read this story later? Save it in Journal.
Yahoo Kafka Manager
Yahoo Kafka Manager is an open-source managing tool for Apache Kafka clusters. With Kafka Manager, you can:
- Manage multiple clusters
- Easy inspection of cluster state (topics, consumers, offsets, brokers, replica distribution, partition distribution)
- Run preferred replica election
- Generate partition assignments with option to select brokers to use
- Run reassignment of partition (based on generated assignments)
- Create a topic with optional topic configs
- Delete topics
- Batch generate partition assignments for multiple topics with option to select brokers to use
- Batch run reassignment of partition for multiple topics
- Add partitions to existing topic
- Update config for existing topic
- Optionally enable JMX polling for broker level and topic level metrics.
- Optionally filter out consumers that do not have ids/ owners/ & offsets/ directories in zookeeper.
KafDrop
KafDrop is an open-source UI for monitoring Apache Kafka clusters. The tool displays information such as brokers, topics, partitions, and even lets you view messages. It is a lightweight application that runs on Spring Boot and requires very little configuration.
Kafdrop 3 is a UI for navigating and monitoring Apache Kafka brokers. The tool displays information such as brokers, topics, partitions, consumers and lets you view messages.
This project is a reboot of Kafdrop 2.x, dragged kicking and screaming into the world of JDK 11+, Kafka 2.x and Kubernetes.
LinkedIn Burrow
LinkedIn Burrow is an open-source monitoring companion for Apache Kafka that provides consumer lag checking as a service without the need for specifying thresholds. It monitors committed offsets for all consumers and calculates the status of those consumers on demand. An HTTP endpoint is provided to request status on demand, as well as provide other Kafka cluster information. There are also configurable notifiers that can send status out via email or HTTP calls to another service. Burrow is written in Go, so before you get started, you should install and set up Go.
Kafka Tool
Kafka Tool is a GUI application for managing and using Apache Kafka clusters. It provides an intuitive UI that allows one to quickly view objects within a Kafka cluster as well as the messages stored in the topics of the cluster. It contains features geared towards both developers and administrators. Using Kafka Tool, you can:
- View metrics on cluster, broker, topic and consumer level
- View contents of messages in your partitions and add new messages
- View offsets of the Kafka consumers, including Apache Storm Kafka spout consumers
- Show JSON and XML messages in a pretty-printed format
- Add and drop topics plus other management features
- Save individual messages from your Kafka partitions to local hard drive
- Write your own plugins that allow you to view custom data formats
The tool runs on Windows, Linux and Mac OS.
Comparison and Conclusions
If you cannot afford commercial licenses then your options are Yahoo Kafka Manager, LinkedIn Burrow, KafDrop and Kafka Tool. In my opinion, the former is a comprehensive solution that should do the trick for most of the use-cases.
If you are running relatively big Kafka Clusters, then it is worth paying for a commercial license. Confluent and Lenses offer more rich functionality compared to the other monitoring tools we’ve seen in this post and I would highly recommend both of them.
For an updated list of available Kafka UI Monitoring tools, make sure to read my recent article below:
Become a member and read every story on Medium. Your membership fee directly supports me and other writers you read.
