avatarTechletters

Summary

This website content outlines a project to create a live map of London showing current bus line locations using real-time data processing technologies like Apache Kafka, Python, and Leaflet.js.

Abstract

The article introduces the concept of real-time data processing and its significance across various sectors, focusing on two specific examples: real-time fraud detection in the financial industry and real-time public transport tracking. It emphasizes the practical application of these technologies by detailing a project that involves building a live map to track London buses. The project leverages Apache Kafka for event streaming, Python with the Pykafka client for producing and consuming Kafka messages, Flask for web services, and Leaflet.js for map visualization. Additionally, HTML5 Server-Sent Events are used for real-time updates to the web interface. The author breaks down the project into a series of four articles, inviting readers to follow along and provide feedback. The series begins with an introduction and proceeds to cover the implementation of Kafka producers and consumers in Python, culminating in the creation of the front-end map visualization with Leaflet.js.

Opinions

  • The author suggests that real-time data availability is crucial for digitalization and success in various sectors.
  • The article conveys the opinion that real-time fraud detection can significantly reduce manual efforts and limit potential damage caused by attackers.
  • The author expresses enthusiasm about the project's potential to enhance the public transport experience for users in London.
  • By inviting feedback and offering a step-by-step guide, the author shows a commitment to community engagement and education in the field of real-time data processing.
  • The author endorses a cost-effective AI service, ZAI.chat, as an alternative to ChatGPT Plus (GPT-4), indicating a belief in the value provided by this service.

Where is my bus?

Real-time Maps with Kafka, Python and Leaflet

Building a live map of London showing current bus line locations.

The world map (Source).

Everybody is talking about real-time data nowadays and indeed the need is real. Real-time availability of data is becoming a crucial success factor and driver for digitalization in nearly all areas like business analytics, financial data, cybersecurity, in the public sector, and so on.

Let’s have a closer look at two examples of which we will implement the second one in this blog series.

(1) Financial data, Real-time fraud detection

Your bank for sure owns algorithms that can protect your account in detecting fraud transactions. In the past, those algorithms were scheduled once a day giving a bulk output of possible fraud transactions. Employees need to take care of reversing malicious transactions with huge manual efforts and attackers may have caused already large damage.

The possibility to detect and stop fraud transactions in real-time as they appear would save huge manual efforts and limit the potential damage attackers could cause. Fortunately, real-time fraud detection is possible nowadays.

(2) Public sector, Real-time maps

Imagine you are on a sightseeing trip in— let’s take London — and you want to take a bus to visit an attraction like Buckingham Palace. Imagine the city offers an App that provides a live map showing the routes & locations of the current bus lines. You could use this map to locate the next best bus line and hop-on-stop to your desired destination.

This is the example we are creating during this series — a map of London with real-time location updates of three different bus lines as shown in the below picture and video.

We’ll make use of the following technologies:

  • Apache Kafka as high performant low latency event streaming platform
  • Python Pykafka Client to write Kafka Producers and Consumers
  • Python Flask as webserver and for API setup
  • Javascript Leaflet.JS for visualization of the real-time map
  • HTML5 Server-Sent Events to enable the web page to automatically get updates from the Webserver/Kafka Consumer

Additionally, if you are interested in analyzing Twitter data in real-time, you can check out my recent article.

I will split this story into a series of overall 4 stories as shown in the below overview. This allows me to get into more detail of each specific part of the overall solution. As this is my first medium post ever, I would highly appreciate any kind of feedback.

Have fun :-) If you want to follow along with all my stories & support me, you can register on Medium. If something is unclear or you need help, just drop a comment. I will answer it for sure.

Kafka
Python
Flask
Kafka Python
Web Development
Recommended from ReadMedium