Summary

The article discusses implementing a Java log file tailer in Spring Boot using Server Sent Events (SSE) to stream real-time updates to clients.

Abstract

The article provides a detailed guide on how to create an endpoint in a Spring Boot application that mimics the tail -f command functionality. It explains the inefficiency of polling for file updates and instead recommends using SSE to stream changes to connected clients in real-time. The implementation involves exposing a Text Event Stream endpoint, managing client connections, detecting file changes, and reading new log lines. The solution leverages Java 8's file reading utilities and the WatchService API for file change notifications. The article also includes code examples and references to further reading on SSE and related Spring Boot features.

Opinions

The author suggests that streaming file changes is more efficient than repeatedly fetching updates, likening it to making multiple trips to the grocery store.
The article promotes the use of Server Sent Events for unidirectional streaming from the server to the client, noting its suitability for scenarios where continuous updates are needed.
The author indicates a preference for the SSE approach over traditional polling due to its efficiency and the ability to maintain a single HTTP connection for updates.
The article hints at future exploration of WebSockets for bidirectional communication needs, implying that SSE is not the solution for all streaming scenarios.
A recommendation is made for an AI service, ZAI.chat, as a cost-effective alternative to ChatGPT Plus (GPT-4), suggesting the author's endorsement of the service for similar performance and functionality.

Java Log file tailer (tail -f) in Spring Boot

In this article, we will focus on streaming file changes on a Spring Boot endpoint. You probably used the tail -f command to follow the changes in a file. We will see the same output from our new endpoint.

The most simple approach is to return the content we read from a file in our endpoint. This feature is already available here in Spring Boot Actuator based on this class.

If we try to compare this situation to some real-life activity: you don’t go to the grocery store to buy one item, go home, leave again, get something else, and so on. You are losing a lot of time on your way to the store.

Technology vector created by stories — www.freepik.com

Streaming the content is a better idea in this situation. We can use Server Sent Events to provide file updates in the first http connection.

There are some key tasks for this flow to work:

Expose a Text Event Stream endpoint (GET /logs)
Send updates to the connected clients (StreamService)
Detect file changes (MonitoringService)
Read new lines (MonitoringService)

Text Event Stream endpoint

You can find different articles on how to achieve this here or here. Long story short, depending on the spring boot flavor you use, in the spring-mvc context, you have to use a SseEmitter to be able to send updates to the calling client.

We can see the browser is waiting for our request to finish due to the Content-Type: text/event-stream header.

The next challenge is to store all the connections in our application. We do this because we want to broadcast all the file changes to all the connected clients.

You can find below a sample class allowing us to register new connections, send messages to a specific topic while removing invalid connections.

Please note there is a very long timeout value you may want to change when creating the new SseEmitter(timeout)

Now that we have a way to remember all the connections, we have to tell the MonitoringService object to invoke our Consumer<Path> when changes happen. Here we broadcast messages when a file change is detected for each new line.

Java 8 has some utility methods for reading files we can use: Files.lines is returning a Stream we can iterate over while ignoring previously processed lines.

The final piece of the puzzle uses the WatchService API to detect when our logs file is changed.

A watch service that watches registered objects for changes and events. For example a file manager may use a watch service to monitor a directory for changes so that it can update its display of the list of files when files are created or deleted.

When a change is detected we need to invoke all the available callbacks with the given file as an argument. This class could potentially be used for multiple files, but for our use case we are interested only in the spring.log changes.

Accessing again our endpoint http://localhost:8080/logs returns new events when the log file is being updated.

Each event has an id corresponding to the line number, making it easier for a UI to easily render it.

In this article we replaced the conventional polling approach (a client is periodically asking for updates) with the SSE approach — given an existing connection, the server responds with updates when they are detected. If there is also a need for continuous updates from the client to the server (bidirectional communication) we can use WebSockets — which I plan to describe in a future post.

The source code for running this example is here.