Async template program using Watchdog + RabbitMQ + ini file deluxe — part1

Introduction

This is part1 in the series. This part talks about why a template program is useful when you are working with a distributed system. It covers the overall architecture of the example and the installation of required components.

Here’s a brief outline of the article parts:

Introduction and installation of required components (this article).
Template program example.
Template program worker example.
Common program tools not mentioned elsewhere.
Enhanced INI file handling.
Watchdog (file system supervision) handling.
RabbitMQ (message broker) handling.

Background

An async template program becomes useful when you have lots of them, like when you are building a distributed system. The general idea behind a distributed system is that each component (program or service) is responsible for one thing only and that they react on different events in the overall system. By using a message broker like Kafka or RabbitMQ they are loosely coupled and it’s easy the extend the system with new functionality.

The architecture used for building a distributed system is as diverse as the number of ice-cream flavors available in a Italian bistro during the summer. If your system is not fully deployed in the cloud, but rather is spread over a number of servers, this might be interesting for you to look at. It also might be interesting to look at this template code since there’s some useful coding techniques that are used

The template program is fully async. Most of the external packages that are used are also async, with one exception and that is watchdog. I had to envelope that with some code so that it works in harmony with the other async code.

The following building blocks have been used:

Template program functionality: The program is using JSON messages, both internally between methods, when needed and externally with other programs by using an internal message broker. This results in loosely coupled code that’s easily expanded. The program uses both configuration and INI file handling to dynamically adapt to different environment on different servers. More about this in part2+3 of these articles.
Common tools not mentioned elsewhere: There are a bunch of useful tools that will be covered in part 4 in these articles.
INI file functionality: It is based on the standard python ConfigParser module. To that I have changed interpolation to ExtendedInterpolation and added some extra functionality on my own. It is now possible to reference both environment variables and secrets (they are both expanded). Validation is performed on the required parameters in the file by using pydantic. Directory paths are also checked for existence. More about this in part5 of these articles.
Watchdog functionality: This is a really good package that uses native operating system APIs to detect changes in the file system. A variety of platforms and file systems are supported. I use it to detect new files in selected directories. More about this in part6 of these articles.
RabbitMQ functionality: A message broker handles the distribution of messages between concerned parties. The package used for accessing the broker is aio-pika. You can publish and subscribe on both work queue and topic queue messages (what that means is described here). Topic subscriptions can be permanent or temporary, work queue subscriptions are always permanent. More about this in part7 of these articles.

Installing required components

External building blocks

If you already have access to RabbitMQ then you can skip the whole external block part and go directly to the python installation part.

The building components that I have used are:

Docker Desktop to manage the external components below.
RabbitMQ as a Message Broker.

Installing Docker Desktop

Having Docker as a tool in your development toolbox is a good thing. The reason I’m thinking this is good is that it is so easy install and to remove different components when you don’t need them anymore. No more droves of DLL, or packages that might be hard to remove when you are finished due to their interdependencies.

I’m using the Docker Desktop since it’s free for personal use. There are alternatives, but that is outside the scope of this article.

There are install kits for different platforms on the Docker site. Just make sure your computer meets the requirements before you start:

When you have the Docker Desktop program up and running it’s time to install the required components.

You should create a docker directory separate from the template program example to store the two docker files. I mention this since it is might be a good idea to centralize the definition of commonly used resources. Since you might install more resources in the future, like MySQL, MongoDB or Redis.

If you want to keep it local to the example then start with creating a directory named async_template_program where you want to develop this example. Then create a subdirectory named docker. Create the following two files in that docker directory.

docker-compose.yaml

Insert the following content into the file:

version: '3.7'

services:

  rabbitmq:
    build:
      context: .
      dockerfile: Dockerfile.rabbitmq
    restart: always
    container_name: rabbitmq
    volumes:
      - rabbitmq-log:/var/log/rabbitmq
      - rabbitmq-data:/var/lib/rabbitmq
    ports:
      - 5672:5672
      - 15672:15672
    networks:
      - proxynet

volumes:
  rabbitmq-log:
  rabbitmq-data:

networks:
  proxynet:
    name: custom_network

Dockerfile.rabbitmq

Insert the following content into the file:

# pull official base image
FROM rabbitmq:management

# Set local timezone.
ENV TZ=Europe/Stockholm
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

Note that I’m using a specific time zone. If you are in another time zone you have to change it. If you remove it completely you will get the default UTZ time zone.

Then start a terminal window in the Docker directory and run the following two commands (one at a time):

$ docker-compose build
$ docker-compose up -d

This should build the image for RabbitMQ and start it in a container.

Configure RabbitMQ

We need to define a virtual host named /dev. The following steps are required:

start a web browser and enter the following URL: http://localhost:15672/
Use the default credential to login: guest, guest respectively.
Click on the Admin tab, and then on the Virtual Hosts tab, located on the right side of the window.
Click on Add a new virtual host. As name specify /dev and press Add virtual host.

Installing the python environment

We need to setup the python environment for the example by creating a virtual environment, called .venv under the async_template_program directory by starting a terminal window in that directory and issuing the following command:

$ python -m venv .venv

Activate the environment in the command window like this (for Windows):

$ .venv\scripts\activate

If you are on Linux or MacOS you type:

$ source .venv/bin/activate

You can see it activated when (venv) is shown as a prefix in the command line prompt.

Then we need to install third-party python packages. Create the the following file under the async_template_program directory:

requirements.txt

aio-pika==9.3.0
aiofiles==23.2.1
apscheduler==3.10.4
loguru==0.7.2
pydantic==2.4.2
pydantic-settings==2.0.3
pywin32==306
pyyaml==6.0.1
watchdog==3.0.0

Install the content of the created file by running the following command in the venv activated command window created above:

(venv)$ pip install -r requirements.txt

Conclusion

The example code is available in my GitHub repository. Documentation can be found here.

In this article we have taken a quick look at the problem space and a solution to simplify the distributed development work. And we have installed required components.

I hope you enjoyed this article and are stooked for the upcoming parts of this article series. Remember, if you like this article don’t hesitate to clap (I mean digitally 😊).

Happy coding in Gotham.

/Anders