Mask Detection using Detectron2

Mask Detection using Detectron2. This post continues from the previous articles — Facial mask overlay with OpenCV-dlib and Face recognition for superimposed facemasks using VGGFace2 in Keras

We will implement the new Detectron2 Library by facebook to be trained on our “masked faces” dataset for object detection. Thereafter, we will run our Detectron2 inference on our test images where the face masks are synthetically superimposed using openCV and dlib library.

We will be running on Google Colab as we will be utilizing its free GPU resource for our training of our Object Detector. The dataset that we are using can be downloaded from Kaggle. We select this dataset as there are 3 classes in this dataset namely — with mask, mask not properly worn, not wearing mask. You can find the colab notebooks at the end of this post.

Overview of Detectron2

Detectron2 is an advanced model zoo that provides state-of-the-art detection and segmentation algorithms. First released by Facebook AI Research (FAIR)group in Feb 2020, it is the successor of Detectron and maskrcnn-benchmark. It consists of:

• Training recipes for object detection, instance segmentation, panoptic segmentation, semantic segmentation and keypoint detection.
• 80+ pre-trained models to use for fine-tuning (or training afresh).
• Dataset support for popular vision datasets such as COCO, Cityscapes, LVIS, PASCAL VOC, ADE20k.

Detectron2 registers datasets in COCO JSON format.

Overview of our dataset

We will be training our custom Detectron2 detector on face mask dataset from Kaggle. This dataset consists of 853 images with 3 classes — people with mask, mask not properly worn, not wearing mask.

As Detectron2 is not trained on detecting face mask, we will need to train the underlying networks to fit our custom task. Before that, let us perform exploratory data analysis on the dataset. [source code]

The distribution of the data can be shown in Figure 1 where clearly there is an imbalance of class types across the 3 classes. The spread is about 80% of people with mask, 17% of people without mask and 3% of people with mask_weared_incorrect. We dive deeper to check the distribution of the bounding boxes in the images. Figure 2 describes the spread using histogram and it is obvious that about 70% (close to 600 images) of the images have less than 5 bounding boxes.

There are several methods to deal with this type of imbalance dataset — Synthetic Minority Oversampling Technique (SMOTE), Oversampling the minority class, Undersampling the majority class, etc. However, for the purpose of this article, we will proceed to train our object detector with our dataset and keeping in mind that our dataset is biased.

Getting Started

Installing Detectron2 will be relatively straightforward when we are using Google Colab. It can be installed with the following lines:

At this point, we will need to restart the notebook runtime to continue.

Next we will proceed to import the necessary Detectron2 dependencies.

Registering the dataset

To use a dataset with Detectron2, we need to register it so that we can use the predefined data loaders. As our dataset is not in the COCO format, we will need to write a function to return all the needed information about the data as a list and passing the result to DatasetCatalog.register.

Detectron2 keeps track of a list of available datasets in a registry, so we must register our custom data with Detectron2 so it can be invoked for training. We register our metadata which tells Detectron2 about which class id corresponds to which class which helps in visualization later.

Visualize Detectron2 training data

This step is optional as we have viewed our training dataset during our exploratory data analysis phase. However, we can make use of the visualizer library in Detectron2 to easily view our training data to ensure the data has imported correctly.

Fine-Tuning a pre-trained model using transfer learning

After registering the data-set we can simply perform transfer learning on a pre-trained model using the DefaultTrainer class. Detectron2 allows you many options in determining your model architecture, which you can refer to using the Detectron2 model zoo.

In our case, we use the following script to fine tune our selected pre-trained model. This may not be the ideal configuration and we may be able to modify certain parameters such as learning rate, maximum iterations, batch size, for better results. However, for this post, we will use these values as defaults as the test results will show a decent performance from the trained model.

Now we can start training our model. The training process will run for close to an hour and we can visualize the training and loss plots using tensorboard.

There are a lot of metrics of interest in there — most notably total_loss We can see that the total loss converges to a reasonably low value after 999 iterations. We can of course improve the model performance and fine tune the model by increasing the maximum iterations to 1500 or more.

Run Detectron2 inference on test images

With the model is trained, we can use our custom Detectron2 for inference on the test set which are the images that we have created using mask-overlay-opencv-Dlib.

Now, we can perform inference on our test set by creating a predictor object.

The results of the test images is shown in Figures 1 and 2 below. We can observed that the custom trained detectron2 performs well in detecting if the drawn face mask is worn. It does not detect and classify correctly people who wore the drawn face mask incorrectly. The main reason is due to the training dataset which has very small number of images with people wearing mask incorrectly.

We may also consider playing with the SCORE_THRESH_TEST to change the confidence threshold that the model requires to make a prediction.

Save your model and config

We can now save the weights in the os.path.join(cfg.OUTPUT_DIR, "model_final.pth") for future inference by exporting to Google Drive.

To use the model for inference you need both the model weights and config. To save the config use:

Conclusion

In this post, we have successfully trained a Detectron2 object detection model using a custom face mask dataset and used it to test on our “masked” faces that has been generated using OpenCV and dlib library. The results showed that pre-trained model does well to detect the drawn face mask but unable to detect if the mask is worn correctly. The results should improve if we are able to train with a dataset with more balanced classes especially on the class — face mask worn incorrectly.

The results also show that the script in my article — Facial mask overlay with OpenCV-dlib used can provide another alternative solution to create image datasets of people with face masks that can be used to train and evaluate facial recognition systems.

References

Source of images

You can download my complete codes here

https://github.com/xictus77/Facial-mask-overlay-with-OpenCV-Dlib.git