Using custom metrics for callbacks in Keras model training

Keras provides several in-built metrics which can be directly used for evaluating the model performance. However, it is not uncommon to include custom callbacks, to extend beyond keras' capabilities, as I myself had to do recently. In this post, I discuss how to use custom calculated values (metrics, losses) with in-built callbacks.

Before proceeding, we will need to define our own custom callback to carry out calculations that we want for tracking the training progress or evaluate the model performance. Below is a dummy code for defining a custom callback.

CustomCallback(keras.callbacks.Callback):
   def on_epoch_end(self, epoch, logs={}):
            :
      do_somthing_here
            :
      logs["metric_name"] = metric_value

Once custom callback is defined, the related custom metric can be included in the other callbacks

ckpt_callback = ModelCheckpoint(
    path_to_file,
    monitor="metric_name",
    mode="Specify depending on desired behaviour of the metric",  
    save_freq="as above",
)

This behaviour is not limited to exchanging data from custom callbacks to the in-built callbacks, but can be extended across all callbacks, since the 'log' object is passed automatically to all of them through model.fit(). Similar workflow will work with other default callbacks like EarlyStopping, etc. where the functionality is dependent on a custom defined value/metric.

Attention: Make sure that the callbacks are lined up in model.fit() in the order of information update. In the above example, if ModelCheckpoint callback is listed before the custom callback, it will work asynchronously with respect to the custom metric, as it will be looking at the metric value from the previous step.

Hence, here the desired order will be as follows:

model.fit(
    train_generator,
         :         ,
    epochs,
    callbacks = [CustomCallback, ckpt_callback]
     )

With that, you are now ready to control the training of your Machine learning model more flexibly.

If you find stories like these valuable and would like to support me as a writer, please consider following me or signing up for Medium membership.