Best Practices For Monitoring Machine Learning Models In Production
In A Comprehensive Guide on How To Monitor Your Models In Production, we saw how deploying a Machine Learning model isn’t the last step, and why monitoring is essential. It’s essential because through monitoring, you’ll understand if your model is performing as expected, or it needs to be trained again, or the entire application needs to be redefined, more on the article above.
We explored how to monitor the three most important aspect of a Machine Learning application: data, model and output.
We’ll now go over some of the best practices teams can adopt in order to monitor their models effectively.
General best practices
Before diving into the best practices while monitoring data, model and output, let’s go over some general tips.
- Monitoring doesn’t start with deployment: start when the experimentation steps start, monitor your experiments, your logs, your ideas and troubleshooting.
- More is not better, for the tools: don’t use dozens of tools, it can get messy to work with. Use as less tools as you can to get the same job done.
- More is better, for the people: don’t give all the power to one person, decentralize your team, everyone handles a single task, and no one will be overwhelmed.
Practices for data monitoring
- Always use the same pipeline: whether the data is batched or streaming. This will help troubleshooting when needed.
- Check for any drifts: don’t stop at the dataset drift, features drift will provide more insights.
Practices for model monitoring
- Always setup a metadata storage: make sure you always store trained and versioned models hyperparameters. This will help traceability, troubleshooting and rollbacks when needed
- Shadow deployment: before deploying a new model, deploy it in shadow mode, so that it can be tracked along with the current model, log its predictions and performances.
- Check the performance: the model will degrade over time, it’s inevitable, so set up tools which can help detecting it.
Practices for output monitoring
- Track absurd outputs: always track wrong predictions with high confidence, given a specific set of inputs.
- Make sure you exploit all the possible metrics to detect model and concept drift: more on the previous article
Alerting
This is another important aspect: if you’re not alerting in case something goes wrong, why are you even monitoring?
- Agree with everyone on the media: make sure to choose the proper way to send alerts, like slack, mattermost, emails, etc.
- Divide et impera: make sure certain alerts goes to certain teams or people. For example: data alerts should go to the DevOps / Data Engineers.
- Don’t send alerts for everything: choose what actually matters to the business and to the application. Too many alerts will create a hell of noise.
Logging
Log as much as possible: your limit is the sky (or the storage). To make sure you have insights for anything that might go wrong, setting up a good logging system is crucial.
Here’s some things that might be worth monitoring:
- Data events during the pipeline: start and end timings, job failures, etc.
- Production data: always useful when retraining.
- Model metadata: versions and hyperparameters.
- Every prediction result.
- Performances: timings, hardware usage.
No, it doesn’t end with monitoring
Now that you have an entire infrastructure around your Machine Learning model, you will always be ready to keep providing actual business value.
Other articles you might find useful:
If you liked the post, consider following me on Medium and my website: alessandroai.com.
You can also support my work directly and get unlimited access by becoming a Medium member through my referral link here!
Originally published at https://www.alessandroai.com on January 3, 2022.