Hyper-Parameter Optimization with Optuna

How to generate the optimal version of your model.

Hyper-Parameter Optimization is a difficult task. However, it can be made easier with tools like Optuna.

In this post, I show how to tune the hyper-parameters of a CatBoost model using Optuna.

The code here for Optuna can be quickly adapted to whatever model you are training. Feel free to bookmark this post for future use.

Hyper-Parameters

One often underappreciated task of building data science models is hyper-parameter optimization. This task is usually placed at the end of building out a model.

However, it can make the difference between a horrible model and a fantastic model. These hyper-parameters are in a class of their own as they facilitate the structure of the model you are using.

In contrast, regular parameters are those learned by a machine learning algorithm during training. In the most basic example, linear regression uses parameters to determine how much weight to multiple with each feature.

These parameters control the base output of the model. However, for linear regression, you also can add regularization to the model accompanied by a hyper-parameter controlling how much to weigh the regularization term.

This additional term with a hyper-parameter alters how the model will predict overall. Changing this parameter can make a decent model predict nonsense or transform the model into something incredibly robust.

Changing Model Behaviour

This behaviour is even more prominent in more complex models such as decision trees, where the many different hyper-parameters available control the depth of the tree, the number of leaves, how splits are performed, and many other options.

Each of these options controls the structure of the model and how it makes decisions. While you still have decision trees in each configuration, the type of tree can vary wildly. It’s the difference between a decision pine tree versus a decision bonsai tree.

To complicate matters, further ensemble models poses an additional challenge. These complex models are built on many different models, each having its hyper-parameters.

Most of the time, you select hyper-parameters which each model uses to train the models in the ensemble. For example, in tree-based ensemble models such as CatBoost, these initial parameters control the set of trees. Both the shape of the tree and the number.

Hyper-parameter Optimization

There are several options available when it comes to hyper-parameter optimization. The most commonly used approach is a variation of grid search.

Grid Search

Grid search is a simple brute force method that generates models for each combination of hyper-parameters that you feed into the search space. Thus, a model is created for each variety and then compared. While appealing at first, there are a few critical aspects to realize.

First, determining the optimal hyper-parameters is an NP-Hard problem since you are dealing with combinations of hyper-parameters. Thus a brute force search is incredibly costly in terms of complexity.

The second aspect to notice is that you may be training models that are consistently performing far worse for a large portion of the search. But grid search is designed to build and train these models.

Suppose you are building a decision tree, and you have a grid search that, among other things, includes the change from ‘Gini’ to entropy for the criterion. Furthermore, suppose you find that ‘Gini’ is far superior in performance in the first few tests that you run. However, even if entropy performs worse in every subsequent model, grid search will still search every one of these combinations.

Random Search

An alternative to grid search is a random search. Which at first glance may appear to be a worse option than grid search. However, it has been proven that a random search performs better than a grid search.

The rationale very crudely is that a random search avoids many redundant searches that a grid search performs. For example, by searching through hyper-parameter spaces at uneven intervals, it is more likely that a more substantial local optimum of hyper-parameters is found.

Alternatives

Now there are alternatives to the two hyper-parameter searches discussed so far. These alternatives are precisely the target for this piece.

Since both of the previous methods do not incorporate any structured approach to searching for an optimal hyper-parameter set, they are at a disadvantage. Instead, libraries such as scikit-optimization and Optuna have directed hyper-parameter searches.

For more details about how grid search and random search compare, you can read my article on hyper-parameter optimization. Additionally, I show how another method, Bayesian optimization performs better than both of the methods.

Optuna

Optuna is an optimization tool that lets the user run experiments on the hyper-parameters space. Importantly, it also always users to pause searches, try other combinations of hyper-parameters and then continue the optimization process.

Additionally, Optuna supports a tree-based hyper-parameter search known as TPESampler ‘Tree-structured Parzen Estimator’. This approach relies on Bayesian probabilities to determine which hyper-parameter selections are the most promising and iteratively adjust the search.

Optuna Setup

Optimizing hyper-parameters with Optuna follows a similar process regardless of the model you are using. The first step is to set up a study function. This function dictates the sample distributions of each hyper-parameter.

The most common options available are categorical, integer, float, or log uniform. Log-uniform is ideal when you want to check values through the range 0.001, 0.01, and 0.1, where each of the values will have the same probability of being selected.

Another massive benefit to Optuna is the ability to set up conditional hyper-parameters. Unfortunately, many hyper-parameters are only relevant when used in combination with others. Therefore simply using variations of all of them can cause errors. Adding cases for each selection ensures that these situations don’t happen.

Model Optimized

To illustrate a use case of Optuna, I’ve chosen to optimize a CatBoost model. These models have an incredible amount of hyper-parameters. While this post only showcases a fraction of those available, many Optuna features such as conditional hyper-parameters are displayed.

Catboost

Catboost is a tree-based ensemble method. It is an incredibly robust model.

One of the immediate benefits of CatBoost, in contrast to other predictive models, is that CatBoost can handle categorical variables directly. Hence the name ‘Cat’ is short for categorical.

This property of CatBoost makes it ideal for lazy data scientists. Converting categorical variables into numeric variables can take some time and is required to support the assumptions of other models. Additionally, you need to determine the correct representation of the feature.

However, with CatBoost, you only need to define the categorical parameters and then adjust the hyper-parameters to handle these categorical features.

The hyper-parameter ‘cat_features’ dictates which features are categorical. Without all of these categorical features specified, CatBoost will throw a float expected error as the model generally assumes that the remaining features are numeric.

CatBoost Hyper-Parameters

• loss_function — Training loss function, for regression you can use RMSE or MAE.
• iterations — Limits the number of trees. However, other hyper-parameters may limit the number of trees resulting in a total less than the number of iterations.
• learning_rate — The learning rate is used during optimization, i.e. during gradient descent.
• l2_leaf_reg— Specifies the coefficient for the regularization term. The term is L2 and added to the cost function.
• depth— Depth of the tree.
• min_data_in_leaf— Specifies when to stop splitting. When the number of instances is below this minimum, the node becomes a leaf.
• one_hot_max_size— One-Hot encodings for parameters with unique values less than or equal to this value.
• boosting_type — ‘Ordered’ or ‘Plain’ ordered is better on smaller datasets but slower than the Plain scheme. Therefore, for larger datasets, the ‘Plain’ boosting type is recommended.
• rsm— ‘Alias: colsample_bylevel’ Defines the percentage used to select features at a split and when features are selected again at random.
• bootstrap_type— Method to sample weights, either ‘Bayesian’, ‘Bernoulli’, ‘MVS’, ‘Poisson’, or ‘No’
• bagging_temperature— Defines the settings of the Bayesian bootstrap. Adds weights according to an exponential distribution when the parameter is set to 1.
• subsample— The Sample rate for bagging is used when ‘Poisson’, ‘Bernoulli’, or ‘MVS’ is used for the bootstrap method.

Experiments

For this example, I am using the diamonds dataset under a public domain license. This data set consists of a combination of categorical and numeric variables.

The dataset aims to predict the price of the diamonds given the other attributes. Some of the variables are categorical, which would generally require some preprocessing.

However, with CatBoost, it is possible to generate a model without any preprocessing. Furthermore, even missing values are handled using CatBoost, making this an incredibly robust and easy to use model.

Simply load the dataset and run.

After setting up the study in Optuna, a few parameters still need to be adjusted. First is the sampler. Here I manually change the sampler to use the TPESampler discussed earlier. This choice ensures that the search will be more structured and directed instead of the standard grid search.

A few other options to make a note of are the direction, n_trials, and the timeout. The direction dictates how the optimization is performed. Make sure this matches the expected optimization of the loss function you’re using.

Next, the n_trials controls how many samples of the hyper-parameter space will be performed. Combined with the timeout, these two factors influence how long the study will run. If you find yourself in a time crunch, these can be lifesavers.

Since the final state of the search will be saved (and can be restarted later), you can sequential search for better and better hyper-parameter selections using the same study.

Visualization of Hyper-parameter search

Once the study has terminated, whether after reaching the final iteration or reaching the timeout limit, the next step is to review the study results.

The script above will output the optimal model performance and the hyper-parameters used. You can also view the study’s progress with Optuna’s built-in visualization functions.

Hyper-Parameter Importance

Determine which parameters had the most significant impact on the overall performance of the model.

Performance over multiple iterations

This plot shows the performance of the model over many iterations. The expected behaviour is that the search sequentially increases the model performance.

Performance of Individual Hyper-Parameters

This plot shows the progression of the different hyper-parameters over multiple trials. The colour indicates the trial number.

Therefore, the expected behaviour is that the hyper-parameters will converge to a single value as the study progresses.

Wrap Up

Without proper tuning of hyper-parameters, a model’s performance can suffer greatly. Several options are available for optimization; however, tools like Optuna make this process simple and effective.

Optuna offers a Bayesian-based approach to hyper-parameter optimization and effective search structuring. Users can search, stop, search more, and save results.

For complex models with many options such as CatBoost, this ability to search through the available model configurations becomes paramount.

Optuna provides an ideal solution to the practical hyper-parameter tuning of your models.

If you’re interested in reading articles about novel data science tools and understanding machine learning algorithms, consider following me on medium. I always include code in my articles that you can apply to your work!

If you’re interested in my writing and want to support me directly, please subscribe through the following link. This link ensures that I will receive a portion of your membership fees.