Automatic Machine Learning

Auto is the new black (2) — AutoGluon, H2O AutoML and Google AutoML Tables

How they work and what’s behind them

While Machine learning keeps achieving considerable successes in solving and improving any kind of problems, an ever-growing number of disciplines rely on it. However, this success crucially relies on machine learning experts to perform manual tasks. AutoML promises to change this reality and transform it into an experience of the type “data in, model out”.

For an introduction to the Automated Machine Learning topic and the description of the methods used by Google AutoML (ENAS), Microsoft Automated ML, AutoKeras and auto-sklearn, visit:

In this post, I will try to explain and help you build some intuition about how AutoGluon, H2O AutoML and Google AutoML Tables work behind the hood (since although all having the word “auto” in their names, they share nothing in common)

AutoGluon (Tabular)

Open-source: Yes Cloud-based: No, but also available in AWS SageMaker AutoPilot Supports: Classification, Regression Techniques: Compilation of pattern and practices common in the ML field Training framework: Scikit-learn

Rather than trying to use a “meta-learning” approach as its competitors do, the idea behind AutoGluon is to codify a set of ‘tricks of the trade’ which are well understood in the data science community. In this sense, the framework is a comprehensive compilation of pattern, practices, and well-known techniques that tend to work well on average in most successful models. As the paper itself quote, this technique is “admittedly less glamorous” but it has a considerable effect on accuracy. It doesn’t bring something new to the table, but it seems to be the first framework to codify this extensive set of best practices inside a framework. Which are those practices?

Data preparation: AutoGluon relies on two sequential stages of data processing: model-agnostic preprocessing that transforms the inputs to all models, and model-specific which is only applied to the data used for a particular model, including:

• Embeddings are generated for categorical features.
• N-gram features are generated for text features (although only top frequent n-grams are retained for the sake of memory footprint). These vectors are incorporated as new features.
• Date/time features are also transformed into suitable numeric values.
• Missing values are inputted (although not fully specified in the paper, it seems the strategy depends on the type of data. For categorical features, missing values are inputted to an specific unknown category. For numerical values it is not specified).
• Numeric features are scaled and normalized.
• Finally, a feature vector is generated by concatenated all the previous features.

Model ensemble: AutoGluon relies on stacking as the ensemble method of chose to generate better performant models. Particularly, AutoGluon introduces a multi-layer stack ensembling, where the first layer has multiple base models, whose outputs are concatenated and then fed into the next layer, which itself consists of multiple stacker models. These stackers then act as base models to an additional layer. Stacker models take as input not only the predictions of the models at the previous layer, but also the original data features

Model considered: A set of models and order of evaluation is predefined. Models include neural networks, LightGBM boosted trees, CatBoost boosted trees, Random Forests, Extremely Randomized Trees, and KNN. The specific implementations of these models are from Scikit-learn.

For the specific case of neural networks, AutoGluon uses a predefined architecture only with similar design choices as the models of Howard & Gugger (2020) and Cheng et al. (2016). It is basically a 3-layer feedforward network with skip-connection with drop-out modules and bath normalization. The following image depicts the architecture.

Repeated k-fold Bagging: To utilize all of the available data for both training and validation, AutoGluon implements k-fold ensemble bagging of all models at all layers of the stack. This is achieved by randomly partitioning the data into k disjoint chunks, and subsequently training k copies of a model with a different data chunk held-out from each copy. AutoGluon bags all models and each model is asked to produce out-of-fold predictions on the chunk it did not see during training. This technique also allows the framework to mitigate overfitting.

Hyper-parameter tuning: The framework uses hard-corded pre-defined hyper parameter values for the models it trains. However, it does support hyper-parameter tuning though it is not indicated which strategy it uses.

Note: AutoGluon also contains an implementation for Neural Architecture Search inside the package. However, it is an stand-alone implementation and doesn’t apply to this part of the framework which is referenced as AutoGluon-Tabular. Check that out in the Github repository.

H2O AutoML

Open-source: Yes Cloud-based: No (Note that H2O Driveless AI has a different implementation) Supports: Classification, Regression Techniques: Compilation of pattern and practices common in the ML field Training framework: XGBoost and H2O

H2O AutoML uses an strategy similar to AutoGluon in the sense that it encodes a set of patterns and practices for training machine learning models, but with a bit more focus on the hyper-parameters used. Instead of using a predefined set of parameters or, in the other extreme, relying entirely on hyper-parameter tuning, it goes for an approach somewhere in the middle: depending on the algorithm, it might:

• Try several instances of the same model with some combination of hyper-parameters
• Use a predefined set of values for the hyper-parameters.
• Run a random grid search for the best combination of hyper-parameters, but in a pre-defined search space.

Data preparation: Data processing is handled by the framework itself. However, not all data types are supported. In the current version, only tabular structure data is supported. This does not include text. Some of the preprocessing steps depend on the model under consideration.

Model considered: According to its reference page, the current version tries the following algorithm in order: three pre-specified XGBoost GBM models, a fixed grid of GLMs, a default Random Forest (DRF), five pre-specified H2O GBMs, a near-default Deep Neural Net, an Extremely Randomized Forest (XRT), a random grid of XGBoost GBMs, a random grid of H2O GBMs, and a random grid of Deep Neural Nets.

Model ensemble: AutoML trains two Stacked Ensemble models: one ensemble contains all the models, and the second ensemble contains just the best performing model from each algorithm class/family. The “Best of Family” ensemble is optimized for production use since it only contains six (or fewer) base models. It should be relatively fast to use (to generate predictions on new data) without much degradation in model performance when compared to the “All Models” ensemble.

Note: Remember that stacking is an ensembling technique where several algorithms are used to train models on the training data (known as the bottom models). Then, k-fold cross-validation is performed and the predictions are saved. Finally, a new model is trained on the cross-validation data as independent variables and the labels as target. The resulting model is known as the top model.

Google AutoML (Tables)

Open-source: No Cloud-based: Yes, on Google GCP Supports: Classification, Regression Techniques: Automated Feature Engineering, Reinforcement learning, Transfer Neural AutoML (probably — or something similar to that) Training framework: TensorFlow

If you read my previous post, you saw that I already listed Google AutoML. However, Google reshaped they portfolio into several products in the AutoML space. In this post I will cover only AutoML Tables, which is AutoML technology for tabular datasets.

GCP AutoML is designed to be an end-to-end automatic model solution for any machine learning problem: meaning data-in, model out — with zero human intervening. This product is designed to generate not (necessarily) the best model , but a model that ranks on the top 20% solutions while being relevant for the 80% of the problems in the domain. On top of that, it has to work well for several budget plans.

The idea behind AutoML is to automate the machine learning process on each of its stages. This process looks as follows:

On those problems where you can leverage Transfer Learning (like Computer Vision) a transfer learning approach is used. For instance, AutoML Vision uses transfer learning from a NasNet architecture trained in a Google images dataset to finally perform fine tuning on your specific dataset. (if you are interested, the process is explained by Yifeng Lu from Google Brain). However, transfer learning is hard to apply in a tabular dataset. Because of that, feature engineering and architecture search is performed in AutoML Tables.

Automatic Feature Engineering: Automatic transformations are applied for each feature column depending on the column type:

• Numerical: Values are transformed to float32 and extra features are calculated including z-value, bucket index based on quantiles, log(values+1), z-score of the log value, flags if the value is null.
• Numerical array: Last N values are averaged and used as feature
• Categorical: Text is sanitized and encodings calculated. Values appearing less than 5 times are tagged as unknown (I wonder why)
• Text: Sanitization and embeddings is performed for 1-grams and 2-grams. The embedding of all elements are then combined into a single embedding using the mean.
• Timestamp: Determine the year, month, day, and weekday. Treat each value from the timestamp as a Categorical column.
• Struct: Struct values are automatically flattened into fields.

Automated Architecture Search: The next step into the process is to look for a model architecture to use with the prepared data. We saw in my previous post that a way to achieve this is to use Neural Architecture Search — or (E)NAS. However, it is expensive in terms of computational resources given it requires training many networks before getting something useful. As explained in “A Deepdive into AutoML Tables”, the service uses a similar approach to NAS in the sense that it is based on a controller (model) which takes a set of decisions about how to construct the architecture of the model (they call it trial). Then, the model is executed and finally the model evaluation metric calculated. The controller updates its belief based on the resulting performance. This process is iteratively ran until an exit criteria is met which can be time or convergence.

The main difference here is that the action space from where the controller can choose from is rather limited. It will try only linear models, tree-based models (Gradient Boosted Decision Tree) or Neural Networks (AdaNet).

I also suspect (I can’t confirm this) that they might be using also Transfer Neural AutoML (or something similar). The idea of this technique is to avoid training the controller mentioned before from scratch but rather apply transfer learning from a controller trained for generating architecture for a wide variety of tasks. The controller learns both hyperparameter choices common to multiple tasks and specific choices for individual tasks. Then, such controller is used as a “base controller” to find good models for new tasks and leverage the learned priors over performant models. When a new architecture for a new dataset needs to be generated, training is resumed from the “transferred” controller. According to the paper, this technique reduces the time to converge in both text and image domains by over an order of magnitude in most tasks. The paper was published by Google Brain so I think they might be using something like that, but it is just an opinion (or may be a desire cause I find the strategy pretty appealing).

Automatic model selection and ensemble generation: On the final stage, a model ensemble is generated by picking the top performing models from all the previous trials and tested against k-fold cross-validation. This is pretty similar to the strategy applied by auto-sklearn where instead of throwing away all the models, you try to generate an ensemble from them. Models are also picked up considering diversity of methods.

Conclusions

Each of these methods have their own advantages and disadvantages as well as a different spaces to compete with although those boundaries start to blur as their creators start to have the same understanding of what automated machine learning looks like. For instance, almost all the methods seem to agree on the fact that some sort of data preprocessing needs to be done.

However, at the core the methods are very different. AutoGluon and H2O AutoML preffer to hard-code good patterns and practices from ML experts inside a coherent SDK. They even publish it as open-source which is good for transparency. Google keeps the “meta-learning” strategy trying to understand and train models able to generate good calls about model architectures. However, we can see signs that this strategy is not enough by looking how ensemble models are generated.

Bonus: if you are interested in the AutoML field, have a look at Quac Le interview, Google researcher and author of many of the advances in this field: