Summary

The undefined website discusses the concept of Linear Trees, a variant of Decision Trees that uses Linear Models at the leaves, and introduces the linear-tree Python package for their implementation.

Abstract

The undefined website introduces Linear Trees as a powerful alternative to traditional Decision Trees in the realm of machine learning. Linear Trees combine the interpretability of Decision Trees with the predictive capabilities of Linear Models by fitting linear estimators at each leaf. This approach allows for more nuanced predictions compared to the constant value outputs of standard Decision Trees. The article highlights the linear-tree Python library, which integrates seamlessly with scikit-learn and enables the construction of Linear Trees for both regression and classification tasks. The website provides visual examples demonstrating the linear approximations made by Linear Trees in contrast to the piecewise constant approximations of Decision Trees. It emphasizes the flexibility and potential advantages of Linear Trees, while also noting that their effectiveness is context-dependent and requires proper validation.

Opinions

The author suggests that Linear Trees are underutilized compared to their potential, indicating that they are not as well-known as standard Decision Trees.
There is an opinion that Linear Trees offer a "simple improvement/alternative" to Decision Trees without sacrificing interpretability.
The article conveys that the linear-tree package simplifies the process of building Linear Trees, making it as straightforward as using traditional Decision Trees.
The author positively endorses the use of Linear Trees for both regression and classification tasks, presenting them as a valuable tool in the data scientist's arsenal.
It is implied that Linear Trees could mitigate the overfitting tendency of Decision Trees by using linear models at the leaves, which may generalize better to unseen data.

Linear Tree: the perfect mix of Linear Model and Decision Tree

Discovering and Developing Linear Model Trees, a valuable alternative to Decision Tree

Decision Trees are well-known algorithms in the field of statistics and machine learning. For sure, every data scientist has ever tried to fit Decision Trees. They are one of the first topics in every course of data mining where we learn how to deal with supervised and unsupervised tasks. When fitting a Decision Tree, the goal is to create a model that predicts the value of a target by learning simple decision rules based on several input variables.

The predictions of a Decision Tree are simple constant approximations obtained at the end of the optimal data splitting process. This base feature made Decision Trees widely adopted. They are simple to understand and interpret. Not require particular assumptions about the input data (no data preparation, no constraints on the features types). On the contrary, they tend to overfit easily if the growth is not monitored. The instability can be a serious problem if some unseen samples fall in low explored sections during the training phase.

Not everybody knows simple yet effective variations of the Decision Tree algorithm. These are known as Model Trees. They learn an optimal splitting of the training data, as in the standard Decision Tree, with the main difference that the goodness of splits is evaluated fitting models. A particular case of Model Trees is known as Linear Tree. This implies having Linear Models in the leaves instead of simple constant approximations. It can be seen as a simple improvement/alternative from the standard Decision Tree preserving at the same time the interpretation ability.

In this post, we introduce how to easily create a Linear Tree using the linear-tree package. linear-tree is a python library to build Model Trees with Linear Models at the leaves.

linear-tree is developed to be fully integrable with scikit-learn. LinearTreeRegressor and LinearTreeClassifier are provided as scikit-learn BaseEstimator. They are wrappers that build a Decision Tree on the data fitting a linear estimator from sklearn.linear_model. All the models available in sklearn.linear_model can be used as linear estimators.

In other words, we only have to choose a Linear Model to build our Linear Tree. The linear-tree wrapper simply develops a Decision Tree structure on the data. At each node, it searches for an optimal split simply evaluating if the weighted loss of the children is lower than the loss of the parent. The losses involved are derived from the fitting of Linear Models on the data sections.

The splits thresholds in each feature are derived through quantile binning to make the process faster. The only parameters involved are the loss criterion, max_depth, min_samples_split, min_samples_leaf and max_bins (which are self explainable and the same as in classical Decision Trees). Let’s see it in action.

LINEAR TREE FOR REGRESSION

In this section, we use a Linear Tree to model a regression task. To make it understandable and visually explainable, we fit a 1D time-series data.

1D sinusoidal data (image by the author)

We operate a fit at various depths to see how the Linear Tree splits the input space and made predictions.

Linear Tree Regressor at various depths (image by the author)

It’s clearly visible that the Linear Tree operates a linear approximation in the splits. This is in contrast with a classical Decision Tree which operates constant approximations on the same data.

Linear Tree and Decision Tree Regressor at depth 6 (image by the author)

Once trained, we can plot the tree to see the learning paths followed in the training procedure.

Plot Linear Tree Regressor (image by the author)

LINEAR TREE FOR CLASSIFICATION

In this section, we use a Linear Tree to model a classification task. To make it understandable and visually explainable, we fit a 2D dataset.

2D classification data (image by the author)

We operate a fit at various depths to see how the Linear Tree splits the input space and made predictions.

Linear Tree Classifier at various depths (image by the author)

It’s clearly visible that the Linear Tree operates a linear approximation in the splits. This is in contrast with a classical Decision Tree which operates a piecewise constant approximations on the same data.

Linear Tree and Decision Tree Classifier at depth 6 (image by the author)

Once trained, we can plot the tree to see the learning paths followed in the training procedure.

Plot Linear Tree Classifier (image by the author)

SUMMARY

In this post, we introduced a variant of classical Decision Trees, know as Model Trees, which evaluate the splits fitting more complex models instead of making simple constant approximations. Then we presented linear-tree, a python framework to build Model Trees with Linear Models. We showed, with simple examples, how a Linear Tree works and what are the differences with Decision Trees. Linear Trees are not known as the standard Decision Trees but they reveal to be a good alternative. As always, this is not true for all the cases, the benefit of adopting this model family may vary according to the case of study and must be properly validated.

CHECK MY GITHUB REPO

Keep in touch: Linkedin