Custom Implementation of Feature Importance for your Voting Classifier Model

Scikit-learn package lacks feature importance implementation for Voting Classifier unlike other models

Machine learning models are becoming increasingly employed in complex high settings such as financial technology, medical science, etc. Despite the increase in utilization, there’s a lack of techniques to explain the model. The higher the interpretability of the model, the easier it becomes for someone to comprehend the results. There are various advanced techniques and algorithms to interpret the models including LIME, SHAP, etc.

Feature Importance is the simplest and most efficient technique to interpret the importance of the feature for the estimator. Feature Importance can help to get a better interpretation of the estimator and lead to model improvements by employing feature selection. There are various techniques to compute the feature importance score of the estimator:

• Scikit-Learn built-in implementation of Feature Importance
• Feature Importance computed with the Permutation method
• Feature Importance computed with the SHAP value

In this article, we will further discuss the first feature importance strategy. Scikit-Learn package comes up with a function model.feature_importances_ to compute the feature importance for most of the estimators except the Voting Classifier estimator. Scikit-learn employs different algorithms to compute the feature importance score. For Random Forest or Decision Tree models, it computes the importance score using Gini Importance, for Logistic Regression it uses the vector weight.

In this article, we will discuss the implementation to compute the Importance score for a Voting Classifier model.

What is Voting Classifier?

Voting Classifier is an ensemble technique that combines the predictions of various models together predicts an output (class) based on their highest probability. The voting classifier algorithm simply aggregates the findings of each classifier passed into the model and predicts the output class based on the highest majority of voting.

Voting Classifier supports two types of voting techniques:

• Hard Voting: The predicted output of the voting classifier estimator will be calculated based on the highest majority of votes i.e the class which had the highest probability of being predicted by each of the classifiers.
• Soft Voting: The output class is the prediction based on the average probability given to that class.

To get a better understanding of the Voting Classifier, read an article by Mubarak Ganiyu:

Feature Importance:

Unlike other model estimators, Scikit-learn does not come up with the feature importance implementation of Voting Classifier. But each of the base estimators used for the Voting Classifier has its implementation. The idea is to combine the importance score of each of the estimators based on the weights.

The steps of the algorithm to compute the feature importance of the Voting Classifier is:

1. Compute Feature Importance score of each of the base estimators
2. Multiply the weights of the base estimator to the importance score of each of the features.
3. Average out the features importance score (from step 2) for each feature.

Conclusion:

In this article, we have discussed a technique or hack to compute the feature importance score for the Voting Classifier ensemble model. Feature importance score can be helpful to understand the top important features for the model and may lead to model improvements by employing feature selection.

References:

[1] Voting Classifier Documentation from Scikit-Learn: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html

Thank You for Reading