The provided content discusses the impact of resampling techniques on imbalanced datasets, using the credit card fraud dataset as a case study, and employs the deepchecks library to compare the original dataset with resampled ones.
Abstract
The article delves into the effects of resampling imbalanced datasets, focusing on the credit card fraud dataset, which is characterized by a significant class imbalance. It explores three common resampling techniques: random undersampling, random oversampling, and SMOTE (Synthetic Minority Over-sampling Technique). The author uses the deepchecks library to perform data validation checks, comparing the distributions and relationships of features before and after resampling. The analysis reveals that while resampling can improve model performance, it also introduces issues such as feature distortion and the creation of artificial relationships between features and the target variable. The article emphasizes the importance of understanding the trade-offs when using resampling methods and illustrates how automated data validation can be integrated into machine learning workflows using GitHub Actions.
Opinions
The author suggests that resampling techniques, while beneficial for model performance in imbalanced datasets, can lead to significant distortions in data distribution.
It is implied that the use of deepchecks is a valuable approach for identifying potential issues introduced by resampling, such as data drift and the introduction of noise.
The author expresses that data validation is a critical step in the machine learning pipeline and should be automated to ensure consistency and reliability of results.
The article conveys a preference for using SMOTE over random oversampling and undersampling, as it tends to cause fewer issues while still addressing class imbalance.
The author encourages the reader to explore the full reports generated by deepchecks for a comprehensive understanding of the dataset's characteristics and the implications of resampling.
The author advocates for the use of GitHub Actions to streamline the data validation process, making it less tedious and error-prone.
Constructive feedback from readers is welcomed by the author, indicating an openness to dialogue and collaboration within the data science community.
Investigating the effects of resampling imbalanced datasets with data validation techniques
Learn what is the impact of the popular resampling approaches to dealing with class imbalance
When dealing with imbalanced data, one of the go-to approaches is to resample the training data to reduce the class imbalance. This can involve undersampling the majority class, oversampling the minority class, or a combination of both. To make it even more interesting, there are many approaches we can follow for each of the three mentioned categories of methods.
One of the well-known disadvantages of resampling the training data is that we distort the initial distribution of features and the relationships between them. Naturally, we might be perfectly fine with that as long as the performance of our models improves. However, it would be interesting to know how severe the problem actually is. There are many ways to check that, for example, with simple visualizations (which are not feasible with a very high-dimensional dataset), or by inspecting the correlation matrix.
In this article, we will take a bit of an off-the-beaten-track approach and use the deepchecks library to compare the original dataset with the resampled one. We will investigate the difference between three resampled datasets using the popular credit card fraud dataset. Let’s dive right into it!
Data
We will be using one of the most popular imbalanced classification datasets available, that is, the credit card fraud dataset (available at Kaggle). We summarize the relevant information in the following points:
the dataset contains credit card transactions carried out over a period of two days in September 2013 by European cardholders.
the dataset is highly imbalanced — out of 284,807 transactions, 492 were identified as fraudulent. That corresponds to 0.173% of all transactions.
in order to anonymize the data, the dataset contains 28 numerical features which are the result of a PCA transformation. The only features that were not anonymized are Time (seconds elapsed between each transaction and the first transaction in the dataset.) and Amount (the transaction’s amount). This results in a total of 30 features.
Considered resampling techniques
In this article, we explore three popular resampling techniques. There are many great articles out there describing the nitty-gritty details of algorithms such as SMOTE, hence we only focus on providing a brief refresher.
1. Random undersampling
Random undersampling is the simplest of the available undersampling approaches. We draw random samples (by default without replacement) from the majority class until we obtain a 1:1 ratio of the classes. The biggest issue with this approach is the information loss caused by discarding the majority of the training set.
2. Random oversampling
The oversampling equivalent of random undersampling. Here, we draw samples with replacement from the minority class until the desired ratio of the classes is achieved. While it does not cause information loss, it comes with the danger of overfitting, which is caused by replicating observations from the minority class.
3. SMOTE
SMOTE (Synthetic Minority Oversampling Technique) is an oversampling technique that creates new, synthetic observations from the minority class. This way, the algorithm avoids the problem of overfitting encountered with random oversampling.
The algorithm executes the following steps:
choose an observation from the minority class.
identify its k-nearest neighbors.
create synthetic observations on the lines connecting (interpolating) the selected observation to the nearest neighbors.
repeat the steps above until the desired ratio of the classes is reached.
While it does not cause information loss and tries to account for the overfitting, SMOTE is unfortunately not a silver bullet to imbalanced classes. The algorithm comes with its own set of drawbacks. The biggest one is that it can introduce more noise to the data and cause overlapping of minority and majority classes (the infamous bridges between the observations). The reason for that is that SMOTE does not take into account the observations from the majority class while creating the new observations.
Deepchecks
deepchecks is an open-source Python library that allows for testing and validating both data and the performance of ML models. It does so by, among others, verifying the data integrity, inspecting distributions, validating its splits, etc.
While working with deepchecks, we can use the handy concept of suites (sets of checks), which make working with the library quite effortless. For tabular data, we can use the following suites:
single_dataset_integrity — this suite is aimed at exploring a single dataset (before any splits). It focuses on verifying the data’s integrity by, for example, checking the number of duplicated observations, if there are any issues with the categorical features, detecting outliers, etc.
train_test_validation — this suite is intended for investigating whether any data split (the most common example is the train-test split) results in representative datasets. Examples include: investigating the class distribution/balance, checking if there is no significant change in distributions between the features or labels in each of the classes, looking out for potential data leakage, etc.
model_evaluation— this suite is intended for evaluating the performance of an ML/DL model. Using it we can examine several performance metrics, compare the performance to benchmarks, etc.
We can also use the full_suite, which combines the three suites mentioned above. For our convenience, the library automatically selects which tests from the three suites are relevant for our use case.
For our example, we are mostly interested in the functionalities of the train_test_validation suite, but we will use the full_suite to also examine some data integrity checks.
Preparing the data comparisons
In this section, we describe how to process the data and resample it using the considered techniques. We start by specifying the directories for storing the data at various steps and the intended location for the data validation reports.
Our data is located in three directories:
raw — here we store the original data downloaded from Kaggle.
processed — here we store the data after the train-test split.
augmented — here we store the resampled datasets. It is important to keep in mind that when resampling, we only resample the training data!.
Additionally, we track the data using DVC. You can read more about setting up data versioning with DVC in my other article.
In the following script, we load the data from the raw directory, separate the features from the target, drop the Time feature, split the data into training and test sets (using the 80–20 ratio), and then save it to the processed directory. When working with imbalanced classes, it is crucial to apply the stratified train-test split, so the proportions of classes in the datasets are preserved.
After the split, we have also applied scaled the Amount feature using scikit-learn’s RobustScaler. As we have mentioned before, all the other 28 features are obtained from running the PCA, so we assume that their values were scaled appropriately before the transformation.
In this project, we will be using a Random Forest classifier, which does not require explicit feature scaling. However, resampling algorithms such as SMOTE use k-Nearest Neighbors (kNN) under the hood. For those distance-based algorithms the scale does matter.
In the next step, we apply various resampling techniques (available in the imbalanced-learn library) to the training data and store the results in the augmented directory. To make the code easy to extend, we defined the list of resampling techniques, and the script iterates over it. This way, we can easily experiment with other approaches to resampling, for example, ADASYN. Alternatively, we can use different parameters for the algorithms. In the case of SMOTE, we could specify the number of nearest neighbors to consider while creating the synthetic data points.
In the following excerpt from the log, we can see how the resampling impacted the size of the dataset and the class distribution. Using the default settings, all the approaches aim for a 1:1 ratio of the classes.
undersampling results ----
Shape before: (227845, 29)
Shape after: (788, 29)
Class distribution before: [0.998270750.00172925]
Class distribution after: [0.50.5]
oversampling results ----
Shape before: (227845, 29)
Shape after: (454902, 29)
Class distribution before: [0.998270750.00172925]
Class distribution after: [0.50.5]
smote results ----
Shape before: (227845, 29)
Shape after: (454902, 29)
Class distribution before: [0.998270750.00172925]
Class distribution after: [0.50.5]
In the last script, we generate the data validation reports. We identify all the distinct resampled datasets in the augmented directory and generate reports comparing those datasets to the original one. In order to use deepchecks, we assume that the training data is the original dataset, while the test data is the resampled data.
We should keep in mind that when we create a Dataset using deepchecks, we need to provide both the resampled features and labels separately.
After running the full suite of checks, we save the report into a designated directory in our project. As the reports are stored as an HTML file, they are easy to version and we can share them with stakeholders who do not have Python (and all the dependencies) installed.
In the next section, we explore some parts of those reports.
Deep-dive into the implications of resampling
In this section, we look at the outputs of the data validity checks. As the checks are very comprehensive and the reports quite lengthy, we focus on the most important/relevant parts. However, I strongly encourage you to spend some time and explore the full reports. The considered metrics are clearly described, together with the logic of the tests and their potential implications.
In general, running the suite results in the following report:
We will explore only parts of the reports, primarily in the Didn’t pass category. We look at the results in the order we presented the resampling techniques.
1. Random undersampling
In the table below, we see that quite a few checks failed.
Before diving deeper into some of them, we can address the simpler ones:
Datasets Size Comparison — this one clearly failed, as the undersampled dataset has 788 observations, while the full one has 227845
Data Duplicates — Train Dataset and Data Duplicates - Test Dataset — it seems that there are duplicates in the dataset. We should have identified the issue during EDA, but there are indeed duplicated observations in the dataset. However, this has nothing to do with resampling and they were there from the very beginning. In the report, we can see which rows (identified by their indices) are duplicated.
Train Test Samples Mix — this check investigates if observations in the test data also appear in training data. As the “test” dataset is a resampled one, there is obviously a full overlap of the observations.
Train Test Label Drift — this check verifies that the distribution of the classes is similar in both datasets. To do so, it uses Cramer’s V. This check naturally failed, as in the original dataset, the positive class is only observed in 0.173% of the observations, while in the resampled dataset the ratio is 50–50.
For our purposes, the most interesting checks are the ones concerning feature/data drift. As we have mentioned, the biggest disadvantage of resampling techniques is that they distort the distribution of the features in the datasets. Furthermore, resampling can not only obscure/alter the relationships between the features themselves, but also the relationship with the target. We can (to some extent) observe that in the following tests.
In the Train Test Feature Drift test, we can observe how the distribution of the numerical features differs between the two datasets. The accompanying drift score is simply a measure of the difference between the two distributions. In the following figure, we can see the two features that differ the most between the original and resamples datasets. It is difficult to provide any further interpretation of the differences, as the features were anonymized with the PCA.
To evaluate whether feature drift is present, deepchecks uses the earth mover’s distance (Wasserstein metric) for numerical features and Cramer’s V for categorical features. We will not go into the details of the metrics, as it can be a topic of a separate article. It is also worth mentioning that the feature drift is calculated using a sample of 100000 observations. We can change that value by passing the n_samples argument.
The second test is called Whole Dataset Drift and in it, a separate classifier is trained to distinguish between the two datasets. The features presented in the following figure are the ones that are most important for the domain classifier. The corresponding percentages of the explained dataset difference are the feature importance values calculated using permutation importance. You can read more about permutation importance here.
The last interesting test is the Feature Label Correlation Change, which uses the Predictive Power Score to estimate each feature’s ability to predict the target label.
Using the graph and the test summaries (also presented at the beginning of this section), we can draw the following conclusions:
the PPS for some of the features in the training set is above the determined threshold — normally this is an indication of data leakage, as such a feature might hold information that is based on the target label.
a large difference between the PPS for two datasets — in the traditional train-test split context, a significantly larger training PPS can be an indication of data leakage. In particular, a larger test PPS can indicate drift in the test set that causes a coincidental correlation with the target.
In the figure, we can see for which 5 features the difference between the original and resampled PPS is the largest. This can be a clear indication of the disadvantage we have mentioned before — creating fake relationships between the features and the target.
2. Random oversampling
The oversampled dataset suffers from almost the same set of issues. You can see the summary in the following figure.
Because the issues are very similar, we will only analyze the selected tests. The first one we look into is again the Feature Label Correlation Change. Compared to the random undersampling, this time all features are violating the “Train-Test features’ Predictive Power Score difference is less than 0.2” criterion. As we can see, the PPS values are more extreme for the oversampled dataset. Interestingly, both V14 and V3 features are in the top 5 impacted features for both resampling approaches.
The oversampled dataset is failing a new test, which is also related to the PPS. The Feature Label Correlation — Test Dataset test returns the PPS of all the features in relation to the target. As such, it can be used as an indication of data leakage. For this resampled dataset, 28 out of 29 features are impacted.
3. SMOTE
The dataset resampled with SMOTE also suffers from the majority of the already covered issues. Hence, we will not spend more time covering their meaning.
We will, however, mention the following differences:
the dataset resampled with SMOTE suffers from the least issues identified by deepchecks.
the Train Test Samples Mix test has a significantly different value than in the previous cases. Before, we were dealing with almost 100% overlap of the datasets, as we were either duplicating observations or randomly picking some observations from the already available ones. With SMOTE, the value is close to 50%, which intuitively makes sense as approximately half of the observations should be the new, synthetic observations generated by the algorithm.
Automating data validation with GitHub Actions
Running the data validation checks manually every time we change the data can be a bit tedious and error-prone. Thanks to GitHub Actions, we can easily automate that and many other parts of our workflow. For example, we can create an action that runs every time we modify the CSV files (original or resampled) or the data or the scripts used for generating them. Once triggered, the action loads the data versioned with DVC, runs the data validation script, commits the new data validation reports and pushes the changes to the repository. All done automatically with a short YAML script.
You can find the code I used for creating such an action here. In my other article, I have provided a detailed description of the process.
Evaluating the performance of a Random Forest model with various resampling techniques
To make the analysis complete, we also trained a Random Forest model (with the default settings) using the original data and its resampled counterparts. Below we present the evaluation results.
As we are dealing with imbalanced classes, we focused on looking at metrics such as precision, recall, and the F1-Score (the harmonic mean of the former two metrics). After analyzing the results we can state that:
Not using any resampling approaches resulted in the highest F1-Score on the test set.
Undersampling resulted in the highest recall, at the cost of very poor precision.
When it comes to the oversampling approaches, random oversampling resulted in higher precision and the F1-Score than using SMOTE. On the other hand, the synthetic observations from the minority class helped to improve the recall, at a cost of slightly worse precision.
Takeaways
In this article, we presented one of the possible approaches to investigating how various resampling techniques impact the pattern existing in data. As we have seen, resampling can lead to improved model performance when dealing with highly imbalanced classes. However, such approaches also come with a few issues of which we should definitely be aware.
For our analysis, we have used the default checks available in the deepchecks library. The library also offers the possibility of adding custom checks and modifying the thresholds used for identifying issues. This can definitely come in handy when we want to create tailor-made data validation pipelines suiting our needs.
As always, any constructive feedback is more than welcome. You can reach out to me on Twitter or in the comments. You can find all the code used for this article in this repository.
Liked the article? Become a Medium member to continue learning by reading without limits. If you use this link to become a member, you will support me at no extra cost to you. Thanks in advance and see you around!
You might also be interested in one of the following:
Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. (2002). SMOTE: synthetic minority oversampling technique. Journal of artificial intelligence research, 16: 321–357.
All images, unless stated otherwise, are by the author.