Summary

The web content provides a comprehensive guide on implementing K Fold Cross Validation for deep learning models using Pytorch and sklearn, with a focus on evaluating a CNN model on the MNIST dataset.

Abstract

The article is part of a series aimed at demystifying Pytorch for deep learning practitioners. It specifically addresses K Fold Cross Validation, a robust evaluation technique for machine learning models. The author explains the concept of K Fold Cross Validation, demonstrating its application through the training and evaluation of a Convolutional Neural Network (CNN) on the MNIST dataset. The implementation leverages Pytorch for model training and sklearn for the cross-validation process. The CNN architecture includes convolutional layers, fully connected layers, and dropout layers to mitigate overfitting. The article also covers the practical aspects of setting up the experiment, such as importing necessary libraries, defining the model, preparing the dataset, and initializing the loss function and optimizer. The author provides code snippets for each step, including the creation of data loaders, the training loop, and the evaluation of the model across different folds. The performance metrics, such as loss and accuracy, are collected and averaged over all folds to provide a general overview of the model's performance. The results indicate high accuracy without overfitting, and the article concludes with a reflection on the importance of K Fold Cross Validation and its variations for different types of problems.

Opinions

The author believes that K Fold Cross Validation is a vital technique for evaluating model performance in a robust manner.
There is an emphasis on the importance of understanding the basics of K Fold Cross Validation to effectively apply it in practice.
The author suggests that the provided implementation is a simple version and acknowledges that there are variations of the technique suitable for different scenarios, such as Stratified K Fold Cross Validation and Time Series Cross Validation.
The article implies that the use of Pytorch and sklearn together provides a powerful combination for deep learning experiments, particularly for those new to the field.
The author expresses that the tutorial series aims to make Pytorch more intuitive and accessible, addressing the potential confusion from slight differences across various online tutorials.
By providing a link to the complete code on GitHub, the author shows a commitment to open-source sharing and community contribution.
The author encourages reader engagement and support by inviting readers to become members for unlimited access to similar content and to subscribe for updates on new guides.
The author endorses an AI service, ZAI.chat, as a cost-effective alternative to ChatGPT Plus (GPT-4), indicating a belief in the value and performance of this service.

K Fold Cross Validation with Pytorch and sklearn

The post is the sixth in a series of guides to building deep learning models with Pytorch. Below, there is the full series:

Pytorch Tutorial for Beginners
Manipulating Pytorch Datasets
Understand Tensor Dimensions in DL models
CNN & Feature visualizations
Hyperparameter tuning with Optuna
K Fold Cross Validation (this post)
Convolutional Autoencoder
Denoising Autoencoder
Variational Autoencoder

The goal of the series is to make Pytorch more intuitive and accessible as possible through examples of implementations. There are many tutorials on the Internet to use Pytorch to build many types of challenging models, but it can also be confusing at the same time because there are always slight differences when you pass from one tutorial to another. In this series, I want to start from the simplest topics to the more advanced ones.

K fold Cross Validation

K fold Cross Validation is a technique used to evaluate the performance of your machine learning or deep learning model in a robust way.
It splits the dataset into k parts/folds of approximately equal size. Each fold is chosen in turn for testing and the remaining parts for training.
This process is repeated k times and then the performance is measured as the mean across all the test sets.

Implementation with Pytorch and sklearn

The K Fold Cross Validation is used to evaluate the performance of the CNN model on the MNIST dataset. This method is implemented using the sklearn library, while the model is trained using Pytorch.

Let’s start by importing the libraries and the dataset:

We define the Convolutional neural network architecture with 2 convolutional layers and one fully connected layer to classify the images into one of the ten categories. We add two Dropout layers in the model to limit the risk of overfitting.

We initialize the Cross-Entropy loss function for the classification, the device to utilize the GPU on the code and we set a fixed random number seed.

Later, we concatenate the training and the test sets into a unique dataset by applying the ConcatDataset function.

Moreover, we generate 10 folds using the Kfold function, where we have random splits and replicable results with random_state=42 . So, it divides the dataset into 9 parts for training and the remaining part for testing.

I defined the functions to train and evaluate the model before applying the K Fold Cross Validation. In particular, in the training function, we perform the forward pass and the backward pass.

Now, we can finally build the k fold cross validation procedure by iterating over folds.

In the first for loop, we sample the elements from train_idx and from val_idx and then we convert these samplers into DataLoader objects with batch size equal to 128. We also initialize the model and pass it to the GPU. In the end, we also initialize the Adam optimizer with 0.002 as the learning rate.

In the second loop, we train and evaluate the CNN model passing the functions defined before. The functions will return the loss and the accuracy for the training and test sets selected. We save all the performances into a dictionary, called history. After the training and the evaluation of the model are terminated, all the scores of the specific fold (into the dictionary history ) are memorized in the dictionary foldperf .

In the end, the model is stored using the torch.save function.

We can see the performances of the last two folds. The results seem pretty good, with 99% of accuracy in both training and test sets. Moreover, there is no evident difference between training and test accuracies, so we are not overfitting.

We can calculate the average performances to have a more generic overview with two steps:

we calculate the average score in every fold
once the average score is obtained for every fold, we calculate the average score over all the folds.

We reached good results, as we noticed before without making any average calculations.

To have additional confirmation, we can plot the average loss/accuracy curves across the ten cross-validation folds for the CNN model. In the first 4 epochs, the accuracies increase very fastly, while the loss functions reach very low values. Both curves converge after 10 epochs.

Final thoughts:

Congratulations! Now, you are able to implement the K fold Cross Validation with a model defined using Pytorch. It’s a hard task when you don’t understand the basics of this technique.

In this tutorial, I showed the simplest version of this technique, but there are also variations depending on the type of problem and the data. For example, some other approaches are the Stratified K fold Cross Validation and the Time Series Cross Validation.

The GitHub code is here. Thanks for reading. Have a nice day.

Did you like my article? Become a member and get unlimited access to new data science posts every day! It’s an indirect way of supporting me without any extra cost to you. If you are already a member, subscribe to get emails whenever I publish new data science and python guides!