Python Machine Learning

Part 1 of a series showing how to build neural networks from scratch.

This story is meant to be an introduction to artificial intelligence (AI). Python is a great language to learn since most of the tools are built using it. Deep learning is a technique used to make predictions using data, and it heavily relies on neural networks. This story is part of a is a series leading up to building a neural network from scratch.

In a production setting, you would use a deep learning framework like TensorFlow or PyTorch instead of building your neural network. That said, having some knowledge of how neural networks work is helpful because you can use it to better architect your deep learning models.

Artificial Intelligence Overview

In basic terms, the goal of using AI is to make computers think as humans do. This may seem like something new, but the field was born in the 1950s.

Machine Learning

Machine learning is a technique in which you train the system to solve a problem instead of explicitly programming the rules.

A common machine learning task is supervised learning, in which you have a dataset with inputs and known outputs. The task is to use this dataset to train a model that predicts the correct outputs based on the inputs.

One of the key aspects of supervised machine learning is model evaluation and validation. When you evaluate the predictive performance of your model, the process must be unbiased. Using de>train_test_split() from the data science library scikit-learn, you can split your dataset into subsets that minimize the potential for bias in your evaluation and validation process.

The Importance of Data Splitting

Supervised machine learning is about creating models that precisely map the given inputs (independent variables, or predictors) to the given outputs (dependent variables, or responses).

The goal of supervised learning tasks is to make predictions for new, unseen data. To do that, you assume that this unseen data follows a probability distribution similar to the distribution of the training dataset. If in the future this distribution changes, then you need to train your model again using the new training dataset.

Training, Validation, and Test Sets

Splitting your dataset is essential for an unbiased evaluation of prediction performance. In most cases, it’s enough to split your dataset randomly into three subsets:

1. The training set is applied to train, or fit, your model. For example, you use the training set to find the optimal weights, or coefficients, for linear regression, logistic regression, or neural networks.
2. The validation set is used for unbiased model evaluation during hyperparameter tuning. For example, when you want to find the optimal number of neurons in a neural network or the best kernel for a support vector machine, you experiment with different values. For each considered setting of hyperparameters, you fit the model with the training set and assess its performance with the validation set.
3. The test set is needed for an unbiased evaluation of the final model. You shouldn’t use it for fitting or validation.

In less complex cases, when you don’t have to tune hyperparameters, it’s okay to work with only the training and test sets.

Underfitting and Overfitting

Splitting a dataset might also be important for detecting if your model suffers from one of two very common problems, called underfitting and overfitting:

1. Underfitting is usually the consequence of a model being unable to encapsulate the relations among data. For example, this can happen when trying to represent nonlinear relations with a linear model. Underfitted models will likely have poor performance with both training and test sets.
2. Overfitting usually takes place when a model has an excessively complex structure and learns both the existing relations among data and noise. Such models often have bad generalization capabilities. Although they work well with training data, they usually yield poor performance with unseen (test) data.

Prerequisites for Using train_test_split()

scikit-learn, or sklearn has many packages for data science and machine learning. We will focus on the model_selection package, specifically on the function train_test_split().

Application of train_test_split() in supervised machine learning.

Start with a small regression problem that can be solved with linear regression.

Example of Linear Regression

This example will show how to solve a regression problem. Start by importing the necessary data structures from the other libraries,

import numpy as np from sklearn.linear_model import Linear Regression from sklearn.model_selection import train_test_split

Then create two small arrays, x and y, to represent the observations and then split them into training and test sets. I had to install scikit at levels 0.2.4.2 to get the linear regression algorithm.

$ python -m pip install -U "scikit-learn==0.24.2"

LinearRegression creates the object that represents the model, while .fit() trains, or fits, the model and returns it. With linear regression, fitting the model means determining the best intercept (model.intercept_).

The below text in bold represents the python source code. While the text in italic is the program output.

import numpy as np from sklearn.linear_model import LinearRegression from sklearn.model_selection import train_test_split

x = np.arange(20).reshape(-1, 1) x y = np.array([5, 12, 11, 19, 30, 29, 23, 40, 51, 54, 74, 62, 68, 73, 89, 84, 89, 101, 99, 106])

y

array([[ 0],
       [ 1],
       [ 2],
       [ 3],
       [ 4],
       [ 5],
       [ 6],
       [ 7],
       [ 8],
       [ 9],
       [10],
       [11],
       [12],
       [13],
       [14],
       [15],
       [16],
       [17],
       [18],
       [19]])

x_train, x_test, y_train, y_test = train_test_split( x, y, test_size=8, random_state=0)

model = LinearRegression ().fit(x_train, y_train)

model.intercept_ 3.1617195496417523

3.1617195496417523

model.score(x_train, y_train)

0.9868175024574795

model.score(x_test, y_test)

0.9465896927715023

.score() returns the coefficient of determination, or R², for the data passed. Its maximum is 1. The higher the R² value, the better the fit. In this case, the training data yields a slightly higher coefficient. However, the R² calculated with test data is an unbiased measure of your model’s prediction performance.

The combination of the training data with the machine learning algorithm creates the model. Then, with this model, you can make predictions for new data.

Tune in for the next pair in this series, where I will discuss deep learning.