Understanding Polynomial Regression

Capturing non-linear relationships | Data Series | Episode 4.6

This Article expands on Simple Linear Regression and Multiple Linear Regression, ensure you have a good understanding of these two topic areas before continuing.

What is Polynomial Regression?

Polynomial Regression is used to capture non-linear relationships between variables.

For example:

For linear relationships we use Linear Regression.

Overview

Take a look at the following graph looking at the Humidity and Pressure values in Svged, Hungary.

• We can see there is a trend in the data, which is non-linear so we use Polynomial Regression
• The job of Polynomial regression is to find a suitable relationship between Humidity and Pressure, such as the following:

• We can then use the model produced by polynomial regression to make suitable predictions for Humidity given any Pressure value.

Calculating the Polynomial Regression Model

Preprocessing The Data

The first step in calculating our Polynomial Regression Model is to Preprocess our data.

Take a look at the following data used to plot our Humidity vs Pressure Graph:

Our Regression Model is given by:

Which is linear as the highest power of 𝑥 is 1.

Let’s now square the pressure data and a new column:

Our Regression Model is now given by:

Which is non-linear as the highest power of 𝑥, which is 2, is greater than 1.

— — — — — — — — — — — — — — — — —

Adding 𝑥² to our regression model enables the model to check: Is there a second degree (bowl shape):

relationship between Pressure and Humidity?

— — — — — — — — — — — — — — — — —

Applying Gradient Descent

In order to calculate the values for θ₀, θ₁ and θ₂ we need apply an algorithm called gradient descent.

After applying this algorithm we find:

θ₀ = 29167.74604378328 θ₁ = -57.722545421135 θ₂ =0.02855835729673671

Giving our final polynomial regression model of:

When plotted gives:

Which seems to fit our data quite nicely.

Using our model to make predictions

Let’s say we were given a pressure reading of 1007 millibars and wish to find the kind of Humidity we can expect from this.

We simply input 1007 into our linear regression model:

Which gives a Humidity value of: 0.716

Looking at our graph, this humidity value looks reasonable:

Evaluating our Model

We can see that adding the extra feature 𝑥² to our model seems to capture our relationship between pressure and humidity better than simple linear regression.

We can show this mathematically by comparing each models mean squared error given (MSE) given by:

This formula shows the average distance our regression model predictions(ŷ) are away from all it’s points(y). We discuss this in episode 4.1, where we look at the cost function.

Simple Linear Regression Model MSE: 0.01686

Polynomial Regression Model MSE: 0.00336

We see here that our polynomial regression MSE is much lower and therefore closer to all our data points. When looking at our model visually we also see it captures the general curve in our data and is therefore the model we should go for.

When adding higher and higher degree features to our model:

Our MSE decreases.

We may think that we should go for the model with the lowest MSE, however take a look at the following model which has an extremely low MSE:

Does this capture the relationship we are looking for?

Here, we start to run into a problem called Overfitting, where we add so many features to our model — that it fits our data too well and fails to recognise the general trend.

Overfitting and Underfitting are common problems in Data science and is important to consider when evaluating our model.

This will be discussed after implementing our polynomial regression model in Python in the next episode.

Prev Episode | Next Episode

If you have any questions please leave them below!