avatarAmit Chauhan

Summary

The provided content explains the concept of Simple Linear Regression, a machine learning algorithm used to find the linear relationship between independent and dependent variables.

Abstract

The content begins by defining Linear Regression and its components, including the slope and intercept of a line. It then delves into the types of linearity, namely positive and negative linearity. The article further discusses the concept of slope and intercept, and how the objective of linear regression is to minimize the error by optimizing these parameters. The loss functions, such as L1 loss, L2 loss, RSME Error, and R-squared error, are explained in detail. The content also covers how to find the optimal value of slope and intercept using numerical methods or the gradient descent method. The concept of gradient descent, learning rate, and vectorization technique are also discussed.

Opinions

  • The article emphasizes the importance of minimizing error in linear regression to achieve high accuracy in predictions.
  • The author suggests that the choice of loss function depends on the specific use case and the presence of outliers in the data.
  • The article highlights the importance of choosing an optimal learning rate in gradient descent to avoid missing the minima point or taking too long to reach it.
  • The author advocates for the use of vectorization techniques to speed up the training process in large datasets.
  • The article implies that linear regression is a fundamental concept in machine learning and forms the basis for understanding more complex algorithms.
  • The author suggests that understanding the concept of local minima and maxima from school math can be helpful in understanding the gradient descent method.
  • The article implies that linear regression is a simple yet powerful tool for understanding the relationship between variables in a dataset.

Machine Learning: Explanation of Simple Linear Regression Algorithm

A branch of artificial intelligence

A Simple Linear Regression concept. A photo by Author

What is Linear Regression?

The main working function of the linear regression algorithm is to find the linear relationship between the independent and dependent variables. In machine learning the data is visualized with graphs and gives the intuition of dependency of one variable on another variable in a regress manner i.e. the procedure can be reapplied on the same method. Here, our method is a linear estimator that estimates the result of linear combination.

The estimator is a kind of method or formula with which we can estimate something.

If we break the linear regression into two words i.e. linear is a straight line and regression is a way to go back to the previous state. But our real-life data is not perfect to be linear so the regress method is trying to be perfect with some error.

The equation of a straight line can be in a standard form or with slope-intercept form. For linear regression in classic machine learning, we will be more focused on slope-intercept form as shown below.

Y = mx + c or Y = θ1x + θ0

Where,

m or θ1 = Slope of the line

c or θ0 = intercept of the line

x = independent variable

Y = dependent variable

Types of linearity

  1. Positive Linearity: This type of linearity is defined by the positive increase effect with respect to its cause value.
  2. Negative Linearity: This type of linearity is defined as a negative decrease effect with respect to its cause value.

In a simple linear regression model, the slope (m) and intercept (c) parameters in the formula are the adjustable parameters to define the line. In machine learning, the main idea is to reduce the error of the algorithm so that we can get a good prediction with high accuracy.

Let’s take an example with one independent and one dependent variable.

In the above example, i is a number of samples, x is an independent variable (feature) and y is a dependent variable (target). Our aim is to find the coefficients i.e. m or θ1 and c or θ0 so that we can predict the water sold based on the temperature.

A simple machine learning pipeline flows as shown below.

Concept of slope (m) and intercept (c)

The m is a slope i.e. change in degree and c is an intercept i.e. cutting the y-axis somewhere.

The objective of the linear regression algorithm is to choose the optimized value of slope and intercept so that the line fits the data with less error.

Loss functions

Suppose the model is trained and gives the predicted output then the loss is the difference between the predicted values and actual data values.

Type of loss in a linear model

  • L1 loss: This is the difference between the predicted and actual values. It is also called mean absolute error (MAE).

The model will calculate all the MAE values and add them to find the total L1 Loss. The formula of L1 loss is shown below.

  • L2 Loss: In this loss, we take the squared average difference between the predicted and actual value. It is also known as Mean Squared Error (MSE). The formula of L2 loss is shown below.
  • RSME Error: It tells the error rate by the square root of the L2 loss i.e. MSE. The formula of RSME is shown below.
  • R-squared error: It tells the good fit of the model-predicted line with the actual values of data. The coefficient value range is from 0 to 1 i.e. the value close to 1 is a well-fitted line. The formula is shown below.

Note: In the case of an outlier, we can use L1 losses because with L2 loss the error is being squared to give more loss value. We can remove the outlier from the first and then can use L2 loss.

The cost function used in linear regression in classical machine learning algorithms is L2 loss also called the least square cost function. The loss function is minimum with the optimal value of θ1 and θ0 i.e. slope and intercept.

How to get the optimal value of slope and intercept?

  1. By normal numerical method
  2. By gradient descent method

Before finding the value of slope and intercept, we should know the curvature of the loss (L2) function. I am taking sample data for an explanation as shown in the below image.

This line seems to be a line of best fit.

If we assume the intercept value is zero i.e. θ0 = 0 then the line will intercept at the origin. The convex method is used to calculate the slope of the line with minimum error. The reason behind the convex method is used to get the one minima point with zero gradients. The gradient means the change is stopped at the minima point.

How to get the minima point?

The machine learning model will initialize the line as shown in the below figure.

Then it will calculate the loss with this line and the error will be high and is represented in the curvature diagram below.

Like this, the model will find the minimum error point by fitting the line multiple times as we increase the angle of the line. The wholesome idea of getting the minima point to get the minimum loss function is shown in the below image.

The L2 loss function has a property to have a minimum point.

The method of getting the from high error point to low error point to get the minima point is called gradient descent. The gradient means changes and descent means to go to the origin point and going down or going backward.

If we have a non-convex function as shown in the below image then it might be possible to get stuck in at the local minima rather than global minima and our model will not give the best fit line. So we try to make the curvature to convex function.

How to read Contour plot of Gradient Descent

Here in the below image, we have one global minimum and many local minima. We see the lines in the contour plot and it differs according to the data.

Non-convex plot. Image source

The gap between the lines tells the steepness of the slope whether it is more or less. If the space is more the speed is less to while going down but if the space is less the speed is more towards the minima point.

A general contour plot. A photo by Author

There are different gradient descent to tackle the many local minima points in the curvature.

When we were learning about the concept of local minima and local maxima in school math then we take the derivative of the function and put equal to zero to get the minima point.

The formula to get the optimal value of the slope is shown below.

When the point is going down from the right side then the value of slope comes positive and when we put that value in the formula, we go more to the left side i.e. towards minima point. Likewise, if we have a point on the left side we get a negative slope and when we put this value in the formula then the value becomes positive with the negative sign in the formula and the new value goes towards the right side i.e. towards minima point.

Concept of updating the parameters in linear regression. A photo by Author

When our slope becomes zero then the new value will become equal to the old value and our loss will be minimum.

The minima point. A photo by Author

Learning Rate

The alpha is the learning rate in the gradient descent formula as we seen above. It functions of the alpha to control the speed of the gradient descent to get the minima point. The value of alpha should be optimal so that it won’t miss the minima point or take time to reach the minima point.

When we choose the too low value of alpha then it takes very much time to get the minima point.

When we choose the high value of alpha then it might be a chance to miss the minima point and jump to the sides.

Vectorization Technique

When we get our slope and intercept equation then we also get a summation sign that slows down the training in large data to update the slope and intercept parameters. So with the help of linear algebra, the vectorization of the matrix in gradient descent becomes faster.

I hope you like the article. Reach me on my LinkedIn and twitter.

Recommended Articles

1. 8 Active Learning Insights of Python Collection Module 2. NumPy: Linear Algebra on Images 3. Exception Handling Concepts in Python 4. Pandas: Dealing with Categorical Data 5. Hyper-parameters: RandomSeachCV and GridSearchCV in Machine Learning 6. Fully Explained Linear Regression with Python 7. Fully Explained Logistic Regression with Python 8. Data Distribution using Numpy with Python 9. Decision Trees vs. Random Forests in Machine Learning 10. Standardization in Data Preprocessing with Python

Machine Learning
Data Science
Programming
Artificial Intelligence
Python
Recommended from ReadMedium
avatarIrina (Xinli) Yu, Ph.D.
Understanding Hypothesis Testing: T-Test, Z-Test, Chi-Square Test, and ANOVA

Hypothesis testing is a critical component of statistical analysis, allowing researchers to make inferences about populations based on…

4 min read
avatarDr. Walid Soula
L1 Regularization : Lasso Regression

Discover Lasso Regression, an L1 regularization technique to prevent overfitting in linear models, optimizing accuracy by reducing errors

7 min read
avatarThomas A Dorfer
Bagging vs. Boosting: The Power of Ensemble Methods in Machine Learning

How to maximize predictive performance by creating a strong learner from multiple weak ones

6 min read
avatarPiero Paialunga
Hands-on Time Series Anomaly Detection using Autoencoders, with Python

Here’s how to use Autoencoders to detect signals with anomalies in a few lines of codes

10 min read
avatarLuqman Zaceria
Understanding Convolutional Neural Networks (CNNs)

Hey everyone! We’re going to explore one of the most influential and powerful tools in the world of deep learning: Convolutional Neural…

16 min read
avatarFederico Calabrò
Regression Task with Feedforward Neural Networks

Health Insurance Charge Prediction

6 min read