Neural Networks Bias And Weights
Understanding The Two Most Important Components
This article aims to provide an overview of what bias and weights are. The weights and bias are possibly the most important concept of a neural network. When the inputs are transmitted between neurons, the weights are applied to the inputs and passed into an activation function along with the bias.
If you want to understand what neural networks are then please read:
What Are Weights?
This is an example neural work with 2 hidden layers and an input and output layer. Each synapse has a weight associated with it.
Weights are the co-efficients of the equation which you are trying to resolve. Negative weights reduce the value of an output.
When a neural network is trained on the training set, it is initialised with a set of weights. These weights are then optimised during the training period and the optimum weights are produced.
A neuron first computes the weighted sum of the inputs.
As an instance, if the inputs are:
And the weights are:
Then a weighted sum is computed as:
Subsequently, a bias (constant) is added to the weighted sum
Finally, the computed value is fed into the activation function, which then prepares an output.
Think of the activation function as a mathematical function that can normalise the inputs.
Let’s understand with a scenario:
Assume you are predicting the price of a car in dollars. Your understanding is that the price of the car is dependent on the year it was made and the number of miles it has driven.
Let’s assume that your hypothesis is that the higher the year of the car, the pricey the car. And subsequently, the more the car is driven, the cheaper the car.
The weights are essentially reflecting how important an input is.
This example should help you see that there is a positive relationship between the price of the car and the year it was made and a negative relationship between the price of the car and the miles it has been driven. As a result, we expect to see positive weight for the feature that represents year and negative weight for the feature that represents miles.
w1 is going to be positive and w2 is expected to be negative
What Is Bias?
Bias is simply a constant value (or a constant vector) that is added to the product of inputs and weights. Bias is utilised to offset the result.
The bias is used to shift the result of activation function towards the positive or negative side.
Imagine this scenario:
Let’s assume you want your neural network to return 2 when the input is 0. As the sum of product of weight and input is going to be 0, how will you ensure the neuron of the network returns 2?
You can add a bias of 2.
If we do not include the bias then the neural network is simply performing a matrix multiplication on the inputs and weights. This can easily end up over-fitting the data set.
The addition of bias reduces the variance and hence introduces flexibility and better generalisation to the neural network.
Bias is essentially the negative of the threshold, therefore the value of bias controls when to activate the activation function.
If you want to understand what activation functions are then please read:
If you want to understand what neural network layers are then please read:
If you want to understand how neural network neurons work then please read:
Summary
This article explained what weights and bias are and how they can help a neural network perform better forecasts.
Hope it helps
