Noise Regularization of Neural Networks

Neural Networks and Deep Learning Course: Part 26

Adding noise is a regularization technique that falls under the category of “Other methods” in neural network regularization techniques.

Here, we expand the training data by generating new data. We take each training instance one by one and create different versions of them by adding a small amount of noise to the original training instances. That results in the following benefits.

• By expanding the training data, we increase the size of the training dataset. Training a neural network model on a large dataset increases the performance of the model.
• By adding noise to the data, the model will not solely capture noise in the training data and will generalize well on new unseen data. That results in reducing overfitting.

Note: More specially, noise regularization is effective in reducing overfitting in neural networks when training the model with a small dataset.

In a neural network mindset, adding noise to the training instances is considered as adding noise to the input layer of the network because the input layer holds the training data. Just like we add noise to the input layer, we can also add noise to the hidden layers and the output layer. In case of adding noise to the hidden layers, we implicitly add noise to the weights or activations (if any). In case of adding noise to the output layer, we implicitly add noise to the final output.

The major challenge here is deciding where (input/hidden/output layers) to add noise in the network. That can be addressed by using experience or observing results by trying different network configurations. That also differs from problem to problem.

Implement noise regularization in Keras

Defining noise

In Keras, noise is defined as a layer called the GaussianNoise layer.

from tensorflow.keras.layers import GaussianNoise

noise_layer = GaussianNoise(stddev=0.1)

The noise always has a zero mean and we need to specify a value for the standard deviation as a hyperparameter.

After defining the noise as a layer, we need to add it to the model. Here are some examples of adding noise to Keras models.

Examples of adding noise to Keras models

• Example 1: Adding noise to the input layer of an MLP.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import GaussianNoise

model = Sequential()

model.add(GaussianNoise(0.1, input_shape=(784,)))
model.add(Dense(512, activation='relu'))
...

Here, the noise is added between the input layer and the first hidden layer. If this is not clear to you, defining the input layer explicitly makes this clear for you.

from tensorflow.keras.layers import InputLayer

model = Sequential()

model.add(InputLayer(input_shape=(784,)))
model.add(GaussianNoise(0.1))
model.add(Dense(512, activation='relu'))
...

• Example 2: Adding noise to a hidden layer of an MLP.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import GaussianNoise

model = Sequential()

model.add(Dense(512, input_shape=(784,), activation='relu'))
model.add(GaussianNoise(0.1))
model.add(Dense(256, activation='relu'))
...

Here, we add noise between the first and second hidden layers. That’s actually right after the first hidden layer. Because the activation is also defined in the first hidden layer as a parameter, we add noise to the activated output of the first hidden layer.

When we add noise in this way, there is a problem. The problem is that the output values of that hidden layer will not be kept in a certain range because of the added noise. To overcome this problem, we can apply activation to that layer after adding noise in the following way. The activation function in the hidden layer is responsible to keep the output values of that hidden layer in a certain range.

from tensorflow.keras.layers import Activation

model = Sequential()

model.add(Dense(512, input_shape=(784,)))
model.add(GaussianNoise(0.1))
model.add(Activation('relu'))
model.add(Dense(256, activation='relu'))
...

• Example 3: Adding noise to a pooling layer of a CNN.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D
from tensorflow.keras.layers import GaussianNoise

model = Sequential()

model.add(Conv2D(16, (3,3), activation="relu"))
model.add(MaxPooling2D())
model.add(GaussianNoise(0.1))
model.add(Conv2D(32, (3,3), activation="relu"))
model.add(MaxPooling2D())
...

Here, we add noise to the first pooling layer of the CNN model.

• Example 4: Adding noise to the output layer of an MLP.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import GaussianNoise
from tensorflow.keras.layers import Activation

model = Sequential()

model.add(Dense(512, input_shape=(784,), activation='relu'))
model.add(Dense(256, activation='relu'))
model.add(Dense(10))
model.add(GaussianNoise(0.1))
model.add(Activation('softmax'))

Here, we apply activation to the output layer after adding the noise to it.

Note: Adding noise will not alter the output shape of each layer.

This is the end of today’s post.

Please let me know if you’ve any questions or feedback.

I hope you enjoyed reading this article. If you’d like to support me as a writer, kindly consider signing up for a membership to get unlimited access to Medium. It only costs $5 per month and I will receive a portion of your membership fee.

Thank you so much for your continuous support! See you in the next article. Happy learning to everyone!

Read next (Recommended)

• Overview of Regularization Techniques for Neural Networks
• Apply L1 and L2 Regularization to Keras Models
• Use Dropout Regularization in Neural Networks
• Using Early Stopping to Reduce Overfitting in Neural Networks

Rukshan Pramoditha 2022–07–26