Hebbian Learning: Neurons That Fire Together, Wire Together

Bio-Inspired Design

If we want to talk about Hebbian Learning, we need to start talking about the more general concept of bio-inspired design. Bio-inspired design, also known as biomimicry, is an innovative approach that takes cues from nature to solve human challenges. It involves studying biological systems and using the insights gained to create sustainable solutions across various domains.

Consider the case of high-speed trains and kingfishers [1]. For trains, air resistance in their passage is a serious problem because it causes a lot of noise and can prevent very high speeds from being reached. Engineers studying the kingfisher’s beak thought that designing a train taking inspiration from its shape could solve the problem, as shown in Figure 1.

The kingfisher’s beak is indeed streamlined and tapered, minimizing air resistance. This shape allows the bird to move through the air with minimal drag, reducing the force that opposes its forward motion.

In this case, the best engineering solution has been taken by the observation and imitation of nature.

Can we take inspiration from nature to improve AI systems?

Bio-Inspired Artificial Intelligence

In recent years, we have witnessed a continuous and unstoppable growth of artificial intelligence-based systems, and the vast majority of these impressive advances stem from deep learning. This discipline, which has its roots in neural networks, has revolutionized how machines learn and interpret complex information.

Are the rules of nature behind this success? And can we exploit them more to boost the power of AI?

In that sense, bio-inspired Artificial Intelligence, refers to a field of Artificial Intelligence that draws inspiration from biological systems and processes found in nature. The term “bio-inspired”, exactly like in biomimicry, implies that researchers and developers in AI are looking to the mechanisms, structures, and strategies employed by living organisms to design and enhance artificial intelligence systems. And what is the equivalent of kingfisher for neural networks?

Obviously, the human brain!

The human brain, with its intricate neural networks and the ability to process information in a highly parallel and efficient manner, serves as a particularly rich source of inspiration for bio-inspired AI. By mimicking biological principles, such as the way neurons communicate or the adaptive learning observed in living organisms, researchers aim to create AI systems that exhibit enhanced cognitive abilities, adaptability, and efficiency.

In essence, bio-inspired AI seeks to replicate or simulate certain aspects of biological systems in the development of artificial intelligence algorithms and models. This approach offers new perspectives and potential breakthroughs in creating intelligent systems that can better handle complex tasks.

Neural networks which are the basic idea behind deep learning, are bio-inspired, modelled after the structure and functioning of the human brain. They mimic the interconnected neurons and learning processes observed in biological systems. In a biological neuron, represented in Figure 2, there are the following main parts:

• Dendrites: Neurons receive signals from other neurons through their dendrites. Dendrites are branching extensions of the neuron that act as receivers, collecting incoming information in the form of electrical signals released by neighbouring neurons.
• Cell body and Nucleus: The cell body, also known as the soma, integrates the incoming signals from the dendrites. It contains the cell nucleus, which houses the genetic material (DNA) and regulates the activities of the cell. The nucleus plays a critical role in maintaining the cell’s functions and orchestrating its responses.
• Axon: Once the signals are integrated into the cell body, the neuron transmits the resulting electrical signal along its axon. The axon is a long, slender projection that serves as the neuron’s output pathway.
• Axon Terminals: The axon terminates in small structures called axon terminals. These terminals form connections, or synapses, with dendrites of other neurons.

An artificial neuron, which is the base element composing an artificial neural network, tries to emulate this behavior. Figure 3 shows the mathematical representation of an artificial neuron in which:

• Inputs: x_0, …, x_N are the input signals, which can be any kind of feature. These are the equivalent of the biological signals in the human brain.
• Weights: w_0, …, w_N are the weights on each connection (which is the equivalent of a dendrite) which will be updated through the backpropagation process and represent the “power” of the connection influencing the input coming into the neuron.
• Weighted sum: All the weighted products are summed together and also with a bias term (b) which allows the neuron to learn an offset or a shift in its activation. This happens inside the neuron, which is the equivalent of the cell body.
• Activation Function: The result of the weighted sum plus the bias term is then passed through an activation function (f), introducing non-linearity to the model.

Interest in other bio-inspired AI approaches arises from the recognition that these neural networks, while powerful, have limitations. Biological systems encompass intricate biochemical processes, energy-efficient mechanisms, and specialized adaptations. Exploring alternative bio-inspired methods may unlock new possibilities for creating AI systems that are more efficient, robust, and adaptable to specific tasks and environments.

The main limitation of classical Neural Networks resides in the way they learn. Consider a generic Neural Network composed of a variable number of neurons, each one constructed as previously explained.

Learning in these kinds of structures occurs through two main phases: the forward pass, where data passes through the network to produce an output, and backpropagation, where errors are retroactively propagated through the network to update weights and improve model performance.

The backpropagation algorithm involves iteratively adjusting weights based on a chosen loss function, which calculates the “difference” between the system prediction and the ground truth. This iterative process refines the network’s ability to make accurate predictions over time.

This means that the classical neural networks learn from their errors.

Backpropagation is then very suitable in cases of supervised learning, even if it is also successfully used in unsupervised contexts by creating ad-hoc loss functions.

However, this methodology, although powerful, departs somewhat from the complexity and flexibility of biological neural networks, whose learning mechanisms have been studied since the early days of neuroscience. Indeed, training these networks may be pretty difficult if we can’t rely on well-annotated data which allows us to confidently assess the network performances.

Hebbian Learning

A bio-inspired AI approach that is proving quite successful in certain areas and is attracting the attention of many researchers is Hebbian Learning.

Hebbian Learning is a bio-inspired learning paradigm that takes inspiration from the observed principles of synaptic plasticity in the human brain. The term “Hebbian” comes from the psychologist Donald Hebb, who introduced the concept in 1949 with the famous phrase, “Neurons that fire together, wire together”.

In biological systems, particularly in the brain, synaptic plasticity refers to the ability of the connections (synapses) between neurons to strengthen or weaken based on the patterns of neural activity. As shown in Figure 5 the connection between two neurons is strengthened as the signals pass between them.

Hebbian Learning captures this phenomenon, where the strengthening of a synapse occurs when the connected neurons are activated simultaneously. The biological process underlying Hebbian Learning is tied to the idea that when a neuron consistently triggers the firing of another neuron, the connection between them strengthens. This strengthening is thought to be a mechanism for encoding and retaining information in the neural network.

In simple terms, if neuron A repeatedly stimulates neuron B, the synapse connecting them strengthens, making it more likely that the activation of neuron A will lead to the activation of neuron B in the future.

In the context of bio-inspired AI, Hebbian Learning seeks to replicate this observed synaptic plasticity to enable artificial neural networks to adapt and learn from correlated patterns in data without relying on explicit error signals. Figure 6 shows an example of Hebbian Learning applied to a bunch of connected artificial neurons. The connections between them are strengthened (and so their weights are updated) when they are active together. Mathematically this the rule to update the weights can be formalized as:

Where 𝜂 is the learning rate which expresses how much update the weight. Input values x are also termed pre-synaptic activations, while the output 𝑦 is termed post-synaptic activation and it is obtained by applying the activation function to the input.

This is pretty much different from backpropagation, which instead updates the weights based on the errors made by the network. Hebbian Learning is particularly relevant for unsupervised learning scenarios, where the network learns from the inherent structure of the input data without requiring annotations but trying to learn the patterns in the data.

Hebbian Learning VS Backpropagation

But should we switch to Hebbian Learning? Is it a better way to make our networks learn? Let’s see the main differences between it and backpropagation.

Hebbian Learning

• Biological Inspiration: Hebbian Learning is inspired by biological synaptic plasticity observed in the brain. It reflects the idea that neurons that fire together strengthen their connections, emphasizing a more biologically plausible learning mechanism.
• Parallelism and Online Learning: Hebbian Learning is inherently parallel and can adapt to new data in an online fashion. It doesn’t require an explicit phase of backward passes and can update weights immediately based on input correlations.
• Unsupervised Learning: Well-suited for unsupervised learning scenarios where the network identifies patterns and associations in data without explicit labels. It excels in scenarios where labeled training data is limited or unavailable

Backpropagation

• Iterative and Rigorous Optimization: The iterative nature of backpropagation ensures a systematic error reduction over multiple passes through the training dataset. This approach is well-suited for fine-tuning weights in complex neural architectures.
• Effective for Complex Tasks: Backpropagation, especially when combined with gradient-based optimization algorithms, is highly effective for training deep and complex neural networks. It can handle intricate relationships within data and learn hierarchical representations.
• Global Error Minimization: Backpropagation is designed for supervised learning, aiming to minimize the global error between predicted and target outputs. This explicit error-driven approach allows the network to adjust its weights to improve overall performance.

So, as often happens, the answer to the question “Should we always use Hebbian Learning instead of Backpropagation?” is: “It depends!”.

Hebbian Learning excels in unsupervised and biologically inspired settings, providing adaptability and simplicity. Backpropagation, on the other hand, is powerful for supervised tasks (even if it can be used in any other context), enabling precise optimization and effective training of complex neural networks. The choice between them depends on the nature of the learning task and the availability of labeled data.

It is also possible to approximate the backpropagation algorithm using Hebbian Learning with methods like Contrastive Hebbian Learning, Equilibrium Propagation, and Predictive Coding.

Hopfield Networks

One interesting application of Hebbian Learning is the Hopfield Networks. They are a type of Recurrent Neural Network (RNN) inspired by Hebbian learning principles. The key idea behind the Hopfield Networks lies in its ability to store and retrieve patterns through the modification of synaptic weights. For that reason, the Hopfield Networks are considered a model of the function of biological memory.

In this network, learning occurs by adjusting the connections between neurons based on the correlated activation of these neurons. So, Hopfield Networks can be used by showing them a set of patterns (such as Figure 5) that you want to memorize and it will adjust the weights by applying Hebbian Learning.

Considering these patterns as visual ones and representing an image as a matrix in which each cell is a pixel, we will need a number of neurons equal to the number of pixels, to represent every possible pattern with that shape. As shown in Figure 7, these neurons are all connected and the connection is bi-directional.

In particular, the network’s weights are determined by the simultaneous activation of neurons during the learning phase. As a result, the network can associate certain states or patterns with specific configurations of synaptic weights.

During the retrieval phase, when the network is presented with a distorted input, it strives to converge towards a stored pattern that closely matches the input. This convergence process involves iterative updates of neuron activations, guided by the established synaptic weights.

The network operates recurrently, meaning it can cycle through neuron activations until it reaches a stable state corresponding to a memorized pattern. The dynamics of the network are driven by an energy function, which decreases as the network approaches a stable state. The objective is to minimize this energy, indicating that the network has settled into a recognizable pattern.

Conclusions

In conclusion, the exploration and comprehensive understanding of the human brain’s capabilities remain ongoing challenges in the scientific field. Even today, we have only scratched the surface of the intricate workings of the human mind. However, the study of the brain serves as inspiration for enhancing and evolving artificial intelligence systems. By delving deeper into the mechanisms that drive cognitive processes, we can uncover novel insights that pave the way for the development of more sophisticated AI systems.

Hebbian learning is just an example of what we can do by applying neuroscience to artificial intelligence, representing an initial stride into a vast universe of possibilities.

So, in the continuous research of innovation in the field of AI, never forget to keep an eye on neuroscience.

References

[1] EarthSky: “A kingfisher inspired a bullet train”

[2] Lagani G. et al: “Synaptic Plasticity Models and Bio-Inspired Unsupervised Deep Learning: A Survey”

[3] Muhammad Abduh: “Hopfield Networks”

[4] Lagani G. et al: “Training Convolutional Neural Networks with Competitive Hebbian Learning Approaches”

[5] Lagani G. et al: “Comparing the performance of Hebbian against backpropagation learning using convolutional neural networks”