Are Hopfield networks the key to better understanding our brain?
Hopfield networks are a particular type of neural algorithm that proposes an alternative idea on how our brain stores memories.
As part of my Ph.D. research, I had to delve into recurrent neural networks. These are neural algorithms that integrate the usual input-output connections with feedback connections. In other words, the signal can move back and forth inside the network. It seems a confusing dynamic, and it is. Still, it turns out that these recurrent connections may be the keystone for unlocking the true power of artificial intelligence. In this context, Hopfield networks are probably the most intriguing ones.
What is a Hopfield network?
John Hopfield is an American scientist that in 1982 published a highly influential paper [1] (more than 24K citations) proposing a novel type of associative memory. A memory is associative when it can retrieve an entry starting from random pieces of it. Consider the phrase “This article is purely awesome, said the author”. You want to retrieve this phrase from a memory storage. Classical memories in standard computers require a precise, sequential sequence of words to find it (This article or purely awesome). Associative memories break the sequentiality requirement. A user can input random words (like article, awesome, and author) to retrieve the entire phrase. Hopfield networks achieve this by leveraging the properties of a graph where nodes are entirely connected with the others. In Hopfield’s vision, the nodes are prototypical neurons that can be either active or inactive. This state depends directly on the other states and the weights of the connections. But, wait a second. This dependency also applies to the other nodes because the network is fully-connected. Each node state depends on the others, and the network is circular. Therefore, each node update requires updating the other nodes. It seems like a never-ending cycle, a dog eating its own tail! In reality, John discovered that, in particular circumstances, the process eventually stops in a final configuration.
Energy and stability.
The brilliance of Hopfield was in realizing that there exists a global energy function of the system in case the connections are symmetric. What does this mean? The connection importance (the weight between two nodes) does not depend on the order we consider the nodes. Going from node vi to node vj has the same weight as going from node vj to node vi. Once you impose this rule, it is possible to define a global mathematical function describing the “energy” of the system. We multiply the state of two nodes by the weight of their connection. We iterate this procedure for each pair of nodes and then sum all the results together. Intuitively, we want to mimic the behavior of biological neurons. We observe in the brain that neurons link together when they transmit the signal at the same time. Here the energy function measures a similar concept. When two nodes have the same state, a positive weight lowers the energy, and a negative one heightens it. The converse is true for discord states. Hence, lower energy means higher “biological stability” of the network. Once you reach the lowest point, the network finds a resting place, and the operation ends. In his paper, Hopfield proposes a method to update the nodes’ states to reach these stable positions.
Stable points and how to find them.
The algorithm to find stable points is actually quite simple. Choose a random node vi in the network. For each connection involving the node, multiply the weight wij for the state vj of the node on the other side. Sum all these values together. If the final sum exceeds a fixed threshold (that we conveniently set to 0), the state vi becomes 1. Otherwise, it becomes -1. That’s it, simple. Again, the procedure has biological meaning. Brain neurons forward a signal whenever they receive enough input from preceding neurons. And indeed, we set our node to 1 when the sum of all the contributions exceeds a threshold value. The procedure is repeated on random nodes until the states no longer change. Each update reduces the energy of the system, therefore we have the guarantee that we will eventually reach the lowest point and stop. The algorithm is pretty nifty. We build a network with random node states and random weights, and we have the guarantee that we won’t bounce back and forth forever. But how is this related to associative memories?
Storing patterns in a Hopfield network.
It is possible to tweak the weights so that the stable points become a set of patterns we want to remember. Here we consider simple binary sequences like [1,1,1,-1,-1] or [-1,1,-1,1,-1]. The methodology is very similar to what we have seen so far. First, we prepare a list of sequences to memorize inside the network. We then calculate the new weight wij by averaging the sum of the multiplications between vi and vj in each sequence. In plain words, we use the average concordance between the two nodes as weight. Again, a nice biological interpretation. Once we have iterated the procedure on all the connections, our network stores the binary patterns in the low-energy points. And here is where the magic happens. To retrieve a pattern, it is sufficient to set some random states, and the update procedure previously explained will find the others. Before accusing me of witchcraft, let me give you an explanation of what is going on. The node states can be seen as coordinates in a multidimensional space. The energy function defines the shape of this space. In particular, setting the weights as we did creates profound valleys matching our patterns. When we fill some states, we are providing some coordinates to the network. It is like we said “Hey network, we search for the valley nearest to these coordinates”. The update rule travels down the hill until it finds the bottom point.
Associative memory and the brain.
Hopfield networks fascinate me as they propose an unexpected solution to how our brain may store information. For those who study biology, there is this concept of homeostasis where living beings always tend to preserve a state of general equilibrium. The brain seems to operate on the same principle, with different parts cooperating in finding general accordance. Hopfield networks heavily rely on the homeostasis of the system to memorize patterns. Weights average the contributions of all nodes, and the energy is minimized when the network finds an equilibrium. These considerations make me wonder: is this the mechanism by which we store information? Does our brain use homeostasis to form associative memories? I hope that one day we will finally have an answer.
In this article, we explored Hopfield networks and their potential relationship with human brains. For those interested in deepening the argument, I leave a couple of related links. You can also find a simple implementation of the classic Hopfield Network in my GitHub repository [3]. Enjoy.
[1] John Hopfield: Physics View of the Mind and Neurobiology | Lex Fridman Podcast #76
[2] Neural networks and physical systems with emergent collective computational abilities