Computer Science

What is a Deep Boltzmann machine? (# 36)

If our universe were an ongoing computation, there may be signs of such computation in the structure of our universe. A Generative Adversarial Network (GAN) is an approach to machine learning that scientists used to create a computer program that beat the world’s best GO player. GO is a game considered so complicated that many people thought a computer could never be programmed to beat a human player. This article aims to provide circumstantial evidence that the programming for a GAN is built into the structure of our universe.

Generative Adversarial Networks

GANs are an approach to generative modeling using deep learning methods. These methods are used to program generative models. Two examples of deep learning methods are the Restricted Boltzmann Machine (RBM) and GANs. A Deep Boltzmann Machines (DBM) is like a stack of RBMs where connections between layers are undirected. GANs were developed to overcome the restrictions of generic deep learning algorithms such as DBMs.

The following graph is an example of a Boltzmann Machine with hidden (h) and visible (v) nodes.

There is no clear demarcation between the input and output layer in a Boltzmann Machine. Visible nodes take in the input. The nodes which take in the input return a reconstructed input as output. This is achieved through bidirectional weights which propagate backward and render the output on the visible nodes. Every node is connected to all the other nodes. All links are bidirectional and all weights are symmetric.

Generative modeling is an unsupervised task in machine learning that involves automatically discovering patterns in input data so that a computer can generate new examples identical to those in the original data. The GAN model is based on a generator model for generating new examples and a discriminator model to determine whether generated examples are real or fake.

The two computer models, generator and discriminator, are trained together. The generator generates a batch of samples, and these, along with real examples from the original data, are provided to the discriminator to be classified as real or fake. In the next round, the discriminator is updated to get better at discriminating between real and fake samples, and the generator is updated based on how well the generated samples fooled the discriminator. According to Dr. Jason Brownlee:

We can think of the generator as being like a counterfeiter, trying to make fake money, and the discriminator as being like police, trying to allow legitimate money and catch counterfeit money. To succeed in this game, the counterfeiter must learn to make money that is indistinguishable from genuine money.

AdS/CFT as a Deep Boltzmann machine

According to Koji Hashimoto:

Deep Boltzmann machines are a particular type of neural networks in deep learning for modeling probabilistic distribution of data sets. …

The AdS/CFT correspondence is a holographic duality between a (d+ 1) dimensional quantum gravity and a d-dimensional quantum field theory (QFT) without gravity. The latter lives at the boundary of the gravitational space-time of the former [called bulk spacetime]. …

… the standard AdS/CFT correspondence can be regarded as a deep Boltzmann machine. The neural network architecture, once properly defined, is interpreted as a bulk spacetime geometry. …

From the viewpoint of the QFT, the direction perpendicular to the boundary surface is an “emergent” space direction. … [The] structure of the deep Boltzmann machines suits the scheme of the AdS/CFT, once we identify their visible layers with the QFT, and the hidden layers as the bulk spacetime.

In brief, Hashimoto shows that the hidden nodes of a Deep Boltzmann Machine are part of the bulk space-time inside an Anti-de Sitter (AdS) black hole with the visible nodes being displayed on the boundary inside that black hole. This series of articles suggest that our universe is a display in a boundary inside a black hole. Furthermore, it is proposed that the type of computation carried out by our universe is a less restricted form of a Deep Boltzmann Machine namely a GAN.

Forgetting as part of deep learning

According to Carlos Perez:

Deep learning networks are very good at remembering things. … Its biggest problem, however, is that it does not know how to forget. … Forgetting is critical in building persistent abstractions. …

Advanced deep learning systems employ attention mechanisms to reduce the complexity of the input space and to focus on what is important. … What does not change, can effectively be ignored. …

Throwing away memory (or forgetting) is the process of pruning and consolidating experiences into a form that leads to generalization. … An activation function is a threshold function that modulates the flow of information. This has its effect in the forward flow of inference as well as the backward flow of learning (i.e. back-propagation). It is a deep learning network’s crude way of modulating memory changes. …

Recent experiments on biological brains have shown that the strategy for learning new tasks is not to forget, but rather to reuses [sic] existing neurons. …

… the mechanism for forgetting should happen during the consolidation of existing modules. This is what evolution does and this is also what the brain does. Both systems, re-purpose what already exists. Evolution and the brain aren’t reinventing new parts all the time, rather it is rewiring what already exists. In evolution, this is known as pre-adaptation. When you re-wire you don’t forget, you just repurpose. You save what was useful for another context where it might be eventually useful. This may not be optimal but seems to be the laziest thing that can be done (conforms to the principle of least action). Anything that can be done with the least effort is likely the most natural.

We can use our free will as an instrument of re-purposing. Our emotional responses to past and current events can influence if and how content can be repurposed. Repurposing conforms to the principle of least action; anything that can be done with the least effort is the most likely. We choose to ‘forget’ information to create more complexity.

Principle of Least Action

The physics principle of least action is consistent with a goal for the underlying reality being to increase complexity. As discussed in Article 11 — What is Quantum Computational Complexity — the mathematical concept of entropy could help identify rules that increase complexity. A corollary of identifying ways to increase complexity is to minimize the action required to achieve greater complexity i.e. a principle of taking the action that minimizes any increase in entropy.

The next article discusses the idea of Constructor Theory (CT). CT is a consequence of bulk space-time needing to achieve continuous increases in complexity. CT consists of principles that determine laws of physics that facilitate increases in complexity. The principle of least action is one of those principles. Thus, the action of forgetting or re-purposing is consistent with increasing complexity.

The question for this article is:

Do you believe you could be creative when you forget?

To view the headings of all the articles to be published in this series please click on https://michaeledalton.medium.com/orbiting-stars-and-origin-of-our-universe-338906930f51

To obtain a copy of the book ‘Orbiting Stars’ which contains the first drafts of all these articles, please visit https://www.amazon.com