The Explosive Fusion of Intuition with Logic in Deep Learning

The low hanging fruit in present-day Deep Learning (DL) research can be found in the area where conventional logical systems are combined with intuitive DL systems. This I’ve labeled as ‘hybrid’ systems, they are inspired however by the observation that human thinking is a combination of ‘thinking fast and slow.’ This is known as the Dual Process Theory of cognition and is the original motivation of my research.

It is not immediately obvious how to combine symbolist and connectionist architectures into a single solution. I’ve previously discussed how to coordination rational and intuitive systems in “The Coordination of Intuition and Rational Intelligence.” To summarize, there are three ways cooperative ways that this can be done:

(1) Intuition can learn from simulations driven by a logical model.

(2) Intuition can truncate logical model based planning.

(3) Intuition can aid in selecting rewarding logical goals.

There is however a promising new trend that is a consequence of a new development in DL known as Graph Networks. I wrote about this previously in “Relational Intuitive Reasoning.” The gist of this new approach is that structured knowledge can be captured in graphs and these graphs can be used as the inputs to neural networks. This kind of method forms the mechanism of a kind of intuitionistic logical reasoning. In mathematics, there is a subfield of research known as intuitionistic logic (IL) that relaxes certain axioms of formal logic. Particularly, IL removes the rules of the law of excluded middle and double negation elimination.

It is interesting to notice that the proofs for undecidability require the use of the law of excluded middle.

The existence of some elements in the table have only been assumed for proof by contradiction, and we denote these lines by with the ∃ and / (i.e., There does not exist). I suspect you cannot prove undecidability with intuitionistic or constructive mathematics. This interestingly enough makes intuitive sense!

This limitation shouldn’t be much of a hindrance. As I’ve argued in the previous post, we pragmatically care about knowledge that is attainable. That is knowable unknowns. Let the mathematicians and philosophers worry about unknowable knowns. This reminds me, “unknowable unknowns” are ideas like multi-universes and whether the universe is a simulation.

With that out of the way, let’s examine several research papers that have begun to exploit this idea of intuitionistic logic:

The paper “Toward an AI Physicist for Unsupervised Learning” by Tailin Wu and Max Tegmark explores a system that attempts to derive the laws of physics. Physics (although based on advanced mathematics) has always been driven by a kind of intuition. So it’s not entirely out of left field, that perhaps discovery can be performed by an artificial intuition machine.

The authors examine if they can build automation that makes uses of several heuristics that human physicists employ to build theories:

Divide and conquer — Learning multiple theories that each fit the data well. Occam’s Razor — Minimize the descriptive length of theories. Unification — Explore unifying theories by adding parameters. Lifelong learning — Reuse existing learned solution on new problems.

The approach that is taken in similar to other work in ‘program induction’. This is where a network learns how to compose programs. A couple of years ago, there was a lot of hype generated by Microsoft’s DeepCoder that claimed to be an AI that wrote programs. I previously wrote about this where I discussed a hybrid method that combined language with neural networks: “A Language Driven Approach.”

The impressive aspect of the AI physicist research is how many knowledge discovery methods is leveraged in a coordinated manner to derive new knowledge. It is a more sophisticated kind of program induction in that certain rules of reasoning are used to drive the process. The fact that this even remotely works is astonishing and opens up new ideas for immediate exploration.

A second paper that has come to my attention is a paper that explores the idea of a ‘Semantic Loss.’ It is a curious idea that rather than work at the level of finding solutions to a massively discrete constraint problem, the approach is to work at the level of counting solutions to a massively discrete constraint problem. They call this “lifted inference,” and it has a lot of intriguing possibilities.

A third paper is this work that comes from Microsoft Research “Structured Neural Summarization” that explores creating better text summarization methods by combining LSTMs and Graph Networks. The problem with LSTMs is that they aren’t able to handle long-range reasoning. Graph Networks are used here to mitigate against this problem by linking up sections of text through a graph.

Finally, there is this paper by Preferred Networks (PFN). PFN is more well-known for being the developers of the Chainer Deep Learning framework. They are likely the most prominent Deep Learning company coming out of Japan. According to CrunchBase, PFN has raised $130m in funding. If there’s any other group to watch other than DeepMind, OpenAI, Google, Facebook, etc. then I would highly recommend following PFN.

PFN researchers wrote this extremely innovative paper that explores automated theorem proving using Deep Reinforcement Learning (DRL). Coincidentally, they use an intuitionistic logic theorem prover to serve a baseline of comparison. They encode logical rules into a Graph Network and use a convention theorem prover to drive curriculum training of the system. The results are also astonishing in that they seem to do much better than a handcrafted theorem prover like Coq Tauto. This reminds me of AlphaZero besting the handcrafted chess playing system Stockfish.

In summary, these are four research papers where most of them are just hot off the presses. The basic approach is extremely compelling, and it is what I’ve described as a Level Two capability (i.e., Dual Process — Intuitive Exploration). This indeed where a lot of action will be in the immediate future. It is indeed interesting that Level Three (Intuitive Causal Reasoning) requires some kind of causal graph and that perhaps the use of Graph Networks is a path in the right direction.

Further Reading