Is “Double” Exponential Growth in Deep Learning leading to AGI?
It is important to have an Artificial General Intelligence (AGI) roadmap. Not because we are going to achieve this anytime soon, but rather because we need a framework to understand the progress that is happening in Deep Learning. Ask any researcher and they will tell you that progress in Deep Learning is at breakneck speed. Russ Salakhutdinov (Apple’s lead in AI) in a recent Simon’s Institute lecture remarked that the developments were “crazy”. So, in this context of “crazy”, we got to get our bearings and see exactly where the hell are we at?!
I’ve previously put together a Deep Learning roadmap:
However, this is just a roadmap of the current techniques in Deep Learning. It does not provide a sense of the gaps in capabilities that we need to achieve to arrive at an approximation of AGI.
I’ve written earlier about what DARPA refers to as the Third Wave of AI. DARPA’s framework gives a sense on what may be next, albeit a rather major advance to get to that “what’s next”. I also give a kind of roadmap in terms of capabilities that we may anticipate for Deep Learning. Here’s a graphic depiction of these capability levels:
Unfortunately, this does not give enough of a sense of what is achievable at each capability level. Peter Voss points out a paper by Pat Langley in 2012 in his post about “Cognitive Architectures”. Pat Langley’s paper is a good framework for assessing what kind of progress needs to be made in AGI. I have to admit, I have not spent a lot of energy trying to contrast the different AGI approaches that are out there. I do however find Pat Langley’s paper to be reasonable, simple and conservative enough to be a good basis for assessing current AGI development.
Peter Voss has his own take on the paper, I will however do my own take from the perspective of Deep Learning development. I will ignore any development from other A.I. tribes in my analysis. I encourage the reader to read Langley’s paper or Voss’ blog post prior to proceeding. The paper was written in 2012, prior to the Deep Learning boom, so I am revisiting it today to see if we have made any progress in the AGI fronts that was described in the paper.
- High-Level Cognition
Hybrid architectures like Deep Reinforcement Learning and AlphaGo reveal an extremely compelling way to fuse the intrinsic intuition based cognition in DL system into higher level capabilities that require planning and strategy. We are still a way off in achieving abstract reasoning, comprehension and problem solving
2. Structured Representations
Deep Learning representations are opaque and inscrutable. However, interesting enough, the many forms of neural embedding (i.e. word2vec, Glove etc) seem to be able to capture some semantics to be useful as input features to other networks. It also seems that some prior external knowledge about the world can be introduced by these embeddings. Integration tends to require end-to-end training, however this is an extremely promising area to pursue. What I don’t see happening soon is the interpretability of the representations.
3. System-Level Approach
I think “Cognitive Synergy” is going to be one of the more powerful developments in Deep Learning. Cognitive Synergy is the notion that many agents can work off the same representation. I think Deep Learning is making considerable strides in this space with regards to “Multi-objective” systems as well as in encoder-decoder networks. What we are seeing today is that multiple neural networks (see: Modular Deep Learning) are working in concert to create impressive results. The many stumbling block is that there is a need for ‘late-binding’ of representations. That is something that has yet to be developed.
4. Heuristics and Satisficing
There is a inherent bias in the research community to demand that the mechanisms that are used by Deep Learning have to conform to more rigid mathematical reasoning. There is a lot of emphasis in our models that demand some probabilistic interpretation. I am however on the camp that complex behavior emerges out of simplistic mechanisms. There is still a lot of work that needs to be done here, however we can see how our current models seem to be cracking under the strain of a lot of unexplainable observations.
5. Links to Human Cognition
Deep Learning developments are no where near addressing what Voss’ describes as “ambiguity, abstract conceptualization and reasoning, short-term memory and context, as well as meta-cognition.” There is research on this that explores behavior prediction as well as playing information imperfect games like Poker. Still, it is still an unexplored area.
6. Exploratory Research
Fundamental research on the behavior of Deep Learning takes a back seat to novel techniques that yield impressive results in narrow domains. It will likely stay this way for a while. The ‘attention’ market today commands a higher premium on novelty and benchmarks rather than fundamental insight.
Note that this are 6 assumptions that “were widely adopted during the AI’s first three decades”. The beauty of this is that is that we can contrast what we know today about Deep Learning and see how it fits with decades all perspective of what needs to be invented. As you can see, there are developments in good progress in 5 of 6 fronts.
Experimentation and engineering in Deep Learning far surpasses theory and this trend will not end soon or at all. Fundamental research isn’t given as much a priority in this field over more tangible “state-of-the-art” results. However, that does not imply that we should neglect thinking about a much larger AGI roadmap such as this. This post hopes to shed a little more light on how far Deep Learning needs to go to achieve AGI.
