I think you are far too pessimistic about the possibility that humans can understand what is learned by AI systems. Human chess players have learned many things by working with computer chess systems. The result is that many of today’s human players could defeat the world champions of 50 years ago. After the triumph of AlphaGo, the go players of the world are now studying its tactics and changing how they play.

It is not surprising that humans can learn from AI systems even though neither humans nor AI systems can articulate their knowledge in simple models. We’ve known for a long time that much human knowledge is implicit and based on statistical patterns rather than on elegant principles. And we’ve known that most human explanations and justifications are reconstructive and not causally related to the human thought process. At least with computers, we can produce justifications that are accurate.

It is useful to view the regularities found by today’s machine learning systems as similar to the empirical laws discovered by scientists such as Kepler and Boyle. These laws make useful predictions, but they do not provide a reductive explanation. Much more work is needed to find deeper and more parsimonious models that can provide those explanations. Such models typically allow us to make predictions in situations that are far-removed from the data that was employed to discover the empirical laws. The reductive models often also provide causal explanations that predict the effects of interventions, whereas empirical laws often cannot do this.

In my research, I’m hoping to create tools that can help us analyze and understand the regularities that our machine learning tools discover. You may wonder whether any such regularities have been discovered. But the fact that our deep neural networks are able to generalize to new situations means necessarily that they have found some regularities. There is a deep relationship between the ability to compress the raw observations and the ability to generalize to new situations. So those regularities exist, we just need to build tools to help us articulate them.