The Inability of Experts to Comprehend or Forecast AI Behaviors

The enigma of the interpretability problem, also known as AI explainability, lies in our quest to comprehend and anticipate the reasoning behind AI’s actions. When technologists create artificial intelligence and assign a goal, these intelligent machines embark on unpredictable paths to achieve that objective. Intriguingly, AIs do not leave clear indications for humans to fathom the hows and whys of their decision-making process.

So, why is it challenging for the very experts who conceived, constructed, and programmed these machines to comprehend them? According to Blake Richards, a neuroscientist, computer scientist, and esteemed AI expert who previously worked alongside Geoffrey Hinton, the ‘godfather of AI,’ these systems are engineered for optimization rather than tailored to specific purposes. Hinton’s revolutionary deep learning systems are the foundation of powerful yet perplexing machines like Bard and ChatGPT, image generators like Midjourney, deepfake apps, and even AlphaFold, the AI driving scientific breakthroughs.

Deep learning systems employ artificial neural networks inspired by the networks in the human brain. These networks consist of computational units called artificial neurons or nodes. By feeding vast amounts of human texts, images, and data into billions of artificial neurons organized in artificial neural networks, the capabilities of AI models surge. However, this increased capability comes at the cost of explainability.

Currently, our understanding of these systems is so opaque that AI models are often referred to as “black boxes.” Eliezer Yudkowsky, a decision theorist and AI expert at the nonprofit Machine Intelligence Research Institute, goes as far as describing them as “giant inscrutable matrices of floating-point numbers.”

According to Aliya Babul, an AI/quant expert with a background in computational astrophysics, the AI interpretability problem revolves around the ability to establish cause and effect relationships between an AI model’s inputs and outputs. Explaining why an AI made a particular decision may boil down to a translation issue. If the underlying causes for an AI decision are exceedingly complex, it may defy human comprehension.

“Most AI models are based on association, not cause and effect. They identify patterns in data, but they do not possess an understanding of why those patterns manifest,” explains Alaa Negeda, Chief Technology Officer at a prominent telecommunications firm. “This means they can make informed guesses without comprehending the underlying causal mechanisms, thereby making it difficult for humans to grasp their decision-making process.”

Interestingly, there is another elusive machine that humans struggle to understand and decipher: our own brains. Having worked as an AI research assistant in Hinton’s lab, Blake Richards pursued a graduate degree in neuroscience at the University of Oxford, where he now conducts interdisciplinary research spanning neuroscience and AI. Richards believes that the AI interpretability problem is not entirely surprising, given the mysterious nature of the human brain that serves as the inspiration for deep learning AI. “When we insert electrodes into human brains, the complexity of what we encounter bewilders us. It’s a chaotic puzzle,” remarks Richards. “Deciphering the functioning of different neurons is an arduous task. Therefore, I believe we are quite similar to AI in this regard. We are intricate systems molded by a combination of life experiences and evolution.”

Richards considers the AI interpretability problem to be a fascinating challenge but not a prerequisite for addressing other pressing AI concerns such as alignment – the quest to ensure that machines exhibiting unpredictable behavior operate in accordance with our best interests. Richards fails to discern a direct correlation between explainable AI and aligned AI. “I fail to comprehend why these two notions are frequently intertwined,” asserts Richards. “We manage to achieve alignment with other human beings without necessitating interpretability.”

Watch the interview with Blake Richards as he delves into the intricacies of explainable AI, the limitations of modeling AI after human brains, and various other AI-related topics: