Book Review: “The Book of Why” by Judea Pearl
How Causal Graphs are being used to advance predictive analytics and AI
Around two million years ago our ancestors began to ask the question “why?”. Why are some of our hunts successful? Why do others fail? They began to co-opt the pattern recognition circuitry in our brains to model the world around us and perform what-if scenarios. They did this better than any being that ever lived on our planet.
Today, many of the new Deep Learning AI algorithms have achieved a level of pattern recognition that equals these ancestors of two million years ago. But they fail to come even close to a three-year-old when asked simple questions about how our world actually works. Our human brains have developed complex neural pathways that allow us to model the structure of the world around us and do far more than simply recognize patterns. We now inherently understand the concept of causality and can mentally consider alternate outcomes based on our actions. We have developed advanced planning and understand the consequences of poor planning.
Judea Pearl in his May 2018 “The Book of Why” takes us on a journey through the wonderland of Causality. He gives us better understanding of the processes that underlay causality and give us a path forward for anyone working in the fields of data science, AI and healthcare. Mr. Pearl is a Turning Award-winning expert in the field of computer science and champion of the “Causality Revolution”. He is trying to get statisticians to go beyond just looking at correlations. He wants us to build causal models that help us make better predictions about the world around us. The book is full of examples from the areas of diseases, drugs, nutrition and the social sciences.
Although the book does contain some mathematical formulas, I think Mr. Pearl has done the best he can to make these concepts understandable even to non-mathematicians. He starts with the decisively simple but powerful notion of a causal graph. Causal graphs are simple diagrams of nodes and directed arrows that model the world around us. They represent the way that causal information flows between hidden values and items that we can measure. By applying clever mathematics to these directed graphs, we can understand the causal relationships between variables. For example, we can understand how smoking impacts lung cancer, how diet impacts health and how drugs impact patient mortality.
Mr. Pearl also gives us a roadmap for future developments in AI and robotics. He tells us how we can move from a world where AI can only predict future events based on correlations between observations (Seeing) to prescribing how we can make changes in the world to understand the impact of interventions (Doing) and finally create a world where robots can start to ask theoretical questions about different future outcomes (Imagining). Mr. Pearl calls these three steps “The Ladder of Causality” and uses this metaphor throughout the book to encourage us to continue to think beyond simply seeing data to higher levels of analysis.
The book also tells many stories of the people who broke the early ground in causal analysis and bravely fought the against the status-quo of statistical analysis. These stories are well written and approachable to a broad audience. It also helps understand how there is still plenty of room for innovation in AI and data science.
For me personally, this book is consistent with my strong belief that many processes in our world could be improved by integrating structural models and retaining structure in our data. This goes beyond using graphical databases, calculating search relevancy and using document hierarchies that describe our world. I believe that the flat “Big Data” in our Data Lakes need to be continuously enriched with metadata and interlinked with other data to build useful knowledge that can be reused across multiple domains. This notion of “transferability” or “transportability” of knowledge is also discussed in the final chapter of the book. This is a key topic that anyone designing Enterprise Knowledge Graphs is grappling with today.