Blockologies: A Pattern Language for AI Data Flows

Last week I was part of team from Optum ventured out to Stanford University in Palo Alto CA for a three-day conference sponsored by the Association for the Advancement of Artificial Intelligence (AAAI). We all attended a track that combined Machine learning And Knowledge Engineering. (MAKE). We found that the conference organizers were creating a new pattern language for communicating high-level AI data flows. Pattern languages were initially created by architects like Christopher Alexander in the 1970s to help them discuss good designs using named concepts that keep recurring in problem solving.

In the new pattern language these diagrams are called“Blockologies”. Blockologies are high-level abstract data flow diagrams and that combine different types of AI data processing with human curation. The key is to standardize on the labels for each of the blocks so that other AI architects can quickly understand your data flow diagrams and architecture. This leads to higher-bandwidth discussions of architectural trade-offs using abstractions.

For those of you not familiar with the term knowledge engineering, it is the sub-field of AI that deals with the problem of knowledge representation and reasoning. It is my personal feeling that over half of the time building AI systems requires a careful understanding of the trade-offs of how to represent knowledge for reasoning. At the heart of reasoning is the concept of abstraction. Our brains do both low-level pattern matching and high-level symbolic reasoning. It is the intersection of these two areas where real AI happens. The current explosion of deep learning innovation is happening mostly at the low-level pattern recognition level. This conference was really about combining low-level pattern recognition with reasoning.

Knowledge Graphs are a current buzzword that surround the ability to use very large distributed graph databases to store complex networks of concepts that can be “traversed” using a highly parallel graph query language. Many knowledge graphs in companies like Google, Facebook, LinkedIn, Amazon (product graph) and Pinterest (interest graph) have over 100 billion vertices. So the past scalability problems for graph databases clearly have been solved.

In the past, the predominant way of building knowledge graphs was to use hand coded knowledge and an inference engine that could leverage higher-level RDF-based standards such as RDFS, OWL and SKOS. Now organizations are using machine learning to build seed concept graphs using natural language processing. There is also a strong shift to use the more flexible labeled-property-graphs (LPGs) to do similar reasoning. On the horizon is the ability to do these queries using specialized hardware tuned to do fast pointer hopping at scale.

The conference was organized by a group of AI researchers and practitioners that have been focused on combining patterns from the five different “tribes” of AI. The first two tribes, the symbolists and the connectionists dominated this conference. Here are the definitions of all five tribes from the book “The Master Algorithm” by Pedro Domingos:

The Symbolists: These are the people that try to build graphs of concepts using mostly human experts that can be used for inductive reasoning. This camp includes traditional taxonomists and ontologists using standards like OWL and SKOS. These hand-curated systems take a long time to build and become increasingly fragile as they grow. Data processing steps that use symbolic reasoning use the symbol “SYM”.

The Connectionists: These are the people using machine learning to build neural networks. To be effective, the neural networks need to be trained with large data sets, typically millions of labeled training examples. Because of the advances in machine vision and language processing they have become by far the largest tribe. Sometimes people new to AI are ONLY exposed to their methods. They have a “ML” symbol to denote a machine learning, and it should be noted that machine learning includes other algorithms other than just neural networks.

The Bayesians: These people focus on building causality graphs. Their data flows through graphs where each step includes probability weights that reflect the casualty relationships between data elements. Our brains seem to be really good at understanding causality. We intuitively understand causal relationships like the fact that clouds cause rain, but rain does not create clouds. To think like humans our AIs need to have a deep appreciation of causality. The patron saint of the Bayesian tribe is Judea Pearl who also spoke at the AAAI-MAKE conference. Their symbol might be something like “BAY”.

The Evolutionaries: These are the people that are using genetic programming algorithms to continually evolve their data processing steps. Some of these algorithms are called Evolutionary Strategies. Reinforcement Learning is can be thought of a type of evolutionary learning because it is constantly evolving to better fit the rules of the system. Their symbol might be “EVO” or “RL” if you are doing reinforcement learning.

The Analogizes: These are the people that are trying to lift partial knowledge graphs from one area in order to reuse generalized knowledge across multiple domains. In the symbolic space we often call this ontology linking. In the neural networking space this is often called Transfer Learning. I am not sure they will have a symbol in a data flow diagram since it is more of a process of building a knowledge graph by trying to isolate reusable components.

Since we are going to be combining these systems together in data flow diagrams, we also want to include symbols for raw data “DATA” and for human curation “HC”.

Here are some sample blockology flow diagrams that use these symbols. The first one is pretty simple. We take some data (like labeled images), put the data through a multi-level neural network such as a CNN or AlexNet and then return a list of symbols that classify the images such as “hot dog” or “not a hot dog”. This workflow is a simple linear workflow as described in Figure 1:

Figure 1: Simple linear machine learning blockology that produces symbols from data.

You could also use this same workflow for using any entity extraction pipeline where your inputs are not images but documents. The output of the ML step would be a list of entities and their location in the documents in the form of markup or character offsets.

Now what if we just classified images as indoor or outdoor pictures first and then used that classification to guide the next step of classification. The second level classifiers each can use a smaller list of things to classify in an image. This blockology is listed in Figure 2:

Figure 2: A two-step image classification system that has better results

This is similar to what Danilo Nunes, Leonardo Anjoletto Ferreira, Paulo E. Santos and Adam Pease did in their paper: Representation and Retrieval of Images by Means of Spatial Relations Between Objects. They also took it to the next level and used the SUMO ontology to look relative placement of items in the image. The key is they combined machine learning and abstract knowledge of the world in the form of an Ontology to classify items within an image.

Now let’s get a little more complicated. Lets assume we have a log of 1,000,000 chat dialogs from a helpdesk where people are asking specific questions about a specific topic. We want to train a chatbot to mimic a real help desk staff. We would take the incoming chats, classify them with machine learning using a curated taxonomy of topics. We would group questions with intents together and map these similar questions to a taxonomy of concepts stored in SKOS. We can then classify answers using the same taxonomy using a classification algorithm and then build a chatbot that matches questions to answers. That workflow is here in Figure 3.

Figure 3: A semantic search chatbot example has a more complex Blockology that adds additional detail.

You can start to group the initial workflow steps into an design-time training process. The production chat responses only need to classify the new questions according to a taxonomy and use this to help rank the search results. I am leaving several steps out to simplify the diagram. The point is that the Boxology diagram helps us explain the ways that both machine learning and symbolic reasoning work together with human created taxonomies and ontologies.

The take home point is that in order to go to the next level of AI, we need to get beyond the limitations of low-level pattern recognition. We need to combine high-level symbolic information with machine learning to get the best outcomes.

I want to also acknowledge the work that Frank van Harmelen and Annette ten Teije have done at the Vrije University of Amsterdam for their work building a pattern language for this area. I think they are working in the right direction and I hope others enlarge this pattern library. My hope is that we work together to write a book on this topic so that future AI experts can use these patterns to communicate their designs.