Explanations from machines
Introducing "explainable AI" (XAI) and the two major classes of explanation
Decisions taken by machines play an increasingly important part in our lives, from mapping street signs and the ads you see during the election, to driving cars and criminal sentencing. The promise of AI is machines that can take faster, more objective and more reliable evidence-based decisions than humans. The pitfall is when AI decisions undermine accountability, obscure biases, and entrench disadvantage.
One strand of research hoping to tackle some of these pitfalls is explainable AI (XAI, sometimes also called interpretable AI). XAI aims to augment the machine with the ability to justify its decisions in a way that a human can understand. Why did the machine deny my loan? How did the system decide on a custodial sentence? What were the reasons the self-driving vehicle stopped at the intersection? The hope is that explanations that accompany decisions helps to build trust in machines and provides the transparency that the humans involved expect.
Explanations of automated decisions in business are already being viewed as “only fair” and a consumer right. The EU’s enactment of GDPR (General Data Protection Regulation) in 2018 is arguably a step towards making explanations a legal right. The growing range of decisions that rely on AI, combined with the growing awareness of a “right to explanation”, are only set to increase the relevance and impact of XAI research in the future.
AI is an enormously diverse field, so it will come as no surprise that the approaches to generating automated explanations in research are similarly varied. However, regardless of the different techniques adopted, it is possible to identify two major types of explanation in XAI, which we might loosely call “how” and “why” explanations.
“How” explanations
One possible response to a request to explain an automated decision is to describe the steps used by the machine in arriving at that decision. The first and most famous AI tool to attempt such explanations was the MYCIN system for medical diagnoses, developed at Stanford in the 1970s [1]. The MYCIN system had two major subsystems designed to assist physicians in choosing appropriate drugs to prescribe to patients: a consultation module, which combined a pharmaceutical database with a rule base and reasoning engine; and an explanation module, which attempted to justify its reasoning in terms of the steps and rules it used to arrive at a particular recommendation.
A catalogue of explanation types proposed by AI pioneer Deborah McGuiness and her team [2] details several different forms of “how” explanations, including trace-based (i.e., what steps did the machine take to arrive at the recommendation), case-based (i.e., in what other similar situations has this same reasoning step been applied), and scientific and statistical explanations (e.g., what is the experimental or statistical basis for this response). In my own recent work on XAI, we used a mix of automated trace-based, case-based, and statistical explanations as part of an automated reasoning system for detecting IUU (illegal, unreported, and unregulated fishing, figure below) [3].
“Why” explanations
“How” explanations, like those above, can help humans understand the mechanisms behind automated decisions. However, some researchers argue that less mechanistic, more “human” explanations should be the focus of the next generation of XAI. An excellent exploration of these types of approaches in AI is given by Tim Miller in [4]. A key observation is that natural, human explanations are frequently as much about what didn’t happen, as what did happen. The question “Why did the ship’s captain sound the horn?” requires a very different response depending on what was expected instead (e.g., “Why did the ship’s captain sound the horn instead of the first mate?” versus “Why did the ship’s captain sound the horn instead of the alarm?”.
Such interpretations of explanations are especially difficult for machines to master because the counterfactual expectation is often unstated (as in the example above, “Why did the ship’s captain sound the horn?). More challenging still, identifying the correct counterfactual relies on commonsense and contextual background knowledge—still very much areas where humans reign supreme over machines.
References
[1] Shortliffe, E., Davis, R., Axline, S. G., Buchanan, B. G., Green, C. C., & Cohen, S. N. (1975). Computer-based consultations in clinical therapeutics: Explanation and rule acquisition capabilities of the MYCIN system. Computers and Biomedical Research, 8(4), 303–320.
[2] Chari, S., Gruen, D. M., Seneviratne, O., & McGuinness, D. L. (2020). Directions for explainable knowledge-enabled systems. In I. Tiddi, F. Lécué, & P. Hitzler (Eds.), Knowledge graphs for explainable artificial intelligence: Foundations, applications and challenges (pp. 245–261). Amsterdam, The Netherlands: IOS Press.
[3] Duckham, M., Gabela, J., Kealy, A., Khan, M., Legg, L., Moran, W., Rumi, S.K., Salim, F.D., Sharmeen, S., Tao, Y., Trentelman, K., Vasardani, M. (2022). Explainable spatiotemporal reasoning for geospatial intelligence applications. Transactions in GIS, in press.
[4] Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1–38.