2.2.1. Right to explanation

The Right to Explanation is a proposed fundamental human right that would guarantee individuals access to an explanation for any AI system decision that affects them. The Right to Explanation was written into the EU’s 2016 GDPR regulations, and reads as follows: “[the data subject should have] the right … to obtain an explanation of the decision reached.”Footnote ⁷ The legal meaning and obligation of the text have been debated heavily by legal scholars, who are unsure under which circumstances it applies, what constitutes an explanation (Selbst and Powles, Reference Selbst, Powles, Friedler and Wilson2018), and how the right is applicable to different AI systems (Doshi-Velez et al., Reference Doshi-Velez, Kortz, Budish, Bavitz, Gershman, O’Brien, Scott, Schieber, Waldo, Weinberger, Weller and Wood2017). The Right to Explanation is an example of how emerging AI technologies may “reveal” additional rights that need to be considered by lawmakers and legal experts (Parker and Danks, Reference Parker and Danks2019).

The EU’s recently proposed Artificial Intelligence Act simultaneously reinforces the idea that explanations about AI systems are a human right, while slightly rolling back the Right to Explanation by acknowledging that there are both nonhigh-risk and high-risk AI systems. Discussions about the Right are likely to continue, and will be a central part of debates on regulating AI transparency. In fact, some local governing bodies have already taken steps to adopt the Right to Explanation. France passed the Digital Republic Act in 2016, which gives the Right to Explanation for individuals affected by an AI system in the public sector (Edwards and Veale, Reference Edwards and Veale2018). Hungary also has a similar law (Malgieri, Reference Malgieri2019).

3.1.1. Technologists have the right technical expertise

Transparency has been a central topic of AI research for the past decade, and is motivated beyond just regulatory compliance by ideas like making systems more efficient, debugging systems, and giving decision-making agency to the decision subjects (i.e., those affected by AI-assisted decisions) or to the users of AI systems (i.e., those making decisions with the help of AI). New technologies in transparent AI are being created at a fast pace, and there is no indication that the rapid innovation of explainable AI will slow any time soon (Datta et al., Reference Datta, Sen and Zick2016; Ribeiro et al., Reference Ribeiro, Singh and Guestrin2016; Lundberg and Lee, Reference Lundberg, Lee, Guyon, von Luxburg, Bengio, Wallach, Fergus, Vishwanathan and Garnett2017; Covert et al., Reference Covert, Lundberg and Lee2020), meaning that of all the stakeholders involved in the socio-technical environment of AI systems, technologists are the most likely to be aware of available tools for creating transparent AI systems. Assuming any reasonable rate of advancement in AI technology, the knowledge of technologists will always outpace legislator’s ability to draft meaningful legislation. This means they are the stakeholder best positioned to mitigate the potential risks of opaque AI systems as new knowledge comes to light. Furthermore, there are currently no objective measures for the quality or transparency in AI systems (Gunning et al., Reference Gunning, Stefik, Choi, Miller, Stumpf and Yang2019; Lu et al., Reference Lu, Lee, Kim and Danks2019; Yang et al., Reference Yang, Kandogan, Li, Sen and Lasecki2019; Abdul et al., Reference Abdul, von der Weth, Kankanhalli and Lim2020; Holzinger et al., Reference Holzinger, Carrington and Müller2020). In lieu of legislative and regulatory mechanisms, numerous best (and worst) practices are emerging, and these are most easily understood by technologists.

3.1.2. Technologists are the least-cost avoiders

This idea is based on the principle of the least-cost avoider, which states that obligations and liabilities should be allocated entirely to the party with the lowest cost of care (Stoyanovich and Goodman, Reference Stoyanovich and Goodman2016). AI practitioners are the least-cost avoiders because they are already equipped with the technical know-how for building and implementing transparency tools into AI systems, especially when compared to policymakers and the individuals affected by the outcome of the system. Notably, given the wide range of existing transparency tools, implementing the “bare minimum” is trivially easy for most technologists.

One argument practitioners give against building transparent systems is that they may be less accurate than highly complex, black-box systems (Huysmans et al., Reference Huysmans, Baesens and Vanthienen2006). However, there has been a growing amount of evidence suggesting that building transparent systems actually results into little to no trade-off in the accuracy of AI systems (Stiglic et al., Reference Stiglic, Povalej Brzan, Fijacko, Wang, Delibasic, Kalousis and Obradovic2015; de Troya et al., Reference de Troya, Chen, Moraes, Bajaj, Kupersmith, Ghani, Brás and Zejnilovic2018; Bell et al., Reference Bell, Rich, Teng, Orešković, Bras, Mestrinho, Golubovic, Pristas and Zejnilovic2019, Reference Bell, Solano-Kamaiko, Nov and Stoyanovich2022; Rudin, Reference Rudin2019). In other words, building transparent systems is not a Pareto-reducing constraint for practitioners.

3.1.3. Technologists already bear the responsibility for implementing transparency into AI systems

A study interviewing AI technologists found that using AI responsibly in their work is viewed as the practitioner’s burden, not the institutions for which they work. Practitioners noted that existing structures within institutions are often antithetical to the goals of responsible AI, and that it is up to them to push for structural change within that institution (Rakova et al., Reference Rakova, Yang, Cramer and Chowdhury2020). Section 2 shows that AI regulation is converging on requiring transparent AI systems that offer meaningful explanations to stakeholders. Therefore, it is in the best interest of practitioners to continue the bottom-up approach of building transparent AI systems in the face of looming regulations.

4.1.1. Transparency

In this article, we will use the term “transparency” in a broad sense, and define it as “the degree to which a human can understand an AI system.” This is an adaptation of Christoph Molnar’s definition of “explainability” (Molnar, Reference Molnar2019), and it is appropriate here because it centers on the role of a human—rather than ascribing the property of being “transparent” to an algorithm or a system—and so takes a step in the direction of including stakeholders. This definition is general, and so it necessarily lacks concreteness and nuance. We use it as a starting point, and will expand on it in the remainder of this section, where we discuss different stakeholders, goals, and purposes for AI transparency.

4.1.2. Explanation

We use the term “explanation” to refer to an instantiation of transparency. For example, to ensure transparency for a system, a technologist may create an explanation about the data it uses.

4.1.3. Automated decision systems

The approach described in this article applies to all automated decision systems (ADS), which is any system that processes data to make decisions about people. This means that AI systems are a subset of ADS, but there are two key distinctions: (a) an ADS is underpinned by any algorithm and not just AI or machine learning, and (b) an ADS implies a context of use and some kind of impact. For a formal definition of ADS, see Stoyanovich et al. (Reference Stoyanovich, Howe and Jagadish2020). Henceforth, we will use the term ADS.

Notably, while many regulations are written to specifically mention “AI systems,” all the ideas they contain about transparency could be applied to all ADS. It is likely that future regulations will focus broadly on ADS, as seen in NYC Local Law 144 of 2021 and France’s Digital Republic Act.

4.3.1. Stakeholders

Much of ADS transparency research is focused on creating novel and innovative transparency methods for algorithms, and then later trying to understand how these methods can be used to meet stakeholders needs (Preece et al., Reference Preece, Harborne, Braines, Tomsett and Chakraborty2018; Bhatt et al., Reference Bhatt, Xiang, Sharma, Weller, Taly, Jia, Ghosh, Puri, Moura and Eckersley2020). Counter to this rationale, we propose a starting point that focuses on ADS stakeholders: assuming algorithmic transparency is intended to improve the understanding of a human stakeholder, technologists designing transparent ADS must first consider the stakeholders of the system, before thinking about the system’s goals or the technical methods for creating transparency.

The existing literature and taxonomies on ADS transparency have identified a number of important stakeholders, which include technologists, policymakers, auditors, regulators, humans-in-the-loop, and those individuals affected by the output of the ADS (Meyers et al., Reference Meyers, Vorsanger, Peters and Pierre2007; Amarasinghe et al., Reference Amarasinghe, Rodolfa, Lamba and Ghani2020; Meske et al., Reference Meske, Bunde, Schneider and Gersch2020). While there is some overlap in how these stakeholders may think about transparency, in general, there is no single approach to designing transparent systems for these disparate stakeholder groups, and each of them has their own goals and purposes for wanting to understand an ADS (Sokol and Flach, Reference Sokol and Flach2020). In fact, even within a stakeholder group, there may be variations on how they define meaningful transparency (Hohman et al., Reference Hohman, Head, Caruana, DeLine, Drucker, Brewster, Fitzpatrick, Cox and Kostakos2019).

There are two additional considerations we would like to surface for designers of ADS when thinking about stakeholders. First, it may be worthwhile to weigh the needs of different stakeholders differently. For example, it may be more meaningful to prioritize meeting the transparency needs of affected individuals over those of AI managers or auditors. Second, in certain contexts, stakeholders may want to be thought of as “groups,” rather than as individuals, because of a single, unified transparency goal. This can be important for issues related to fairness and accessibility. For instance, one stakeholder group that has a unified transparency need may be members of the Blind and Low Vision community, or of the Deaf community (Wolf and Ringland, Reference Wolf and Ringland2020).

Importantly, by staking transparency on the needs of stakeholders, technologists will be better positioned to meet criteria for proposed and existing citizen-aware AI transparency regulations like the Right to Explanation, and those that require audits of ADS. This is also relevant for stakeholder groups, which may be those groups protected under legislation. For example, although there only exists proposed legislation mandating recourse for AI systems, technologists can get ahead of these mandates by following the approaches we lay out in this article.

Running example. In the ADS used by IEFP in Portugal, there are four main stakeholders: the technologists who developed the ADS, the policymakers who reviewed the ADS and passed laws for its implementation, the job counselors who use the system, and the affected individuals who are assessed for long-term unemployment. In the development of the AI, explanations were created to meet the varying goals of many of these stakeholders including practitioners, policymakers, and the job counselors. Unfortunately, and significantly, affected individuals were not considered. Had the practitioners adopted a robust stakeholder-first approach to designing transparent systems they could have better considered how to meet the goals of this key stakeholder group. For example, a person may want to appeal being predicted low risk because they feel they are high risk for long-term unemployment and need access to better interventions.

4.3.2. Goals

There has been little consensus in the literature on how ADS transparency goals should be classified. Some researchers have focused broadly, classifying the goals of ADS as evaluating, justifying, managing, improving, or learning about the outcome of an ADS (Meske et al., Reference Meske, Bunde, Schneider and Gersch2020). Others have defined goals more closely to what can be accomplished by known transparency methods, including building trust, establishing causality, and achieving reliability, fairness, and privacy (Marcinkevics and Vogt, Reference Marcinkevics and Vogt2020). Amarasinghe et al. identified five main goals (designated as use-cases) of transparency specifically in a policy setting: model debugging, trust and adoption, whether or not to intervene, improving intervention assignments, and for recourse. In this context, the term intervention refers to a policy action associated with the outcome of an ADS.

Notably, the goals of transparency are distinct from the purpose. The purpose addresses a context-specific aim of the ADS. For example, if an explanation is created for an ADS with the purpose of explaining to an individual why their loan was rejected, the goal may be to offer individual recourse against the rejection. This distinction is made clear in Section 4.3.3.

For our stakeholder-first approach, we make two changes to the existing body of research work. First, we require that the goal of transparent design must start with a stakeholder. Since all transparency elements of an ADS are intended for a human audience, defining a stakeholder is implicit in defining goals. Second, we have established six goal categories, which encompass those found in literature. These categories are validity, trust, learning and support, recourse, fairness, and privacy, and are defined in Table 2 alongside concrete examples of how these goals may be implemented.

Table 2. Definitions and examples of stakeholder goals for the six categories of ADS transparency goals

An important discussion surrounding goals are the justifications for pursuing them. For example, fairness and privacy goals may be justified for humanitarian reasons (they are perceived by the stakeholders as the “right thing to do”). Other justifications may be to prevent harm, like offering recourse to stakeholders against an outcome of an ADS, or for a reward, like an explanation that supports a doctor’s correct diagnosis. For reasons of scope, we will not delve into the issue of goal justification in this article.

Running example. In our case study, transparency is built into the ADS with the goal of offering learning and support to job counselors. The ADS generates explanations about what factors contribute to an individual being classified as low, medium, or high risk for long-term unemployment, which job counselors use to help make better treatment decision. Furthermore, the job counselor may also use the explanation to offer recommendations for recourse against a high-risk score.

4.3.3. Purpose

Miller proposed that the purpose of transparency is to answer a “why” question (Miller, Reference Miller2017), and gives the following example: In the context where a system is predicting if a credit loan is accepted or rejected, one may ask, “why was a particular loan rejected?” Liao et al. expanded on this significantly by creating a “question bank” which is a mapping from a taxonomy of technical transparency methodology to different types of user questions. Instead of just answering why questions, the works show that transparency can be used to answer 10 categories of questions: questions about the input, output, and performance of the system, how, why, why not, what if, how to be that, how to still be this, and others (Liao et al., Reference Liao, Gruen, Miller, Bernhaupt, Mueller, Verweij, Andres, McGrenere, Cockburn, Avellino, Goguey, Bjøn, Zhao, Samson and Kocielnik2020). These questions have two important characteristics. First, they are context-specific and should address a direct transparency goal of the stakeholder. Second, and importantly for technologists, these questions can be mapped onto known methods for creating explanations, meaning that a well-defined purpose for transparency acts as a bridge between the goals and methods.

Thoughtfully defining the goals and purpose of transparency in ADS is critical for technologists to be compliant with regulators. It is not sufficient to try and apply general, one-size-fits-all design like simply showing the features that were used by an ADS. For instance, both the proposed Algorithmic Accountability Act in the United States and the Artificial Intelligence Act in the European Union specifically mention that ADS should have transparency mechanisms that allow individuals to have recourse against a system outcome. Researchers have noted that feature-highlighting transparency lacks utility when there is a disconnect between the explanation and real-world actions (Barocas et al., Reference Barocas, Selbst and Raghavan2020). For instance, if someone is rejected for a loan and the reason for that decision is the person’s age, there is no action that they can effectively take for recourse against that decision.

Running example. In the long-term unemployment use case, there were two main purposes of transparency: to understand why an individual was assigned to a particular risk category, and to understand what could be done to help high-risk individuals lower their chances of remaining long-term unemployed.

4.3.4. Methods

Once the stakeholders, goals, and purposes for algorithmic transparency have been established, it is time for the technologist to pick the appropriate transparency method (sometimes called explainablity method). Over the past decade, there has been significant work in transparent ADS research (sometimes called “explainable AI” research or XAI) on developing new methods for understanding opaque ADS. There are several existing taxonomies of these methods, which show that explanations can be classified on a number of attributes like the scope (local or global), intrinsic or post hoc, data or model, model-agnostic or model-specific, surrogate or model behavior, and static or interactive (Molnar, Reference Molnar2019; Arya et al., Reference Arya, Bellamy, Chen, Dhurandhar, Hind, Hoffman, Houde, Liao, Luss, Mojsilovic, Mourad, Pedemonte, Raghavendra, Richards, Sattigeri, Shanmugam, Singh, Varshney, Wei and Zhang2020; Marcinkevics and Vogt, Reference Marcinkevics and Vogt2020). Furthermore, researchers have created a number of different tools to accomplish transparency in ADS (Datta et al., Reference Datta, Sen and Zick2016; Ribeiro et al., Reference Ribeiro, Singh and Guestrin2016; Lundberg and Lee, Reference Lundberg, Lee, Guyon, von Luxburg, Bengio, Wallach, Fergus, Vishwanathan and Garnett2017; Covert et al., Reference Covert, Lundberg and Lee2020).

In contrast to the complex classification of transparency methods by technologists, regulations have focused on two elements of ADS: (a) what aspect of the ADS pipeline is being explained (the data, algorithm, or outcome)?, and (b) what is the scope of the explanation (for one individual or the entire system)? Table 3 shows how different regulations speak to different combinations of pipeline and scope. In our stakeholder first approach to transparency, we focus on these two main attributes. We will not discuss specific methods in detail, but for the convenience of technologists we have underlined them throughout this discussion.

Table 3. How different laws regulate the ADS pipeline (the data, algorithm, or outcome), and within what scope (local or global)

Data, algorithm, or outcome. Transparency methods have focused on generating explanations for three different “points in time” in an ADS pipeline: the data (preprocessing), the model/algorithm (in-processing, intrinsic), or the outcome (postprocessing, post hoc) (Ventocilla et al., Reference Ventocilla, Helldin, Riveiro, Bae, Boeva, Falkman and Lavesson2018; Arya et al., Reference Arya, Bellamy, Chen, Dhurandhar, Hind, Hoffman, Houde, Liao, Luss, Mojsilovic, Mourad, Pedemonte, Raghavendra, Richards, Sattigeri, Shanmugam, Singh, Varshney, Wei and Zhang2020). Importantly, transparency is relevant for each part of the machine learning pipeline because issues likes bias can arise within each component (Yang et al., Reference Yang, Huang, Stoyanovich and Schelter2020).

Transparency techniques that focus on the preprocessing component of the pipeline, that is, on the data used to create an ADS, typically include descriptive statistics or data visualizations.

Data visualizations have proved useful for informing users and making complex information more accessible and digestible, and have even been found to have a powerful persuasive effect (Pandey et al., Reference Pandey, Manivannan, Nov, Satterthwaite and Bertini2014; Tal and Wansink, Reference Tal and Wansink2016). Therefore, it is advisable to use data visualization if it can easily address the purpose of an explanation. However, visualizations should be deployed thoughtfully, as they have the ability to be abused and can successfully misrepresent a message through techniques like exaggeration or understatement (Pandey et al., Reference Pandey, Rall, Satterthwaite, Nov and Bertini2015).

Techniques for creating in-processing or postprocessing explanations call into question the important consideration of using explainable versus black-box algorithms when designing AI. The machine learning community accepts two classifications of models: those that are intrinsically transparent by their nature (sometimes called directly interpretable or white-box models), and those that are not (called black box models) (Marcinkevics and Vogt, Reference Marcinkevics and Vogt2020). Interpretable models, like linear regression, decision trees, or rules-based models, have intrinsic transparency mechanisms that offer algorithmic transparency, like the linear formula, the tree diagram, and the set of rules, respectively. There are also methods like select-regress-round that simplify black-box models into interpretable models that use a similar set of features (Jung et al., Reference Jung, Concannon, Shroff, Goel and Goldstein2017).

As an important design consideration for technologists, researchers have studied the effect of the complexity of a model and how it impacts its ability to be understood by a stakeholder. A user study found that the understanding of a machine learning model is negatively correlated with its complexity, and found decision trees to be among the model types most understood by users (Allahyari and Lavesson, Reference Allahyari, Lavesson, Kofod-Petersen, Heintz and Langseth2011). An additional, lower-level design consideration is that model complexity is not fixed to a particular model type, but rather to the way that the model is constructed. For example, a decision tree with 1,000 nodes will be understood far less well than a tree with only three or five nodes.

In contrast to in-process transparency, which is intrinsically built into a model or algorithm, post hoc transparency aims to answer questions about a model or algorithm after is has already been created. Some of the most popular post hoc methods are LIME, SHAP, SAGE, and QII (Datta et al., Reference Datta, Sen and Zick2016; Ribeiro et al., Reference Ribeiro, Singh and Guestrin2016; Lundberg and Lee, Reference Lundberg, Lee, Guyon, von Luxburg, Bengio, Wallach, Fergus, Vishwanathan and Garnett2017; Covert et al., Reference Covert, Lundberg and Lee2020). These methods are considered model-agnostic because they can be used to create explanations for any model, from linear models to random forests to neural networks. Some methods create a transparent surrogate model that mimics the behavior of a black-box model. For example, LIME creates a linear regression to approximate an underlying black-box model (Lundberg and Lee, Reference Lundberg, Lee, Guyon, von Luxburg, Bengio, Wallach, Fergus, Vishwanathan and Garnett2017). More work needs to be done in this direction, but one promising study has shown that post hoc explanations can actually improve the perceived trust in the outcome of an algorithm (Bekri et al., Reference Bekri, Kling, Huber, Martnez-Álvarez, Lora, Muñoz, Quintián and Corchado2019).

However, post hoc transparency methods have been shown to have two weaknesses that technologists should be aware of: (a) in many cases, these methods are at-best approximations of the black box they are trying to explain (Zhang et al., Reference Zhang, Song, Sun, Tan and Udell2019), and (b) these methods may be vulnerable to adversarial attacks and exploitation (Slack et al., Reference Slack, Hilgard, Jia, Singh, Lakkaraju, Markham, Powles, Walsh and Washington2020). Some researchers have also called into question the utility of black-box models and posthoc explanation methods altogether, and have cautioned against their use in real-world contexts like clinical settings (Rudin, Reference Rudin2019).

Scope. There are two levels at which a transparent explanation about an ADS can operate: it either explains its underlying algorithm fully, called a “global” explanation; or it explains how the algorithm operates on one specific instance, called a “local” explanation. Molnar further subdivides each of these levels into two sublevels: global explanations can either be holistic (applying to an entire algorithm, which includes all of its features, and in the case of an ensemble algorithm, all of the component algorithms) or modular, meaning they explain on part of the holistic explanation and local explanations can either be applied to a single individual, or aggregated to provide local explanations for an entire group (Molnar, Reference Molnar2019).

The scope of an explanation is highly relevant to the stakeholder and goals of an explanation, and is related to whether the stakeholder operates at a system or individual level. Researchers found that the scope of explanation can influence whether or not an individual thinks a model is fair (Liao et al., Reference Liao, Gruen, Miller, Bernhaupt, Mueller, Verweij, Andres, McGrenere, Cockburn, Avellino, Goguey, Bjøn, Zhao, Samson and Kocielnik2020; Islam et al., Reference Islam, Eberle, Ghafoor and Ahmed2021). Policymakers and ADS compliance officers are more apt to be concerned with system-level goals, like ensuring that the ADS is fair, respects privacy, and is valid overall, while humans-in-the-loop and those individuals affected by the outcome of an ADS are likely more interested in seeing local explanations to pertain to their specific cases. Technologists should consider both.

Naturally, there is considerable overlap between stakeholders’ scope needs (for example, an auditor may want to inspect a model globally and look at local cases), but generally, it is important which scope an explanation has. Therefore designers of ADS explanations should be thoughtful of how they select the scope of an explanation based on a stakeholder and their goals.

Running-example. In the IEFP use case, SHAP factors were given to job counselors to show the top factors influencing the score of a candidate both positively and negatively (Zejnilović et al., Reference Zejnilović, Lavado, de Troya, Sim and Bell2020). The transparency provided by SHAP provided a local explanation about the outcome of the model. A bias audit was also conducted on the entire algorithm, and presented to policy officials within IEFP.

Overall, researchers found that the explanations improved the confidence of the decisions, but counter-intuitively, had a somewhat negative effect on the quality of those decisions (Zejnilović et al., Reference Zejnilović, Lavado, de Troya, Sim and Bell2020).