2.1.1. Nonstationary environments

In a policy context, ML models use data about historical events to predict the likelihood of either the occurrence of an event in the future or the existence of a present need, and the context around the problem changes over time. This nonstationary nature of the data introduces strong temporal dependencies that should be considered throughout the modeling pipeline and makes these models susceptible to errors such as data leakage (Kaufman et al., Reference Kaufman, Rosset and Perlich2011; Samala et al., Reference Samala, Chan, Hadjiiski and Koneru2020). For instance, the use of standard randomized k-fold cross-validation as a model selection strategy can create training sets with information from the future, which would not have been available at model training time.

2.1.2. Evaluation metrics reflect real-world resource constraints

The mental health outreach in Bauman et al. (Reference Bauman, Salomon, Walsh, Sullivan, Boxer, Naveed, Helsby, Schneweis, Lin, Haynes, Yoder and Ghani2018) was limited by staffing capacity to intervene on only 200 individuals at a time, and the rental inspections team in Ye et al. (Reference Ye, Johnson, Fu, Copeny, Donnelly, Freeman, Lima, Walsh and Ghani2019) could only inspect around 300 buildings per month. Resource constraints such as these are inherent in policy contexts, and the metrics used to evaluate and select models should reflect the deployment context. As such, these applications fall into the top-k setting, where the task involves selecting exactly $ k $ instances as the “positive” class (Liu et al., Reference Liu, Dietterich, Li and Zhou2016). In such a setting, we are concerned with selecting models that work well for precision in the top $ k $ % of predicted scores (Boyd et al., Reference Boyd, Cortes, Mohri and Radovanovic2012) rather than optimizing accuracy or area under the ROC curve (AUC-ROC) as often done in “standard” classification problems, which would be suboptimal.

2.1.3. Heterogeneous data sources with strong spatiotemporal patterns

Developing a feature set that adequately represents individuals in policy applications typically entails combining several heterogeneous data sources, often introducing complex correlation structures to the feature space not usually encountered in ML problems used in research settings. For instance, in Bauman et al. (Reference Bauman, Salomon, Walsh, Sullivan, Boxer, Naveed, Helsby, Schneweis, Lin, Haynes, Yoder and Ghani2018), the ML model combines data sources such as criminal justice data (jail bookings), emergency medical services data (ambulance dispatches), and mental health data (electronic case files) to gain a meaningful picture of an individual’s state. Additionally, temporal patterns in the data are often particularly instructive, requiring further expansion of the feature space to capture the variability of features across time (number of jail bookings in the last 6 months, 12 months, and 5 years). The combination of such features across a range of domains, geographies, and time frames yields a large (and densely populated) feature space compared to typical structural data-based ML problems we encounter in research settings.

4.1.1. Inherently interpretable ML models

Inherently interpretable ML models are designed such that an end user could understand its decision-making process (Lakkaraju et al., Reference Lakkaraju, Bach and Leskovec2016; Rudin, Reference Rudin2019). In a policy context, an interpretable model could allow a user to (a) understand how the model calculates a risk score (global explanation) and (b) understand what factors contributed to the predicted risk score (local explanation) for a given instance. Several efforts have focused on developing interpretable models for policy domains, such as those for healthcare and criminal justice (Caruana et al., Reference Caruana, Lou, Microsoft, Koch, Sturm and Elhadad2015; Zeng et al., Reference Zeng, Ustun and Rudin2017). These include sparse linear models (Ustun et al., Reference Ustun, Spangher and Liu2013; Ustun and Rudin, Reference Ustun and Rudin2019), sparse decision trees (Hu et al., Reference Hu, Rudin and Seltzer2019), generalized additive models (Hastie and Tibshirani, Reference Hastie and Tibshirani1990; Lou et al., Reference Lou, Caruana and Gehrke2012, Reference Lou, Caruana, Gehrke and Hooker2013; Caruana et al., Reference Caruana, Lou, Microsoft, Koch, Sturm and Elhadad2015), and interpretable decision sets (Lakkaraju et al., Reference Lakkaraju, Bach and Leskovec2016).

Interpretable models often rely on carefully curated representations of data with meaningful input features (Rudin, Reference Rudin2019), often through discretization or binary encoding (Caruana et al., Reference Caruana, Lou, Microsoft, Koch, Sturm and Elhadad2015; Lakkaraju et al., Reference Lakkaraju, Bach and Leskovec2016; Ustun and Rudin, Reference Ustun and Rudin2019). Distilling complex data spaces into a handful of optimally discretized and meaningful features can entail extensive effort and optimization of its own. While careful feature preparation is indispensable in any ML application, regardless of the employed ML algorithm complexity, distilling complex and heterogeneous feature spaces typically found in policy settings into a handful of simple features can prove to be particularly challenging.

4.1.2. Post hoc methods for explaining black-box ML models

Post hoc methods derive explanations from already trained black-box/opaque ML models. As post hoc methods do not interfere with the model’s training process, they enable the use of complex ML models to achieve explainability without the risk of sacrificing performance. However, as black-box ML models are often too complex to be explained entirely, post hoc methods typically derive an approximate explanation (Gilpin et al., Reference Gilpin, Bau, Yuan, Bajwa, Specter and Kagal2018; Rudin, Reference Rudin2019), which makes ensuring the fidelity of the explanations to the model a key challenge in this work. Unlike inherently interpretable models, local and global explanations for opaque complex ML models require different methods. For both types of explanations, both model-specific and model-agnostic methods exist in the literature.

Post hoc local explanations. A local explanation in a typical public policy problem is used to understand which factors affected the predicted risk score for an individual entity. The most common format of local explanation is feature attribution—also known as feature importance or saliency—where each input feature is assigned an importance score that quantifies its contribution to the model prediction (Baehrens et al., Reference Baehrens, Schroeter, Harmeling, Kawanabe, Hansen and Müller2010; Bhatt et al., Reference Bhatt, Xiang, Sharma, Weller, Taly, Jia, Ghosh, Puri, Moura and Eckersley2020b). Several approaches exist for deriving feature importance scores such as: fitting an interpretable surrogate model (linear classifier) around a local neighborhood of the instance in question (Ribeiro et al., Reference Ribeiro, Singh and Guestrin2016; Plumb et al., Reference Plumb, Molitor and Talwalkar2018); feature perturbation- based methods for approximating each feature’s importance using game-theoretic Shapely values (Lundberg and Lee, Reference Lundberg, Erion, Chen, Degrave, Prutkin, Nair, Katz, Himmelfarb, Bansal and Lee2017; Lundberg et al., Reference Lundberg, Nair, Vavilala, Horibe, Eisses, Adams, Liston, Low, Newman, Kim and Lee2018a); and gradient-based techniques (Simonyan et al., Reference Simonyan, Vedaldi and Zisserman2013; Zeiler and Fergus, Reference Zeiler, Fergus, Fleet, Pajdla, Schiele and Tuytelaars2014; Bach et al., Reference Bach, Binder, Montavon, Klauschen, Müller and Samek2015). Among these approaches, methods such as LIME (Ribeiro et al., Reference Ribeiro, Singh and Guestrin2016), SHAP (Lundberg and Lee, Reference Lundberg, Erion, Chen, Degrave, Prutkin, Nair, Katz, Himmelfarb, Bansal and Lee2017), and SA (Zeiler and Fergus, Reference Zeiler, Fergus, Fleet, Pajdla, Schiele and Tuytelaars2014) are model-agnostic methods, whereas LRP (Bach et al., Reference Bach, Binder, Montavon, Klauschen, Müller and Samek2015), deconvolution (Simonyan et al., Reference Simonyan, Vedaldi and Zisserman2013), and TreeSHAP (Lundberg et al., Reference Mothilal, Sharma and Tan2018b) are model specific methods. MAPLE (Plumb et al., Reference Plumb, Molitor and Talwalkar2018) stands out among these methods as it can act both as an inherently interpretable model as well as a model-specific post hoc local explainer.

Other approaches such as influence functions (Koh and Liang, Reference Koh and Liang2017) as well as prototypes and criticisms (Kim et al., Reference Kim, Khanna and Koyejo2016) make use of data instances, rather than features, to provide local explanations. A special form of example-based explanation is counterfactual explanations, which seeks to answer the following question: “what’s the smallest change in data that would result in a different model outcome?” (van der Waa et al., Reference Yang, Rudin and Seltzer2018; Wachter et al., Reference Wachter, Mittelstadt and Russell2018; Molnar, Reference Plumb, Molitor and Talwalkar2019; Barocas et al., Reference Barocas, Selbst and Raghavan2020; Mothilal et al., Reference Mothilal, Sharma and Tan2020; Karimi et al., Reference Karimi, Schölkopf and Valera2021b). In a top- $ k $ setting, the change in outcome can be the inclusion versus exclusion of the individual from the top- $ k $ list. Counterfactual explanations can provide insight into how to act to change the risk score, supplementing the feature attribution methods that explain why the model arrived at the risk score.

Post hoc global explanations. A global explanation in a typical policy problem would be a summary of factors/patterns that are generally associated with high-risk scores, often expressed as a set of rules (Plumb et al., Reference Plumb, Molitor and Talwalkar2018; Ribeiro et al., Reference Ribeiro, Singh and Guestrin2018). Global explanations should enable the users to accurately predict, sufficiently frequently, how the model would behave in a given instance. However, deriving global explanations of models that learn highly complex nonlinear decision boundaries is very difficult (Ribeiro et al., Reference Ribeiro, Singh and Guestrin2016). As a result, the area of deriving post hoc global explanations is not as developed as local explanation methods.

Some approaches for global explanations from black-box ML models include (a) aggregation of local explanations (Lundberg et al., Reference Molnar2020), (b) global surrogate models (Frosst and Hinton, Reference Frosst and Hinton2017), and (c) rule extraction from trained models (Tsukimoto, Reference Tsukimoto2000). A noteworthy contribution to deriving globally faithful explanations is ANCHORS (Ribeiro et al., Reference Ribeiro, Singh and Guestrin2018). ANCHORS identifies feature behavior patterns that have high precision and coverage in terms of their contribution to the model predictions of a particular class. Methods proposed by Lundberg et al. (Reference Molnar2020) and Ribeiro et al. (Reference Ribeiro, Singh and Guestrin2018) are model-agnostic and methods presented by Frosst and Hinton (Reference Frosst and Hinton2017) and Tsukimoto (Reference Tsukimoto2000) are model-specific.

4.2.1. Model debugging

Methods for both local and global post hoc explanations are potentially useful in this use case. Global explanations could help identify errors in overall decision-making patterns (e.g., globally important features can help identify data leakage), and local explanations can help to uncover errors in individual predictions.

Although some recent work lends evidence for the utility of explanations in discovering model errors (Caruana et al., Reference Caruana, Lou, Microsoft, Koch, Sturm and Elhadad2015; Ribeiro et al., Reference Ribeiro, Singh and Guestrin2016; Adebayo et al., Reference Adebayo, Muelly, Liccardi and Kim2020; Abid et al., Reference Abid, Yuksekgonul, Zou, Chaudhuri, Jegelka, Song, Szepesvari, Niu and Sabato2022), the efficacy of these methods is not empirically validated through well-defined user trials in real-world applications. For instance, Ribeiro et al. (Reference Ribeiro, Singh and Guestrin2016) demonstrate how LIME explanations could help users identify model errors through a simplified image classification task and a text classification task. While these studies show that users performed better with the availability of explanations, we argue the fact that both classification tasks were simplified by introducing errors to the model, and explanations were presented to users from AMT oversimplifies the task and deviates the experimental setting from real-world applications, rendering the efficacy of the method to be inconclusive.

4.2.2. Model trust and adoption

As with model debugging, both global and local explanation methods are potentially applicable. However, as the end user is the domain expert, explanations will need to be extended beyond feature attribution while preserving fidelity to what the model has learned. While existing methods discuss user trust as a broad goal, to the best of our knowledge, their ability to help regulators or decision-makers adequately trust ML models is not demonstrated through well-defined evaluations or user trials. The experimental work on the notion of trust has relied on subjective, self-reported measures of trust in performing a simplified task (Ribeiro et al., Reference Ribeiro, Singh and Guestrin2016; Weitz et al., Reference Weitz, Schiller, Schlagowski, Huber and André2019; Buçinca et al., Reference Buçinca, Lin, Gajos and Glassman2020). However, Jacovi et al. (Reference Jacovi, Marasović, Miller and Goldberg2021) in their effort of formalizing the notion of trust in ML, argue that simply asking the user whether they trust the model for a simple task does not evaluate the notion of trust in AI, as the users are not assuming any risk, and they argue that relying on an AI with assumed risk is a prerequisite for trust. A proxy task that is often employed in the evaluation of explainable ML is “forward-simulation” (Ribeiro et al., Reference Ribeiro, Singh and Guestrin2016, Reference Ribeiro, Singh and Guestrin2018; Doshi-Velez and Kim, Reference Doshi-Velez and Kim2017), that is, a person predicting the ML model’s outcome given the input and explanation. This ability to accurately anticipate the model’s output is considered a proxy signal of trust (Hase and Bansal, Reference Hase and Bansal2020; Jacovi et al., Reference Jacovi, Marasović, Miller and Goldberg2021). However, despite these initial efforts, to the best of our knowledge, there have not been experimental efforts that study how existing explainable ML methods impact the notion of trust related to model adoption in decision-making processes, and how it relates to the societal outcomes of interest.

4.2.3. Improving decision-making system performance

Feature attribution-based local explanations are potentially applicable to provide the necessary information to the user. However, feature attribution alone may not be sufficient. Users may need more contextual information such as How does the instance fit into the training data distribution? How does the model behave for similar examples? And what factors did it rely on for those predictions? To that end, there have been some efforts to present visual summaries of explanations to the user (Lundberg and Lee, Reference Lundberg, Erion, Chen, Degrave, Prutkin, Nair, Katz, Himmelfarb, Bansal and Lee2017; Ribeiro et al., Reference Ribeiro, Singh and Guestrin2018; Lundberg et al., Reference Mothilal, Sharma and Tan2018b) which could potentially be useful in this use case. Therefore, available local explanation methods do provide a good starting point.

In contrast to the above use cases, there have been a couple of instances where explainable ML methods were tested using an experimental setting consisting of a real task, real data, and real users. Lundberg et al. (Reference Mothilal, Sharma and Tan2018b) studied the ability of an explainable ML system to assist anesthesiologists to detect hypoxemia risk during surgery for proactive intervention. They showed that their system—Prescience—armed with an ML model and SHAP explanations was able to outperform the anesthesiologists in identifying real-time hypoxemia risk. However, their experiment failed to isolate the marginal effect of the explanations by failing to compare the performance of the ML model + explanations to just the ML model. Therefore, while the combined system with predictions and explanations outperformed the domain expert, it was not possible to isolate whether the effects were due to the ML model prediction alone or due to the combined system. Jesus et al. (Reference Jesus, Belém, Balayan, Bento, Saleiro, Bizarro and Gama2021) studied the impact of presenting ML explanations from three local post hoc explainable ML methods to fraud analysts for assisting fraud detection in credit card transactions. While they organize the experiment to isolate the incremental impact of explanations by running appropriate experiment arms, they make several simplifying assumptions in their experimental design. For instance, they resample the data in their experiment to reflect a 50% rate, whereas, in the real-world context, fraud is a significantly rarer event. Furthermore, they measure the effectiveness of decisions using confusion matrix-based metrics, assuming that all transactions are of the same value, an assumption that is not valid in a real-world business setting. Therefore, while Jesus et al. (Reference Jesus, Belém, Balayan, Bento, Saleiro, Bizarro and Gama2021) have taken steps in the right direction, we argue that their experimental setup still does not reflect the deployment context adequately.

Despite the existence of user studies conducted with real data and real users, the significant limitations of these studies to date mean they fail to provide conclusive evidence that existing explanation methods are effective in assisting humans to make improved decisions in this use case.

4.2.4. How to intervene

As the intervention determinations are often individualized, local explanation methods are potentially applicable for generating the reasons behind the risk score. As with the above use case, users may need more contextual information to supplement the local explanations such as: how the instance fits into the training data distribution, and intervention history for similar—w.r.t. data and w.r.t. explanation—individuals. To the best of our knowledge, there is a lack of evidence in the existing body of work on the efficacy of using these local explanation methods for informing intervention selection.

4.2.5. Recourse

Feature attribution-based local explanations are potentially applicable for deriving reasons behind the decision, and counterfactual explanations can be useful in explaining how to improve the outcomes. The focus of algorithmic recourse work has been on using counterfactual explanations. As simple counterfactual explanations do not guarantee explanations with actionable changes, there has been a range of approaches proposed for deriving counterfactual explanations that are diverse, sparse, plausible, and actionable (Ustun et al., Reference van der Waa, Robeer, van Diggelen, Brinkhuis and Neerincx2019; Karimi et al., Reference Karimi, Barthe, Balle, Valera, Chiappa and Calandra2020, Reference Karimi, Schölkopf and Valera2021b; Mothilal et al., Reference Mothilal, Sharma and Tan2020; Poyiadzi et al., Reference Poyiadzi, Sokol, Santos-Rodriguez, De Bie and Flach2020; Upadhyay et al., Reference Upadhyay, Joshi and Lakkaraju2021). Karimi and colleagues provide a survey of methods for algorithmic recourse including in Karimi et al. (Reference Karimi, Barthe, Schölkopf and Valera2021a). Evaluating methodologies effectively has been a challenge for algorithmic recourse methods and Karimi and colleagues call for better benchmarks (Karimi et al., Reference Karimi, Barthe, Schölkopf and Valera2021a). Therefore, existing method evaluation largely relies on theoretical guarantees and demonstrations using popular experimental datasets such as adult-income, German credit lending, and COMPAS. While those datasets have some connection to the real world, they are not reflective of datasets that most real-world organizations in policy settings have, especially in terms of richness, complexity, and spatiotemporal patterns. Studies that empirically validate the efficacy of these methods are still lacking. Since there are no user studies for any class of models, we rate post hoc local methods with one star.

5.1.1. Guidelines for adequate evaluation of explainable ML in policy contexts

Given the limitations of functional-grounded and human-grounded approaches, evaluation studies of ML explanations in policy contexts should focus on application-grounded approaches. Doshi-Velez and Kim (Reference Doshi-Velez and Kim2017) define an application-grounded evaluation as a study where domain experts perform the intended task. We further extend those requirements and argue that an adequate application-grounded evaluation of an explainable ML method in a policy context cannot exist in the absence of four key elements: (a) A real policy task and related metrics, (b) users who perform the task in the real world, (c) real-world data related to the task that captures the complexities and nuances of the task, and (d) a robust inference strategy that allows conclusions on the incremental impact of explanations. Unfortunately, the relatively small number of application-grounded explainable ML studies that incorporated some aspects of practical evaluation have consistently lacked at least one (and in many cases multiple) of the elements necessary to offer conclusive evidence of real-world efficacy. We elaborate on each of these elements and how existing work violates those requirements below:

Defining the task. The task here entails the decision a user would make, the goals of the decision-making process, and the metrics that evaluate its success. It is imperative to pick tasks that align with a well-defined policy/operational goal and metrics that directly measure the success of the task, going beyond general-purpose metrics such as ROC-AUC, accuracy, and F1-score. For instance, consider the study conducted by Jesus et al. (Reference Jesus, Belém, Balayan, Bento, Saleiro, Bizarro and Gama2021) in a credit-card fraud detection context. While the authors conduct the study with professional fraud analysts performing fraud detection on real-world credit card transactions, they make a simplifying assumption on selecting metrics and choose decision accuracy and other confusion matrix-based metrics. In the context of e-commerce transactions, we argue that the goal is to maximize revenue/profit, and the metric should factor in the transaction value and the relative costs of the two types of errors. Choosing accuracy as the metric ignores both these nuances and thus violates the first requirement of a study. In Poursabzi-Sangdeh et al. (Reference Poursabzi-Sangdeh, Goldstein, Hofman, Vaughan and Wallach2021), the authors used real-world data and a large cohort of real users. However, they choose to use the task of real estate valuation, and how it relates to a decision and an outcome of interest and the utility of explanations in achieving that goal was not established.

Recruiting users. Although their time is often scarce, the domain expert users who act on model outputs must be involved in the evaluation process to ensure it reflects the actual deployment scenario. As the interaction between model predictions, explanations, and users’ domain expertise will dictate the performance of the system, substituting inexperienced users (for instance, from AMT) provides little insight into how well explanations will perform in practice and the results generated (Lou et al., Reference Lou, Caruana, Gehrke and Hooker2013; Lundberg and Lee, Reference Lundberg, Erion, Chen, Degrave, Prutkin, Nair, Katz, Himmelfarb, Bansal and Lee2017; Lundberg et al., Reference Lundberg, Nair, Vavilala, Horibe, Eisses, Adams, Liston, Low, Newman, Kim and Lee2018a; Hu et al., Reference Hu, Rudin and Seltzer2019) may not correlate with results in actual deployment with real users. We argue that even if a study implements explainable ML methods on real-world problems and data that capture the nuances and complexities of the domain, the absence of evaluation with domain experts leads to uncertainty of method efficacy. For instance, Caruana et al. (Reference Caruana, Lou, Microsoft, Koch, Sturm and Elhadad2015) and Zeng et al. (Reference Zeng, Ustun and Rudin2017) describe evaluations of their inherently interpretable models on a real problem with a clear goal and real-world data. However, both studies failed to conduct studies with actual users of the system to evaluate the incremental utility of explanations in terms of performance improvement in the task.

Data. To capture the nuances and characteristics of applying ML to a policy area in practice, the use of data from the problem domain is essential. This is particularly important with evaluating explainable ML methods, as simplified or synthetic datasets might provide an overly optimistic evaluation of their ability to extract meaningful information. Unfortunately, most of the work in this area fails to meet these criteria, focusing only on benchmark ML problems and datasets (e.g., image classification on MNIST data, newsgroups data; Bach et al., Reference Bach, Binder, Montavon, Klauschen, Müller and Samek2015; Ribeiro et al., Reference Ribeiro, Singh and Guestrin2016, Reference Ribeiro, Singh and Guestrin2018; Shrikumar et al., Reference Shrikumar, Greenside and Kundaje2017). While benchmark problems play a crucial role in explainable ML method development and refinement, these problems are far removed from deployment contexts encountered in public policy settings and thus fail to provide convincing evidence of method effectiveness in informing the decisions of domain experts. It is important to note that even in studies that do use data from the problem domain, seemingly trivial simplifying assumptions can violate this requirement. For instance, in Jesus et al. (Reference Jesus, Belém, Balayan, Bento, Saleiro, Bizarro and Gama2021), the authors adjust the class distribution to artificially create a 50–50 distribution to remove the “class imbalance” problem, whereas the actual fraud rate is around 15%. This simplification can impact the expected prior beliefs of fraud analysts and the findings of the study.

Defining the inference strategy. In addition to setting up problems, data, and users consistent with the deployment context, one aspect where existing studies have faltered is the design of the inference strategy of the experiment. It is essential to design the inference strategy to support conclusions on the incremental impact of the explanations in the context. A robust inference strategy entails evaluating the appropriate hypotheses, appropriate control/treatment groups, sufficient sample sizes to preserve statistical power, and analytical methodologies that capture the uncertainties in data. Consider the study conducted by Lundberg et al. (Reference Mothilal, Sharma and Tan2018b), where the authors evaluate an explainable ML system on its ability to support anesthesiologists detect hypoxemia during surgeries. The authors implemented the method on the actual task, recruited domain expert users, and used historical data captured during surgeries. Unfortunately, the study fails to evaluate the incremental impact of the explainable ML method as it does not evaluate the appropriate hypotheses. The authors compare the performance between users making decisions only using data and users making decisions with the help of data, ML prediction, and explanation. The study concludes that users perform better when they have access to ML predictions and explanations compared to only having access to the data. However, the authors fail to compare the performance of users with access to data and ML prediction (no explanations) to the user performance with all three pieces of information. We argue that this is an essential component of an experimental design aimed at evaluating the incremental impact of explainable ML, and this oversight prevents us from attributing that performance increment to the explanations.

As we can see, even the relatively few application-grounded explainable ML studies that have incorporated some aspects of practical evaluation have consistently lacked at least one (and in some cases multiple) of the elements necessary to offer conclusive evidence of real-world efficacy (see Table 4). This gap is particularly acute in the context of public policy problems, given the characteristics (see Section 2) that set them apart from ML problems we encounter in research settings. Therefore, we argue that for implementing explainable ML systems in policy settings, there should be concerted efforts from ML and policy practitioners to conduct well-designed application-grounded evaluation studies.

Table 4. Analyzing the existing handful of application-grounded evaluation studies with respect to the proposed desiderata

Note. Please note that we include studies that at least satisfies one requirement.

5.1.2. An example evaluation study design that satisfies the desiderata

To make the desiderata more concrete, this section provides an overview of an experimental design for evaluating an explainable ML method in a policy context that implements the four necessary elements we presented. While this example describes a setting that uses a post hoc explainable ML method, the same setup could be used to evaluate an inherently explainable model. The policy problem presented here is similar to the one discussed by Bauman et al. (Reference Bauman, Salomon, Walsh, Sullivan, Boxer, Naveed, Helsby, Schneweis, Lin, Haynes, Yoder and Ghani2018).

The policy problem. The Mental Health Center (MHC) of a mid-sized suburban county is establishing a program to conduct proactive mental health outreach to individuals at risk of criminal justice involvement due to unmet behavioral health needs. The MHC has the resources to perform outreach and assist about 100 individuals each calendar month. The broad policy goal is to minimize repeated criminal justice involvement among county residents, and the specific goal of the human–ML collaborative system would be to maximize the efficient use of county resources.Footnote ⁴

ML model. In order to prioritize individuals for outreach, a ML-based predictive model is used. Among the county residents who were released from jail in the last 2 years, the model predicts the risk of each individual being booked into jail within the next 12 months and maps them into an ordinal scale.

Users: Mental health clinicians of the Mobile Crisis Response Team (MCRT) will act on the predictions of the ML model by conducting outreach and offering appropriate services to these individuals based on their needs.

Task and metrics. Given an individual that is deemed to be at risk of future criminal justice involvement (by the ML model), the MCRT clinician is tasked with verifying the risk and selecting them for mental health outreach. In different experimental conditions, the clinicians would have different information pieces at their disposal to make this decision. Since the goal is maximizing efficiency, the objective is to correctly identify people who are actually at risk. Therefore, the metric we should be optimizing for should capture how accurate the MCRT clinicians are at identifying the individuals for outreach, that is, given that an individual is selected for outreach, maximizing the probability that they are actually at risk of being booked into jail the following year, which can be captured by maximizing the positive predictive value (PPV)/precision.

Data: The ML model learns from historical data from the county jail, emergency medical data, and behavioral health service involvement data at the individual level and makes predictions about future criminal justice involvement. The study uses historical data to train and evaluate the model as well as to evaluate the task performance of the MCRT clinicians.

Explainable ML method. The related use case for the task is Use Case No. 03 (deciding whether to intervene) and we use a feature attribution-based post hoc model agnostic explanation method for the evaluation (e.g., SHAP and LIME).

Inference strategy. In this trial, we are interested in learning whether the explainable ML method is effective in helping MCRT clinicians correctly pick individuals for mental health outreach. At a minimum, the trial should evaluate the following hypotheses:

1. 1. MCRT clinicians select individuals for outreach at a higher precision when presented with ML predictions than when they only have access to data.

2. 2. MCRT clinicians select individuals for outreach at a higher precision when they have access to explanations of ML predictions than when they have access only to ML predictions.

In order to evaluate these hypotheses, we need to create three experimental groups/arms where the clinicians have access to different levels of information:

1. 1. Clinicians have access only to the data of the individuals (Group 1).

2. 2. Clinicians have access to the data, and the predicted risk score, for example, a calibrated probability (Group 2).

3. 3. Clinicians have access to the data, predicted risk score, and explanations generated by the post hoc method for the prediction (Group 3).

In the ideal case, we would need a large enough user base where the unit of randomization could be the clinicians, and we could randomly assign clinicians to the three groups. However, in policy applications, having access to a domain expert user base of significant size to enable sufficient statistical power is rare. In that case, we could design the trial in stages like Jesus et al. (Reference Jesus, Belém, Balayan, Bento, Saleiro, Bizarro and Gama2021) did and randomize at the level of data instances (an individual at a specific point in time). While this limits the hypotheses we can evaluate (e.g., how clinicians with different levels of experience use and interact with ML explanations), we can still measure the efficacy of an explanation method in the use case. With this setup, we can compare Group 2 against Group 1 to evaluate the first hypothesis and similarly compare Group 3 against Group 2 for the second.

5.1.3. Importance of evaluating the performance–explainability trade-off (if any) for inherently interpretable models

In addition to evaluating the effectiveness of ML explanations in helping domain experts in public policy settings, it is important to assess the viability of inherently interpretable models in terms of predictive performance. As inherently interpretable models rely on carefully curated input features, exploring the trade-off between performance and scalability in practice is crucial to ascertain their broad applicability. To that end, the models should be implemented on several real policy problems, evaluating: (a) the trade-off between feature preparation efforts and performance and (b) their ability to generalize to future data under strong temporal dependencies.

The prospect of inherently interpretable ML models that are intelligible without sacrificing predictive performance certainly holds considerable appeal. However, their current implementations are limited to a handful of practical contexts. To understand potential trade-offs in practice, we must rigorously test these models against more opaque models across problem domains. Even if there are performance limitations, there may be critical applications where the intelligibility of the model cannot be compromised and some applications where there could be built-in guardrails to protect against unintended harm. Understanding the limitations of methods through experimentation will help practitioners make more informed implementation decisions and support the complete spectrum of use cases.