Explainable AI (XAI) methods have traditionally aimed to provide people with an understanding of why and how AI systems behave. Recognizing the potential benefits this can have for AI-assisted decision-making, researchers, media, and funding agencies have turned to XAI as a solution to help people validate, interpret, or justify AI outputs in high-stakes decision-making contexts. We refer to this paradigm as output-oriented XAI. While some controlled studies have shown that advancements in XAI techniques can lead to improved human–AI decision-making, other studies have demonstrated that these efforts have resulted in negative outcomes. Human-centered AI researchers argue that these inconsistent findings stem from a "one-size-fits-all" approach that ignores decision-makers diverse expertise, roles, and information needs. Building on the one-size-fits-all critique, this thesis argues that these mixed results stem from two additional limitations: the assumption that XAI methods lack imperfections and the narrowly scoped design space centered on output-oriented XAI. The existing focus on designing output-oriented XAI risks overlooking the emerging uses of AI explanations that can support complex, human–AI decision-making processes beyond validating, interpreting, or justifying AI outputs.

To address the ongoing challenges that XAI faces in human–AI collaboration, Part I of this thesis presents two empirical studies with crowdworkers and domain stakeholders that expose the impacts of existing output-oriented XAI designs. These studies show how such explanations can be misaligned, misleading, or imperfect, shaping decision-makers' reliance on AI. Although these findings may suggest XAI is not well-suited for human–AI collaboration, Part II presents two exploratory studies that investigate the emerging uses of AI explanations in two different human–AI decision-making contexts. Across two different AI systems, these studies highlight three emerging uses of AI explanations: supporting sensemaking, augmenting communication, and facilitating knowledge transfer. These emerging uses reveal needs that AI explanations could support, but that are not well supported by existing output-oriented XAI designs. However, recent work suggests that generative AI (GenAI) may be able to support such needs. Therefore, Part III first examines the explanatory potential of multimodal GenAI in medical imaging interpretation workflows. Then, the second half of Part III presents process-oriented XAI, a reframing of the XAI design space that positions XAI techniques as a support tool for users throughout workflows. We build a process-oriented XAI system for medical imaging interpretation and investigate its impact on the clinical reasoning processes of medical students and radiology residents.

This thesis expands the design space for AI explanations by acknowledging the impact of their imperfections, exploring their emerging uses, and proposing a process-oriented XAI paradigm. Across six empirical studies, this work challenges traditional conceptions of XAI and advances our understanding of how to design for, evaluate, and think about XAI in human–AI collaborations. In conclusion, we encourage researchersto design XAI for uses beyond validation, interpretation, or justification of AI outputs.

231 pages

Thesis Committee:
Adam Perer (Chair)
Motahhare Eslami
John Zimmerman
Xiang 'Anthony' Chen (University of California, Los Angeles)

Brad A. Myers, Head, Human-Computer Interaction Institute
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science homepage

This page maintained by reports@cs.cmu.edu