As artificial intelligence (AI) systems are introduced in a range of socially consequential settings, measuring their capabilities, risks, and limitations has become crucial. Yet many properties of AI systems–such as the "helpfulness" of a chatbot response, or the "fairness" of a predictive algorithm–are unobservable latent constructs. Obtaining valid, reliable, and informative measurements of such constructs is challenging in practice, given both their contested nature and the often limited resources available to measure them effectively. In response, this thesis introduces the concept of a measurement intervention: a targeted approach designed to help organizations improve the validity of AI system performance measurements under practical constraints.

The first part of this thesis advances measurement interventions in the context of predictive modeling for algorithmic decision support (ADS). I propose a conceptual framework that characterizes threats to the validity of predictive models used in ADS. I then report findings from qualitative interviews with data scientists, which illustrate the nuanced trade-offs they make while balancing the validity of a predictive modeling formulation with competing desiderata, such as its resource requirements and portability across institutional contexts. Based on these findings, I devise two statistical approaches–"expert anchors" and "uncertainty cancellation"–that help teams more effectively repurpose existing organizational data for new measurement tasks.

The second part of this thesis advances measurement interventions for Generative AI (GenAI) evaluation. I first conduct a formative study with sixteen industry AI practitioners, to examine their current practices for assessing the validity of GenAI evaluations. I find that, while Subject Matter Experts (SMEs) often have a rich understanding of latent constructs being measured in GenAI outputs, AI practitioners currently lack mechanisms for effectively. incorporating this expertise into the design of GenAI measurement instruments.

Based on these findings, I introduce RubricStudio: an interactive system that supports cross-functional teams of SMEs and AI practitioners in operationalizing validity principles while designing GenAI evaluations. RubricStudio automatically flags potential validity issues in evaluation results, and provides scaffolding designed to help SMEs and AI practitioners diagnose and resolve issues together through synchronous co-refinement sessions. I evaluate RubricStudio through a controlled between-subjects study, which shows that RubricStudio's validity scaffolding helps teams make more targeted diagnoses of validity-related issues, and creates opportunities for SMEs to directly shape the course of the evaluation design process.

Finally, I propose two statistical measurement interventions, designed to help teams scale up GenAI evaluations via an LLM-as-a-judge–i.e., a second GenAI system that acts as an automated evaluator. The first intervention proposes a multi-label rating elicitation, aggregation, and performance measurement framework for validating LLM-as-a-judge systems on subjective measurement tasks, where more than one rating may be reasonably considered "correct." The second leverages a doubly-robust estimation approach to yield informative evaluation results when human ratings are scarce but LLM-as-a-judge ratings cheap and abundant. Based on these overall findings, I conclude by reflecting on the opportunities and challenges for measurement interventions to advance the evaluation of future human-AI systems.

199 pages

Thesis Committee:
Kenneth Holstein (Co-Chair)
Zhiwei Steven Wu (Co-Chair)
Jeffrey Bigham
Alexandra Chouldechova (Abridge)

Brad A. Myers, Head, Human-Computer Interaction Institute
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science homepage

This page maintained by reports@cs.cmu.edu