Software engineering practice is filled with myths–beliefs derived from intuition, experience, and conventional wisdom–that lack empirical foundations. Many of these myths are causal in nature, concerning the consequences of decisions, interventions, and changes to practice. While empirical software engineering research frequently investigates these myths through descriptive, correlational, and qualitative evidence, it rarely engages with the challenging question of whether the evidence can actually support the causal claims beneath these myths. In this thesis, I argue that causal credibility assessments are feasible in empirical software engineering research through explicit articulations of causal theory and assumptions.

To demonstrate this argument, this thesis starts with an accessible introduction to the modern causal inference toolkit for software engineering researchers. Then, across three empirical studies, I demonstrate how causal credibility assessments can reveal systematic gaps between practitioner beliefs and empirical reality. First, using counterfactual analysis and simulations on npm dependency data, I challenge the myth that pinning dependency versions improves project security, showing that it paradoxically increases attack exposure unless upstream packages pin their own dependencies. Second, through a longitudinal analysis of GitHub starring patterns with panel regression models, I challenge the myth that fake GitHub stars provide sustainable promotional benefits, showing instead that these artificial signals become long-term liabilities. Finally, using a difference-in-differences design with staggered adoption, I challenge the myth that AI coding tools unambiguously improve developer productivity, finding that Cursor's initial velocity gains are offset by sustained accumulation of technical debt. In each study, I assess the credibility of causal claims through the following four-step framework: (1) deriving an explicit causal theory from software engineering domain knowledge, (2) defining the causal estimate of interest and specifying the assumption under which the estimate carries a causal interpretation, (3) honestly acknowledging and mitigating limitations and known caveats, and (4) thoughtfully navigating alternative explanations. Collectively, this thesis provides a methodological blueprint for future empirical software engineering research, illustrating how to balance rigor with practicality to generate actionable, trustworthy evidence for software engineering myths.

226 pages

Thesis Committee:
Bogdan Vasilescu (Co-chair)
Christian Kästner (Co-chair)
Rohan Padhye
Narayan Ramasubbu (University of Pittsburgh)

Nicolas Christin, Head, Software and Societal Systems Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu