Machine learning (ML) models are now commonly used as components in systems. Due to their black-box and data-driven nature, ML components can produce erroneous outputs (in the form of mispredictions) that may critically impact the system's quality of service. Such mispredictions may be caused by component changes, environment changes, or due to ML components' inherent uncertainty and inaccuracy. In the face of these changes, and to cope with mispredictions, self-adaptation arises as a natural solution: systems that monitor and adapt themselves at run time to optimize their system utility.

This thesis provides a repertoire of ML adaptation tactics, and a frameworkthat generates policies specifying when to apply each tactic to adapt ML components such that overall system utility is optimized. The development of this framework raises two main challenges: (i) estimating the expected costs and benefits due to the execution of an adaptation tactic; (ii) evaluating the impact of the improved ML predictions on overall system utility. To address the first problem we build predictors that learn to estimate the expected benefits of each adaptation tactic. To solve the second problem we leverage probabilistic model checking methods and instantiate a formal model of the system, capturing the key dynamics of ML components and their impact on expected system utility.

The techniques proposed in this thesis are evaluated via two use-cases: credit card fraud detection and machine translation systems. We show that: the self-adaptive ML-based systems built leveraging the proposed framework achieve better system utility than that achievable when employing simpler baselines to guide adaptation, such as periodic or reactive adaptation schemes; the proposed framework is suitable for run-time adaptation in non-critical domains; the framework can be extended to account for multiple adaptation tactics; and that it can be leveraged to plan for the long term when to adapt ML models

164 pages

Thesis Committee:
David Garlan (Chair, Carnegie Mellon University)
Paolo Romano (Co-chair, IST, University of Lisbon)
Christian Kästner (Carnegie Mellon University)
Bruno Martins (IST, University of Lisbon)
Grace Lewis (Software Engineering Institute)
Valeria Cardellini (University of Roma Tor Vergata)

James D. Herbsleb, Head, Software and Societal Systems Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu