Robotics and cyberphysical systems are increasingly being deployed to settings where they are in frequent interaction with the public. Therefore, failures in these systems can be catastrophic by putting human lives in danger and causing extreme financial loss. Large-scale assessment of the quality of these systems before deployment can prevent these costly damages.

Because of the complexity and other special features of these systems, testing, and more specifically automated testing, faces challenges. In this dissertation, I study the unique challenges of testing robotics and cyberphysical systems, and propose an end-to-end automated testing pipeline to provide tools and methods that can help roboticists in large-scale, automated testing of their systems. My key insight is that we can use (low-fidelity) simulation to automatically test robotic and cyberphysical systems, and identify many potentially catastrophic failures in advance at low cost.

My core thesis is: Robotic and cyberphysical systems have unique features such as interacting with the physical world and integrating hardware and software components, which creates challenges for automated, large-scale testing approaches. An automated testing framework using software-in-the-loop (low-fidelity) simulation can facilitate automated testing for these systems. This framework can be offered using a clustering approach as an automated oracle, and an evolutionary-based automated test input generation with scenario coverage fitness functions.

To support this thesis, I conduct a number of qualitative, quantitative, and mixed method studies that 1) identify main challenges of testing robotic and cyberphysical systems, 2) show that low-fidelity simulation can be an effective approach in detecting bugs and errors with low cost, and 3) identify challenges of using simulators in automated testing.

Additionally, I propose automated techniques for creating oracles and generating test inputs to facilitate automated testing of robotic and cyberphysical systems. I present an approach to automatically generate oracles for cyberphysical systems using clustering, which can observe and identify common patterns of system behavior. These patterns can be used to distinguish erroneous behavior of the system and act as an oracle.

I evaluate the quality of test inputs for robotic systems with respect to their reliability, and effectiveness in revealing faults in the system. I observe a high rate of non-determinism among test executions that complicates test input generation and evaluation, and show that coverage-based metrics are generally poor indicators of test input quality. Finally, I present an evolutionary-based automated test generation approach with a fitness function that is based on scenario coverage. The automated oracle and automated test input generation approaches contribute to a fully-automated testing framework that can perform large-scale, automated testing on robotic and cyberphysical systems in simulation.

142 pages

Thesis Committee:
Claire Le Goues (Chair)
Michael Hilton
Eunsuk Kang
John-Paul Ore (North Carolina State University)

James D. Herbsleb, Director, Institute for Software Research
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu