Computer vision-enabled autonomous drone flight creates challenges with reproducibility and behavior assessment due to inherent variability in the models that provide cognitive function to the vehicle. These challenges are most visible when attempting to validate control logic in the automation pipeline and when benchmarking for the purpose of making objective assessments on agility and maneuverability. In this work we begin by exploring the current state of simulation technology and its particular applications in the development and fielding of autonomous drone systems. This exploration exposes a sudden jump from simulation-heavy prototyping to end-to-end full system testing involving physical flight that increases the difficulty of validating correctness and incurs unnecessary risk.

We present the concept of a SensorTwin, a novel application of simulation technology aimed at facilitating behavior validation and performance benchmarking for vehicle-automation pipeline systems. We provide a concrete implementation of a SensorTwin built over the SteelEagle Autonomous Drone system and demonstrate proof of concept by evaluating the performance of the SensorTwin-augmented system against existing cognitive task evaluations. We show how idealizing the visual feed and computer vision components of a real system contributes to both benchmarking and validation when applied to a system during actual, physical flight.

We also introduce a novel set of tracking agility evaluation tests, referred to as the Autonomous Tracking Agility Benchmark, with the intent of enabling future relative comparison across vehicle platforms, autonomy pipelines, and cognitive function implementations. An initial dataset for the benchmark suite is produced through physical flight augmented by the SensorTwin implementation and analyzed to provide objective and subjective insights on agility assessment specific to the automated drone setting.

SensorTwin Demo Video

62 pages

Thesis Committee:
Mahadev Satyanarayanan (Chair)
Padmanabhan Pillai
Mihir Bala (Cornell University)

Jignesh Patel, Interim Head, Computer Science Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu