Using complex interfaces can be quite challenging, especially for those who do not enjoy exploring unfamiliar interfaces through trial and error. Existing tutorial systems can be cumbersome, and sometimes difficult to use. While augmented reality has been adopted to help users learn to use some interfaces, these AR user manuals are usually designed specifically for one single interface, so creating AR user manuals for a variety of interfaces requires a substantial amount of work, and the authoring tools for these manuals are usually not friendly to users without background knowledge in related areas such as programming and 3D modeling. To solve this problem, we present a system that automatically generates interactive AR user tutorials by demonstration. Our system first guides experienced users to add references that help our system understand the interface and to demonstrate steps to complete certain tasks. Then our system automatically detects user interactions and asks users for supplemental information on the interactions. Based on this information, our system generates an action sequence for each task, and provides a combination of text instructions, visual and audio guidance for new users to access the interface.

44 pages

Thesis Committee:
Jeffrey P. Bigham (Chair)
Patrick Carrington

Srinivasan Seshan, Head, Computer Science Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu