Computing devices have historically maintained a balance of input and output capabilities for positive and productive user experience. However, new and upcoming mobile devices, particularly mixed reality (XR) headsets and smart glasses, have made trade-offs that prioritize mobility and wearability to enable always available content display at the cost of usable input. While advances in display and computation have increased the potential of these devices, limitations in input continue to constrain what users can actually do with them. This has led to an input gap that threatens to increase frustration, confusion, and inefficiency when users are not given enough input capability to interact with the interfaces presented to them.

Prior research has overly focused on improving explicit input techniques, which ignores the physical context wearable devices are inherently situated in. In contrast, in this thesis, I explore an alternative approach: implicit input which uses sensors worn on the body to detect behaviors that occur naturally around devices. The primary part of the body that I focus on is the hands, as hands are the dominant means by which people interact with the world. I present three approaches that demonstrate how implicit input can expand interaction by enabling devices to understand what a user is touching, how they are moving, and which objects they are interacting with. For each approach, through several proof-of-concept systems I show the effectiveness of novel sensing for enabling new forms of implicit input. Together, these systems illustrate how sensing everyday hand and body behaviors can meaningfully augment existing input. I argue that the rise of AI-driven mobile applications further exposes the shortcomings of current input frameworks, which provide limited user context and keep devices largely reactive. As a whole, my research agenda positions implicit sensing of hand interactions as a foundation for more proactive, context-aware devices that both better leverage advances in AI and help close the growing input gap in mobile computing.

178 pages

Thesis Committee:
Haiyi Zhu (Co-chair)
Sarah Fox (Co-chair)
Sherry Tongshuang Wu
Oliver Haimson (University of Michigan)
Amy Bruckman (Georgia Institute of Technology)

Brad A. Myers, Head, Human-Computer Interaction Institute
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science homepage

This page maintained by reports@cs.cmu.edu