Over the past decade, wearable computing devices ("smart glasses") have undergone remarkable advancements in sensor technology, design, and processing power, ushering in a new era of opportunity for high-density human behavior data. Equipped with wearable cameras, these glasses offer a unique opportunity to analyze non-verbal behavior in natural settings as individuals interact. Our focus lies in predicting engagement in dyadic interactions by scrutinizing verbal and non-verbal cues, aiming to detect signs of disinterest or confusion. Leveraging such analyses may revolutionize our understanding on human community, foster more effective collaboration in professional environments, provide better mental health support through empathetic virtual interactions, and enhance accessibility for those with communication barriers.

In this work, we collect a dataset featuring 34 participants engaged in casual dyadic conversations, each providing self-reported engagement ratings, augmented with external raters' assessments of engagement. We introduce a novel fusion strategy using Large Language Models (LLMS) to integrate multiple behavior modalities into a "multimodal transcript" that can be processed by an LLM for behavioral reasoning tasks. This fusion method is one of the first to approach "reasoning" about real-world human behavior through a language model. This work also explores the creation and features derived from LLMs for multimodal models to aid the task of engagement modeling. Smart glasses provide us the ability to unobtrusively gather high-density multiomdal data on human behavior, paving the way for new approaches to understanding and improving human communication with the potential for important societal benefits. The features and data collected during the studies will be made publicly available to promote further research.

57 pages

Thesis Committee:
Fernando De la Torre (Chair)
Daphne Ippolito
Lori L. Holt (University of Texas at Austin)

Srinivasan Seshan, Head, Computer Science Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu