Computer Science Department
School of Computer Science, Carnegie Mellon University


Modeling and Interpreting Multimodal Inputs:
A Semantic Integration Approach

Minh Tue Vo, Alex Waibel

December 1997

Keywords: Multimodal human-computer interaction, multimodal input modeling, multimodal interpretation

Modern user interfaces can take advantage of multiple input modalities such as speech, gestures, increase robustness and flexibility. The construction of such multimodal interfaces would be greatly facilitated by a unified framework that provides methods to characterize and interpret multimodal inputs. In this paper we describe a semantic model and a multimodal grammer structure for a broad class of multimodal applications. We also present a set of grammar-based Java tools that facilitate the construction of multimodal input processing modules, including a connectionist network for multimodal semantic integration.

23 pages

Return to: SCS Technical Report Collection
School of Computer Science homepage

This page maintained by