Computer Science Department
School of Computer Science, Carnegie Mellon University
Modeling and Interpreting Multimodal Inputs:
A Semantic Integration Approach
Minh Tue Vo, Alex Waibel
Keywords: Multimodal human-computer interaction, multimodal input
modeling, multimodal interpretation
Modern user interfaces can take advantage of multiple input modalities such
as speech, gestures, handwriting...to increase robustness and flexibility.
The construction of such multimodal interfaces would be greatly facilitated
by a unified framework that provides methods to characterize and interpret
multimodal inputs. In this paper we describe a semantic model and a
multimodal grammer structure for a broad class of multimodal applications.
We also present a set of grammar-based Java tools that facilitate the
construction of multimodal input processing modules, including a connectionist
network for multimodal semantic integration.