Human-Computer Interaction Institute
School of Computer Science, Carnegie Mellon University
Developing Handwriting-based Intelligent
This dissertation shows that handwriting provides usability benefits in that speed of entry increases, user error decreases, and user satisfaction increases. Furthermore, it shows that handwriting may also provide learning benefits: students solving the same problems by handwriting as others who are typing experience a faster learning rate. Specific math advantages of using handwriting are: a reduction in extraneous cognitive load due to the affordance of handwriting for more direct manipulation, and improved support for the two-dimensional spatial information which is inherently meaningful in mathematics (e.g., vertical fraction notation). This dissertation investigates these factors and their impact.
One concern with the use of handwriting in intelligent tutoring systems, however, is that recognition technology is not perfect. To the extent that the system cannot be confident of correctly recognizing what the student is writing, it cannot identify tutoring opportunities and provide detailed, step-targeted feedback. Therefore, a trade-off is clear between the difficulty of improving recognition accuracy and the need to support step-targeted feedback. One strategy to address this trade-off is using a type of instruction based on worked examples, which provide a sort of feed-forward to guide learners. A second strategy is to investigate technical approaches to improving handwriting recognition accuracy. This dissertation explores two methods of enhancing baseline recognition: training the recognition engine on a data corpus of student writing in order to maximize writer-independent recognition accuracy; and making use of domain-specific context information on the fly to refine the recognition hypotheses.
The approach taken in this dissertation includes technical development, pedagogical development, and user studies. Topics addressed include what the advantages of using handwriting are, how the above factors contribute to these advantages, and how these advantages can be leveraged in real tutoring systems. Reasonable writer-independent handwriting recognition rates can be achieved by a priori data collection and training, and these can be even further improved via the addition of domain-context information. Furthermore, a realistic tutoring interaction paradigm can be achieved through these methods, in spite of imperfect raw recognition accuracy. This dissertation leaves the door open to continued work on basic recognition technology which can improve the achievements reported here even further.