Computer Science Department
School of Computer Science, Carnegie Mellon University
Two Case Studies in Using Cognitive Walkthroughs for Interface Evaluation
Niels Ebbe Jacobsen*, Bonnie E. John
Also appears as Human-Computer Interaction Institute Technical Report
Keywords: Usability evaluation methods, cognitive walkthrough
Previous HCI-studies have compared usability evaluation methods
quantitatively without supplementing these data with detailed
qualitative data about how analysts actually learn and use methods. In
contrast, we present two diary-based case studies that describe the
processes of two novice analysts who learned about and applied the
Cognitive Walkthrough (CW; Lewis, et al., 1990) to the specification of
a multimedia authoring system. Results show that the two analysts easily
learned to use CW but also that they found the technique tedious to use.
Moreover, CW was neither reliable when comparing the two analysts'
processes and outcomes to each other, nor accurate when comparing the
analysts' problem predictions to results from usability tests applied to
a running system. We examine these data in detail, searching for
possible causes of the observed reliability and accuracy. Based on these
analyses, we suggest three changes to CW method to improve its accuracy
and two changes to improve its reliability. Further, we recommend
developing a tool to reduce the tedium and integrate our suggested
improvements to CW.
*Nokia Mobile Phones, Copenhagen, Denmark, Niels.Jacobsen@nokia.com