Visual representations like diagrams are powerful tools for thought. Diagrams are used extensively to understand abstract relationships, explain complex ideas, and solve difficult problems.

I conducted an interview study to understand how domain experts create diagrams and identified key limitations in current tools. To illustrate concepts effectively, experts find appropriate visual representations and translate concepts into concrete shapes. This translation step is not supported explicitly by existing diagramming tools.Our participants reported how they create, adapt, and reuse visual representations using both sketches and digital tools. However, they had trouble using digital tools to transition from sketches and reuse components from earlier diagrams. Based on these results, we suggest four opportunities of diagramming tools–exploration support, representation salience, live engagement, and vocabulary correspondence–that together enable a natural diagramming experience.

The findings from these studies informed the design of PENROSE, a language-based system that allows authors to codify domain-specific concepts and their visual representations. In PENROSE, the visual representation is user-defined in a constraint-based specification language; diagrams are then generated automatically via constrained numerical optimization. The system is designed to be user-extensible to many domains. In contrast to tools that specify diagrams via direct manipulation or low-level graphics programming, PENROSE enables rapid creation and exploration of diagrams that faithfully preserve the underlying visual representation. I demonstrate the effectiveness and generality of the system by showing how it can be used to illustrate a diverse set of concepts from various domains.

Atop PENROSE, I built EDGEWORTH, a tool designed to help educators easily create visual problems. EDGEWORTH works in two main ways: firstly, it takes a single diagram from the user and systematically alters it to produce many variations, which the educator can then choose from to create multiple problems. Secondly, it automates the layout of diagrams, ensuring consistent high quality without the need for manual adjustments. I collected a dataset of diagrammatic multiple-choice problems to show that EDGEWORTH can create problems in three domains: geometry, chemistry, and discrete math. EDGEWORTH generated usable answer options within the first 10 diagram variations in 87% of authored problems. I then performed a user study to measure authors' efficiency at creating translation problems using EDGEWORTH, compared with a conventional drawing tool. The results show that once authors make a correct diagram, they are about 3 times faster at making diagrammatic options for translation problems using EDGEWORTH compared to Google Drawings. Finally, in response to walkthrough demonstrations, expert educators gave positive feedback on EDGEWORTH's utility and the real-world applicability of its outputs.

PENROSE and EDGEWORTH demonstrate that codifying visual representations allow diagrams authors to reuse their design effort, produce new diagrams faster, and thus make diagrams at a larger scale.

226 pages

Thesis Committee:
Kenneth Koedinger (Co-chair)
Joshua Sunshine (Co-chair)
Brad Myers
TItus Barik (Apple)
Shriram Kirshnamurthi (Brown University)

Nicolas Christin, Head, Software and Societal Systems Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu