Software development is an inherently multimodal activity. Developers reason about systems through mental models, and two decades of empirical research show that they externalize those models visually: as whiteboard drawings, napkin diagrams, and informal sketches that mix boxes, arrows, and annotations at a deliberately high level of abstraction. These artifacts are not decoration around the real work; they are the primary medium through which developers build and share understanding of the systems they create.

Modern AI code assistants ignore this practice almost entirely. Built around Large Language Models (LLMs), they consume and produce text: developers must compress visual reasoning into linear prompts, and receive in return large volumes of generated code with no visual handle for inspecting it. Even models that accept images treat them as auxiliary context rather than as carriers of intent, discarding the structure, ordering, and dependencies that a sketch encodes. The consequences are well documented: developers struggle to express design intent to these tools, over-trust generated code that looks plausible, and miss subtle flaws in output they did not write. As AI assistants shift developers from authors to reviewers of code, this mismatch between how developers think and how their tools communicate becomes increasingly costly.

This dissertation demonstrates that AI code assistants can interpret, produce, and maintain the informal visual artifacts developers already use, and that doing so measurably improves how developers express intent to and verify the output of these systems. We focus on three processes of AI-assisted development where the absence of a visual channel is most costly, and contribute systems and evaluation methodologies for each.

1. ) Expressing intent to a code generator through text alone forces developers to abandon the sketches in which they naturally plan. To address this, we contribute the first Visual Code Assistant, an in-IDE tool that treats informal, hand-drawn sketches of machine learning workflows as first-class specifications and converts them into executable Jupyter notebooks. A user study with 19 data scientists produced a dataset of paired sketches and notebooks, an empirical characterization of how developers sketch, and an LLM-as-a-Judge evaluation methodology, validated against human grading, that shows such sketches are viable specifications for code generation.
   
2. ) Understanding code, especially AI-generated code, is hard, and existing tools explain code with yet more text. To address this, we build VisDocSketcher, an agentic system that generates informal, developer-style diagrams directly from source code, together with AutoSketchEval, a reference-free framework that evaluates a diagram by reconstructing code from it, sidestepping the fact that no canonical reference diagram exists. We then establish that such diagrams actually help: in a controlled within-subjects study with 26 developers, JupyterDraw, a tool that renders whiteboard-style overviews of Jupyter notebooks, raised complete detection of silent methodological flaws in AI-generated notebooks from 29% to 67% of review sessions, at no cost in time or workload, and reviewers credited the diagram as their primary detection route five times as often as an equivalent textual summary.
   
3. ) Software does not hold still, so any generated diagram is stale documentation in the making. To address this, we contribute Visual Loop, a framework in which sketch and code co-evolve: it maintains fine-grained correspondences between visual regions and code segments, so that an edit to either artifact propagates locally to the affected parts of the other rather than forcing wholesale regeneration.

133 pages

Thesis Committee:
Jonathan Aldrich (Co-Chair, Carnegie Mellon University)
Rui Abreu (Co-Chair, University of Porto / META)
Vincent Hellendoorn (Co-Chair, Carnegie Mellon University / Google DeepMind)
Bogdan Vasilescu
David Lo (Singapore Management University)
João Saraiva (University of Minho)

Nicolas Christin, Head, Software and Societal Systems Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu