Recent advancements in AI have made it possible for designers to produce images, videos, and 3D content from natural language. However, text prompts require designers to articulate exactly what they want before they have seen what is possible. Designers' best tools often work the opposite way: sketches, moodboards, and CAD let designers work directly on the design itself, drawing, comparing, and adjusting as the artifact takes shape. I argue that AI as a design material should work the same way. Inside, AI models hold a rich understanding of style, structure, and semantic relationships that go well beyond what a text prompt can reach, and internal model controls that designers should be able to discover and manipulate. This gap matters most in design exploration, where designers need to move through wide possibility spaces, combine and remix ideas, and rapidly iterate.

Effective AI steering interfaces for design exploration come from aligning three things: the tasks designers want to do, the control mechanisms a model exposes, and the interface representations that put those mechanisms in the designer’s hands. This thesis refers to this alignment as the Human-AI-Interface (HAI) Triangle. Alignment is operationalized through three vertex conditions: controllability (the AI mechanism exposes handles that a designer can act on), affordance (the controls match the moves the task requires), and intuitiveness (the representation draws on conventions designers already work with). The framework is grounded in four systems, each addressing a distinct design task. VideoMap addresses the navigate task: it projects latent video embeddings onto a 2D map that editors can explore to find visual and semantic connections across footage. Jigsaw addresses the compose task: it represents pre-trained models as puzzle pieces that snap together, so designers can build multimodal AI pipelines without writing code. Inkspire addresses the iterate task: it pairs iterative sketching with AI-generated scaffolds, allowing designers to refine through strokes rather than through prompt edits. GENErator addresses the remix task: it lets designers mix and evolve style-specific AI models in a shared semantic space, surfacing aesthetic directions they could not have specified up front. Across user evaluations, participants navigated, composed, iterated on, and remixed AI capabilities through intuitive and controllable interactions, accomplishing tasks they could not with conventional interfaces.

192 pages

Thesis Committee:
Nikolas Martelaro (Chair)
Aniket Kittur
David Lindlbauer
Michael Terry (Google DeepMind)

Brad A. Myers, Head, Human-Computer Interaction Institute
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science homepage

This page maintained by reports@cs.cmu.edu