AI agents can now operate real user interfaces, produce visual artifacts, and carry out multi-step tasks from a natural-language request. Capability alone does not always decide who should make each choice. At many points in a task, an agent has to choose between acting on its own and bringing the person into the decision. The stakes are higher for blind and low-vision (BLV) people, who often cannot inspect the visual context the agent is working from. An agent may quietly pick among options the user never knew existed, make an error the user cannot verify, or settle on a result that looks reasonable but leaves the user no way to tell whether it is any good, let alone to their taste.

This dissertation takes up the question of who should choose in two parts. The first part asks how an agent can preserve the person's agency while it executes. I built a UI agent that detects steps where the instruction leaves a real choice open, pauses there, describes the alternatives, and lets blind and low-vision users supply the preference the agent is missing [1]. In a study over real-world web tasks, participants working with this agent completed more tasks and reached outcomes that matched their intent more often than participants relying on fully autonomous agents. Interaction of that kind rests on perception, so I also developed a synthetic data pipeline that generates slides and interfaces as code, keeping structural labels attached, which lets models learn to parse visual layouts without large hand-annotated datasets [2]. The same question extends beyond a single application to open-ended activity, where an assistant has to recognize what a person is doing and what help they need [3], and to place events in time before it can judge when to step in [4].

The second part asks how an agent should act once a person hands a choice over (especially a visual one). When blind presentation authors used a system I built to describe visual design changes in shared slides [5], they asked for more than a record of what changed; they wanted enough access to the visual decisions to judge and revise the result. Some design judgment is shared across people, and a model trained on synthetic design flaws and professional ratings can score interface quality in ways that designers broadly agree on [6]. A shared standard only goes so far. I collected twelve thousand pairwise judgments from twenty professional designers over the same generated interfaces [7]; the designers disagreed with one another substantially, and models personalized to a single designer predicted that designer's choices better than models trained on much larger pooled data. The dissertation closes with a computational method that learns a new user's design preferences through a short adaptive onboarding and reranks the candidates of a generative UI model [8]. In an online study, a few pairwise choices guided personalized generation more effectively than a written description of the user's taste.

This dissertation argues that agency in human-agent collaboration has to be allocated deliber ately, decision by decision, rather than granted to the agent up front. The argument is rooted in accessibility, where the cost of a silent choice is easiest to see, but it applies wherever a person hands work to an agent. Each decision should sit depends on how uncertain the agent is, whether the person can reach the relevant state on their own, how much the outcome rests on personal judgment, and how difficult a mistake would be to detect or reverse. Recognizing the moments when a decision belongs to the person requires grounded perception and temporal context. Acting on a choice the person has delegated means learning that person's preferences quickly, because disagreement between people is information about individuals rather than noise to average away. These requirements point toward agents that weigh task progress against the person's control and taste, revisit that balance at every step as circumstances change, and carry what they learn about a person from one task to the next.

83 pages

Thesis Committee:
Jeffrey P. Bigham (Co-chair)
Amy Pavel (Co-chair) (University of California, Berkeley)
Jodi Forlizzi
Graham Neubig
Ming-Hsuan Yang (University of California, Merced / Google DeepMind)

Brad A. Myers, Head, Human-Computer Interaction Institute
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science homepage

This page maintained by reports@cs.cmu.edu