Structured objects such as tables, JSON records, and tool outputs reach decoder- only language models as flat token sequences. This creates a structural addressing problem: locating the relevant part before using its content. I study single-cell table lookup as a controlled case through three linked questions: which cues make a row–column address recoverable and which errors they repair; whether lookup decomposes into identifiable internal operations; and whether causal interventions support that decomposition and distinguish it from simpler accounts of format effects.

I evaluate real tables in a 48-condition grid spanning two open-weight families, table formats, prompts, and row-index conditions, combining behavioral evaluation, attention analysis, and causal interventions. Explicit numeric row indices improve exact-match accuracy by 16–33 percentage points in every tested condition, chiefly by repairing wrong-row errors. Reversal and aligned-pair label swaps indicate that they can serve as content-addressable rather than ordinal landmarks. Across both families, attention profiles place column-name matching before correct-cell retrieval; ablation and activation patching in the primary Llama regime provide causal support for this two-stage account. The division of labor recurs across families, but head identities and causal concentration differ: the named heads form a causal core, not a complete circuit. Format interventions further implicate exact, nameable content rather than pipe counting in the localized column-matching route.

Although the evidence covers only short, flat tables with numeric row keys in an atomic lookup task, it suggests that representations can act as computational scaffolds. Exposing stable landmarks and testing how models use them could inform structured-data interfaces designed jointly for accuracy, robustness, and inference cost.

103 pages

Thesis Committee:
Carolyn Rosé (Chair)
Daphne Ippolito

Jignesh Patel, Interim Head, Computer Science Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu