Recently, Large Reasoning Models (LRMs) have achieved impressive performance on a variety of reasoning tasks, including mathematics and code generation. Their long chains of reasoning enable scaling of inference-time computation, allowing them to solve increasingly complex problems. LRMs are typically post-trained from base models using supervised fine-tuning (SFT), reinforcement learning (RL), or a combination of both. RL is often hypothesized to be a key driver of reasoning ability, as it enables models to explore and discover new solutions. However, recent work suggests that RL may instead concentrate probability mass on existing solutions. The mechanisms by which RL leads to reasoning ability remain poorly understood, and modern RL pipelines bundle on-policy rollouts, variance normalization, KL divergence, entropy shaping, and other components into a single procedure, making it difficult to attribute gains to any specific cause. In this thesis, we study RL training mechanisms through the lens of compositional generalization–a key sub-skill of reasoning that involves combining atomic skills to solve more complex problems. We propose a unified two-axis framework that organizes SFT and RL methods along a data axis (off-policy to on-policy) and a loss function axis (positive-only to positive-plus-negative to GRPO). This framework places existing post-training algorithms as instances of a shared design space and enables controlled ablations of individual components. We instantiate this framework on a string-function composition task, where models must predict the output of composed string transformations of increasing depth. Our experiments yield a clear hierarchy among RL components. On the data axis, on-policy data alone is insufficient: positive-only on-policy SFT contracts response length and entropy and underperforms teacher-distilled off-policy SFT at higher composition levels. On the loss function axis, introducing negative gradients on top of on-policy training recovers a large fraction of the gap to GRPO, identifying the use of negative samples as the primary driver of compositional ability. Adding a group-mean baseline closes nearly all of the remaining gap, while the additional standard-deviation normalization in full GRPO contributes relatively little. These findings suggest that the bulk of GRPO's advantage on compositional generalization comes from the joint use of on-policy rollouts and baseline-stabilized negative gradients, rather than from group-wise variance normalization or other objective-level details. More broadly, effective post-training of LLMs benefits from a component-level view of training: practitioners can mix and match individual components to obtain most of RL's benefits, rather than treating RL as a monolithic improvement over SFT.

39 pages

Thesis Committee:
Chenyan Xiong (Chair)
Aditi Raghunathan

Jignesh Patel, Interim Head, Computer Science Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu