Labeled categories are learned faster, and are subsequently more robust than categories learned without labels. The label feedback hypothesis (Lupyan, 2012) accounts for these effects by introducing a word-driven top-down modulation of perceptual processes involved in categorization. By testing categorization flexibility with and without labels, we demonstrate the ways in which labels do and do not modulate category representations. In Experiment 1, transfer involved a change in selective attention, and results indicated that labels did not impact relearning. In Experiment 2, when transfer involved a change in the behavioral response to categories whose structures did not change, a reversal shift, learning the categories with labels speeded recovery. We take this finding as evidence that the augmentation of perceptual processes by words is on the one hand fairly weak without explicit reinforcement, but on the other allows for category representations to be more abstract, allowing greater flexibility in behavior.