The current study examined a bifactor model of affective dimensions of withdrawal. Specifically, a model which specified a general factor of anxious-avoidant withdrawal (i.e., withdrawal with negative affect), a specific factor of unsociability (i.e., withdrawal without negative affect), and a specific factor of negative affect without withdrawal was specified in the primary sample (n = 238, 56.3% boys, M age = 44.92 months, SD = 5.32 months) and a validation sample (n = 332, 52.6% boys, M age = 47.11 months, SD = 7.32 months). The model provided a good fit to the data in both samples. In the primary sample, longitudinal relations between the bifactor model and peer victimization were examined across three time points (Time 1 in the spring, Time 2 in the fall, and Time 3 in the spring). Results showed that negative affect without withdrawal was concurrently associated with higher levels of relational and physical victimization at T1, unsociability predicted reductions in relational victimization from T1 to T2 as children entered a new classroom, and anxious-avoidant withdrawal predicted reductions in relational and physical victimization from T2 to T3 as children acclimated to the new classroom. Developmental considerations and clinical implications are discussed.