Social withdrawal and social anxiety are believed to have a bidirectional influence on one another, but it is unknown if their relationship is bidirectional, especially within person, and if peer experiences influence this relationship. We investigated temporal sequencing and the strength of effects between social withdrawal and social anxiety, and the roles of peer victimization and acceptance in the pathways. Participants were 2,772 adolescents from the population-based and clinically referred cohorts of the Tracking Adolescents' Individual Lives Survey. Self- and parent-reported withdrawal, and self-reported social anxiety, peer victimization, and perceived peer acceptance were assessed at 11, 13, and 16 years. Random-intercept cross-lagged panel models were used to investigate within-person associations between these variables. There was no feedback loop between withdrawal and social anxiety. Social withdrawal did not predict social anxiety at any age. Social anxiety at 11 years predicted increased self-reported withdrawal at 13 years. Negative peer experiences predicted increased self- and parent-reported withdrawal at 13 years and increased parent-reported withdrawal at 16 years. In turn, self-reported withdrawal at 13 years predicted negative peer experiences at 16 years. In conclusion, adolescents became more withdrawn when they became more socially anxious or experienced greater peer problems, and increasing withdrawal predicted greater victimization and lower acceptance.