Physical activity is a known protective factor, with benefits for both metabolic and psychological aspects of health. Our objective was to verify early and late determinants of physical activity in young adults. A total of 2063 individuals from a birth cohort in Ribeirão Preto, Brazil, were studied at the age of 23–25 years. Poisson regression was performed using three models: (1) early model considering birth weight, gestational age, maternal income, schooling and smoking; (2) late model considering individual’s gender, schooling, smoking and body mass index; and (3) combined (early + late) model. Physical activity was evaluated using the International Physical Activity Questionnaire, stratifying the individuals into active or sedentary. The general rate of sedentary behavior in the sample was 49.6%. In the early model, low birth weight (relative risk (RR) = 1.186, confidence interval (95%CI) 1.005–1.399) was a risk factor for sedentary activity. Female gender (RR = 1.379, 95%CI = 1.259–1.511) and poor schooling (RR = 1.126, 95%CI = 1.007–1.259) were associated with sedentary behavior in the late model. In the combined model, only female gender and participant’s schooling remained significant. An interaction between birth weight and individual’s schooling was found, in which sedentary behavior was more prevalent in individuals born with low birth weight only if they had higher educational levels. Variables of early development and social insertion in later life interact to determine an individual’s disposition to practice physical activities. This study may support the theoretical model ‘Similarities in the inequalities’, in which opposed perinatal backgrounds have the same impact over a health outcome in adulthood when facing unequal social achievement during the life-course.