There is strong evidence that physical activity (PA) has an influence on physical performance in later life. Also, a small body size at birth has been associated with lower physical functioning in older age and both small and high birth weight have shown to be associated with lower leisure time physical activity. However, it is unknown whether size at birth modulates the association between PA and physical performance in old age. We examined 695 individuals from the Helsinki Birth Cohort Study born in Helsinki, Finland between 1934 and 1944. At a mean age of 70.7 years PA was objectively assessed with a multisensory activity monitor and physical performance with the Senior Fitness Test (SFT). Information on birth weight and gestational age was retrieved from hospital birth records. The study participants were divided in three birth weight groups, that is <3000 g, 3000–3499 g and ⩾3500 g. The volume of PA was significantly associated with the physical performance in all birth weight groups. However, the effect size of the association was large and significant only in men with a birth weight <3000 g (β 0.59; 95% confidence interval 0.37–0.81, P<0.001). Our study shows that the association between PA and physical performance is largest in men with low birth weight. Our results suggest that men with low birth weight might benefit most from engaging in PA in order to maintain a better physical performance.