Lower birth weight is associated with a risk of type 2 diabetes in adulthood. However, it is not clear whether this association is modified by physical activity. This study aimed to examine the association between birth weight and glycemic status and whether this association is mediated by moderate- and vigorous-intensity activity (MVPA) in Japanese women. The participants were 103 nondiabetic women (47.4 ± 10.8 years), who underwent health checkups in which data of glycated hemoglobin (HbA1c) were collected. Abdominal circumference (AC) was measured at the umbilical region. Birth weight was obtained from the Maternal and Child Health Handbook records or reported based on the participant’s or his/her mother’s recall. Time (min/day) spent in MVPA (≥3.0 metabolic equivalents) was objectively measured using a triaxial accelerometer (Actimarker EW4800). Birth weight was inversely correlated with HbA1c (r = −0.32, P < 0.01). Multiple linear regression analyses revealed that lower birth weight was associated with increased HbA1c (β = −0.22, P < 0.05) even after adjusting for age, state of menstruation, AC, and family history of diabetes. This association was little changed when MVPA was introduced as an independent variable in the model (β = −0.23, P < 0.05). These results suggest that lower birth weight may be associated with higher HbA1c levels before the onset of type 2 diabetes, irrespective of adulthood physical activity. Early-life development should be taken into account when considering the risk of diabetes in Japanese women, even if they are physically active.