This study examines the relationship between paternal height or body mass index (BMI) and birth weight of their offspring in a Japanese general population. The sample included 33,448 pregnant Japanese women and used fixed data, including maternal, paternal and infant characteristics, from the Japan Environment and Children’s Study (JECS), an ongoing nationwide birth cohort study. Relationships between paternal height or BMI and infant birth weight [i.e., small for gestational age (SGA) and large for gestational age (LGA)] were examined using a multinomial logistic regression model. Since fetal programming may be a sex-specific process, male and female infants were analyzed separately. Multivariate analysis showed that the higher the paternal height, the higher the odds of LGA and the lower the odds of SGA in both male and female infants. The effects of paternal BMI on the odds of both SGA and LGA in male infants were similar to those of paternal height; however, paternal height had a stronger impact than BMI on the odds of male LGA. In addition, paternal BMI showed no association with the odds of SGA and only a weak association with the odds of LGA in female infants. This cohort study showed that paternal height was associated with birth weight of their offspring and had stronger effects than paternal BMI, suggesting that the impact of paternal height on infant birth weight could be explained by genetic factors. The sex-dependent effect of paternal BMI on infant birth weight may be due to epigenetic effects.