Despite high altitude was implicated in adverse birth outcomes, there remained a paucity of evidence on low-to-medium altitude effect. This study aimed to explore the association of low-to-medium altitude with birth outcomes. A population-based cross-sectional survey was performed using a stratified multistage random sampling method among women with their infants born during 2010–2013 in Northwestern China. Altitude was determined in meters based on the village or community of the mother’s living areas. Birth outcomes involved birth weight, gestational age, and small for gestational age (SGA). Generalized linear models were fitted to investigate the association of altitude with birth outcomes. Moreover, the dose–response relationship between altitude and birth outcomes was evaluated with a restricted cubic spline function. A total of 27 801 women with their infants were included. After adjusting for potential confounders, every 100-m increase in the altitude was associated with reduced birth weight by 6.4 (95% CI −8.1, −4.6) g, the slight increase of gestational age by 0.015 (95% CI 0.010, 0.020) week, and an increased risk of SGA birth (odds ratio 1.03, 95% CI 1.02, 1.04). Moreover, there was an inversely linear relationship between altitude and birth weight (P for overall < 0.001 and P for nonlinear = 0.312), and a positive linear relationship between altitude and SGA (P for overall < 0.001 and P for nonlinear = 0.194). However, a nonlinear relationship was observed between altitude and gestational age (P for overall < 0.001 and P for nonlinear = 0.010). The present results suggest that low-to-medium altitude is possibly associated with adverse birth outcomes.