We assessed the heritability of head circumference, an approximation of brain size, in twin-sib families of different ages. Data from the youngest participants were collected a few weeks after birth and from the oldest participants around age 50 years. In nearly all age groups the largest part of the variation in head circumference was explained by genetic differences. Heritability estimates were 90% in young infants (4 to 5 months), 85–88% in early childhood, 83–87% in adolescence, 75% in young and mid adulthood. In infants younger than 3 months, heritability was very low or absent. Quantitative sex differences in heritability were observed in 15- and 18-year-olds, but there was no evidence for qualitative sex differences, that is, the same genes were expressed in both males and females. Longitudinal analysis of the data between 5, 7, and 18 years of age showed high genetic stability (.78 > RG > .98). These results indicate that head circumference is a highly heritable biometric trait and a valid target for future GWA studies.