Obesity has origins extending to antenatal and early postnatal periods; however, the relationship between early postnatal diet and subsequent obesity is not well defined. The aims of this study were to determine whether early childhood dietary quality was associated with (a) infant and adolescent nutrition and (b) body mass index (BMI) in childhood and adolescence. The degree to which early nutrition and growth factors determine BMI throughout childhood and adolescence was also explored. This research was conducted using the Raine Study, a longitudinal survey of Australian children assessed from mid-gestation to 17 years of age. A dietary quality index, the Raine Eating Assessment in Toddler score, was assigned to 2562 participants to assess early nutrition. Linear regression determined that breastfeeding was associated with dietary quality at 1–3 years. Dietary elements at 14 years of age were related to earlier dietary quality. There were no consistent associations between early diet and BMI at 3, 5, 8, 10, 14 or 17 years. In contrast, birth weight and infant weight gain were significantly associated with BMI at these ages. This study suggests that early dietary patterns are associated with aspects of diet in adolescence, likely reflecting the influence of maternal reporting. Birth weight and early growth appear to be more important determinants of adolescent BMI than early diet and nutrition. While optimizing early diet by maternal nutritional education has potential to influence later nutrition, interventions focussing on early weight gain may have a greater impact on the obesity epidemic.