Although dried orange pulp (DOP) may conveniently replace cereals in ruminant diets, few studies have considered similar diet substitution for goats. We hypothesised that DOP could replace cereal-based concentrate in goat diets without detrimental effects on growth performance and carcass quality of suckling kids and milk performance and blood biochemical parameters of dams in early lactation. We also hypothesised that DOP substitution may increase the levels of antioxidants, such as phenolic compounds and vitamin E, in milk and improve its total antioxidant capacity (TAC). Therefore, 44 primiparous Payoya dairy goats were allocated to three experimental groups, each fed a different diet: control (CD, n = 14) based on a commercial concentrate with alfalfa hay as forage; and DOP40 (n = 16) in which 40% and DOP80 (n = 14) in which 80% of the cereal in the concentrate were replaced by DOP. The experiment lasted from the final month of pregnancy to 55 days postpartum. The DOP diets did not affect suckling kids’ carcass quality, but at 28 days, led to improvement in live weight (LW) and average daily gain (ADG) from birth, although no differences were found between DOP40 and DOP80 (for CD, DOP40 and DOP80, LW at 28 days was 8.00, 8.58 and 8.34 kg and ADG was 184, 199 and 195 g/day, respectively). Diet had no significant effect on milk yield (average daily milk yield and total yield at 55 days were 1.66 l/day and 90.6 l, respectively) and commercial and fatty acid composition. Nevertheless, α-tocopherol, total phenolic compound (TPC) and TAC concentration in milk increased with substitution of cereals by DOP (for CD, DOP40 and DOP80, concentration of α-tocopherol was 21.7, 32.8 and 42.3 μg/100 g, TPCs was 63.5, 84.1 and 102 mg gallic acid equivalents/l, and TAC was 6.63, 11.1 and 12.8 μmol Trolox equivalents/ml, respectively). Every plasma biochemistry parameter considered was within reference values for healthy goats; therefore, no pathological effect was detected for these variables due to dietary treatment. However, DOP diets caused a reduction in plasmatic creatine kinase and aspartate aminotransferase, implying reduced oxidative damage to muscles. In conclusion, DOP may be an interesting alternative to cereals in early lactation goat diets for increasing farmers’ income and the healthy antioxidant capacity of milk.