The relation between testosterone levels and aggressive behavior is well established. From an evolutionary viewpoint, testosterone can explain at least part of the sex differences found in aggressive behavior. This explanation, however, is mediated by factors such as prenatal testosterone levels and basal levels of cortisol. Especially regarding sex differences in aggression during adolescence, these mediators have great influence. Based on developmental brain structure research we argue that sex differences in aggression have a pre-pubertal origin and are maintained during adolescence. Evidence of prenatal, adolescent, and adult levels of testosterone in relation to aggression taken together, support Archer's argument for sexual selection as the driver of sex differences in aggression.