Many adolescents start using tobacco, alcohol, and cannabis. Genetic vulnerability, parent characteristics in young adolescence, and interaction (GxE) and correlation (rGE) between these factors could contribute to the development of substance use. Using prospective data from the TRacking Adolescent Individuals’ Lives Survey (TRAILS; N = 1,645), we model latent parent characteristics in young adolescence to predict young adult substance use. Polygenic scores (PGS) are created based on genome-wide association studies (GWAS) for smoking, alcohol use, and cannabis use. Using structural equation modeling we model the direct, GxE, and rGE effects of parent factors and PGS on young adult smoking, alcohol use, and cannabis initiation. The PGS, parental involvement, parental substance use, and parent–child relationship quality predicted smoking. There was GxE such that the PGS amplified the effect of parental substance use on smoking. There was rGE between all parent factors and the smoking PGS. Alcohol use was not predicted by genetic or parent factors, nor by interplay. Cannabis initiation was predicted by the PGS and parental substance use, but there was no GxE or rGE. Genetic risk and parent factors are important predictors of substance use and show GxE and rGE in smoking. These findings can act as a starting point for identifying people at risk.