Psychological distress (PSYCH), somatic distress (SOMA), affective disorders (AD), and substance use (SU) frequently co-occur. The genetic relationship between PSYCH and SOMA, however, remains understudied. We examined the genetic and environmental influences on these two disorders and their comorbid AD and SU using structural equation modeling. Self-reported PSYCH and SOMA were measured in 1,548 twins using the two subscales of a 12-item questionnaire, the Somatic and Psychological Health Report. Its reliability and psychometric properties were examined. Six ADs, involvement of licit and illicit substance, and two SU disorders were obtained from 1,663–2,132 twins using the World Mental Health Composite International Diagnostic Interview and/or from an online adaption of the same. SU phenotypes (heritability: 49–79%) were found to be more heritable than the affective disorder phenotypes (heritability: 32–42%), SOMA (heritability: 25%), and PSYCH (heritability: 23%). We fit separate non-parametric item response theory models for PSYCH, SOMA, AD, and SU. The IRT scores were used as the refined phenotypes for fitting multivariate genetic models. The best-fitting model showed the similar amount of genetic overlap between PSYCH–AD (genetic correlation rG = 0.49) and SOMA–AD (rG =0.53), as well as between PSYCH–SU (rG = 0.23) and SOMA–SU (rG = 0.25). Unique environmental factors explained 53% to 76% of the variance in each of these four phenotypes, whereas additive genetic factors explained 17% to 46% of the variance. The covariance between the four phenotypes was largely explained by unique environmental factors. Common genetic factor had a significant influence on all the four phenotypes, but they explained a moderate portion of the covariance.