Physical activity has been seen as an important tool to prevent both physical and mental disorders like depression and anxiety. However, previous research has mainly focused on mental health problems rather than positive mental health. The aim of the present study was to investigate the association of leisure time physical activity (LTPA) and physical fitness with mental distress and positive mental health. The study sample consisted of 792 men (mean age 26 years) who participated in the study prior their military refresher-training course. Cardio-respiratory and muscle fitness tests were measured, and LTPA, positive mental health (Warwick–Edinburgh Mental Wellbeing Scale, SWEMWBS) and mental distress were assessed with a questionnaire. Analysis of variance and linear regressions were used as statistical methods. Younger age and lower military education were associated with mental distress, whereas older age and higher military education were inversely associated with high positive mental health (flourishing). Moreover, individuals reporting vigorous LTPA at least 1–2 times per week had lower level of mental distress and were more commonly experiencing flourishing compared to their less physically active counterparts. Physical fitness (both aerobic and muscular) was associated with lower mental distress, but not with flourishing. In conclusion, leisure time physical activity and physical fitness may contribute to mental health. The present results highlight that even low amount of leisure-time physical activity may promote mental health and prevent mental disorders. Leisure time physical activity, rather than physical fitness, seems to be more essential for positive mental health.