Underrepresented minorities have higher attrition from the professoriate and have experienced greater negative impacts of the COVID-19 pandemic. The purpose of this study was to compare the impact of COVID-19 on the lives of 196 early-career physician-scientists versus PhD researchers who are underrepresented in biomedical research. Participants in the Building Up study answered questions on the impact of the COVID-19 pandemic on their personal and professional lives, and a mixed-methods approach was used to conduct the analysis. While most participants experienced increases in overall stress (72% of PhD researchers vs 76% of physician-scientists), physician-scientists reported that increased clinical demands, research delays, and the potential to expose family members to SARS-CoV-2 caused psychological distress, specifically. PhD researchers, more than physician-scientists, reported increased productivity (27% vs 9%), schedule flexibilities (49% vs 25%), and more quality time with friends and family (40% vs 24%). Future studies should consider assessing the effectiveness of programs addressing COVID-19-related challenges experienced by PhD researchers and physician-scientists, particularly those from underrepresented backgrounds.