While anecdotal evidence suggests that music may facilitate verbal memory, empirical evidence for this is less clear. Here, we examined whether learners’ characteristics such as age, working memory (WM), and musical training may influence the effect of music on word learning. Young and older adults learned novel word-referent mappings presented in three music conditions (spoken in the presence of background music, sung in-key, and sung out-of-key) and a control condition (spoken in quiet) and their performance was assessed immediately after learning. We found that whereas age and, to an extent, musical training had a general effect on word learning, WM modulated the effect of music: performance was worse in the music conditions relative to the control condition for learners with lower WM whereas the opposite pattern was observed for those with higher WM. Our results thus highlight the importance of considering individual characteristics in determining the effect of music on verbal memory.