No CrossRef data available.
Published online by Cambridge University Press: 19 July 2023
Machine learning (ML) approaches are a promising venue for identifying vocal markers of neuropsychiatric disorders, such as schizophrenia. While recent studies have shown that voice-based ML models can reliably predict diagnosis and clinical symptoms of schizophrenia, it is unclear to what extent such ML markers generalize to new speech samples collected using a different task or in a different language: the assessment of generalization performance is however crucial for testing their clinical applicability.
In this research, we systematically assessed the generalizability of ML models across contexts and languages relying on a large cross-linguistic dataset of audio recordings of patients with schizophrenia and controls.
We trained ML models of vocal markers of schizophrenia on a large cross-linguistic dataset of audio recordings of 231 patients with schizophrenia and 238 matched controls (>4.000 recordings in Danish, German, Mandarin and Japanese). We developed a rigorous pipeline to minimize overfitting, including cross-validated training set and Mixture of Experts (MoE) models. We tested the generalizability of the ML models on: (i) different participants, speaking the same language (hold-out test set); (ii) different participants, speaking a different language. Finally, we compared the predictive performance of: (i) models trained on a single language (e.g., Danish) (ii) MoE models, i.e., ensemble of models (experts) trained on a single language whose predictions are combined using a weighted sum (iii) multi-language models trained on multiple languages (e.g., Danish and German).
Model performance was comparable to state-of-the art findings (F1: 70%-80%) when trained and tested on participants speaking the same language (out-of-sample performance). Crucially, however, the ML models did not generalize well - showing a substantial decrease of performance (close to chance) - when trained in a language and tested on new languages (e.g., trained on Danish and tested on German). MoE and multi-language models showed a better increase of performance (F1: 55%-60%), but still far from those requested for achieving clinical applicability.
Our results show that the cross-linguistic generalizability of ML models of vocal markers of schizophrenia is very limited. This is an issue if our first goal is to translate these vocal markers into effective clinical applications. We argue that more emphasis needs to be placed on collecting large open datasets to test the generalizability of voice-based ML models, for example, across different speech tasks or across the heterogeneous clinical profiles that characterize schizophrenia spectrum disorder.
None Declared
To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about saving to your Kindle.
Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.
Find out more about the Kindle Personal Document Service.
To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.
To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.
Comments
No Comments have been published for this article.