Hostname: page-component-76fb5796d-dfsvx Total loading time: 0 Render date: 2024-04-28T11:44:05.942Z Has data issue: false hasContentIssue false
  • English
  • Français

A computational neuroimaging study of reinforcement learning and goal-directed exploration in schizophrenia spectrum disorders

Published online by Cambridge University Press:  08 February 2023

A. J. Culbreth ,
E. K. Schwartz ,
M. J. Frank ,
E. C. Brown ,
Z. Xu ,
S. Chen ,
J. M. Gold  and
J. A. Waltz Open the ORCID record for J. A. Waltz [Opens in a new window]
A. J. Culbreth
Affiliation:
Department of Psychiatry, Maryland Psychiatric Research Center (MPRC), University of Maryland School of Medicine, Baltimore, MD, USA
E. K. Schwartz
Affiliation:
Signant Health, San Diego, CA, USA
M. J. Frank
Affiliation:
Department of Cognitive, Linguistic and Psychological Sciences, Brown University, Providence, RI, USA Department of Psychiatry and Brown Institute for Brain Science, Brown University, Providence, RI, USA
E. C. Brown
Affiliation:
School of Health and Care Management, Arden University, Berlin, Germany
Z. Xu
Affiliation:
Applied LifeSciences & Systems, Morrisville, NC, USA
S. Chen
Affiliation:
Department of Psychiatry, Maryland Psychiatric Research Center (MPRC), University of Maryland School of Medicine, Baltimore, MD, USA Division of Biostatistics and Bioinformatics, Department of Epidemiology and Public Health, University of Maryland School of Medicine, Baltimore, MD, USA
J. M. Gold
Affiliation:
Department of Psychiatry, Maryland Psychiatric Research Center (MPRC), University of Maryland School of Medicine, Baltimore, MD, USA
J. A. Waltz*
Affiliation:
Department of Psychiatry, Maryland Psychiatric Research Center (MPRC), University of Maryland School of Medicine, Baltimore, MD, USA
*
Author for correspondence: J. A. Waltz, E-mail: jwaltz@som.umaryland.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Background

Prior evidence indicates that negative symptom severity and cognitive deficits, in people with schizophrenia (PSZ), relate to measures of reward-seeking and loss-avoidance behavior (implicating the ventral striatum/VS), as well as uncertainty-driven exploration (reliant on rostrolateral prefrontal cortex/rlPFC). While neural correlates of reward-seeking and loss-avoidance have been examined in PSZ, neural correlates of uncertainty-driven exploration have not. Understanding neural correlates of uncertainty-driven exploration is an important next step that could reveal insights to how this mechanism of cognitive and negative symptoms manifest at a neural level.

Methods

We acquired fMRI data from 29 PSZ and 36 controls performing the Temporal Utility Integration decision-making task. Computational analyses estimated parameters corresponding to learning rates for both positive and negative reward prediction errors (RPEs) and the degree to which participates relied on representations of relative uncertainty. Trial-wise estimates of expected value, certainty, and RPEs were generated to model fMRI data.

Results

Behaviorally, PSZ demonstrated reduced reward-seeking behavior compared to controls, and negative symptoms were positively correlated with loss-avoidance behavior. This finding of a bias toward loss avoidance learning in PSZ is consistent with previous work. Surprisingly, neither behavioral measures of exploration nor neural correlates of uncertainty in the rlPFC differed significantly between groups. However, we showed that trial-wise estimates of relative uncertainty in the rlPFC distinguished participants who engaged in exploratory behavior from those who did not. rlPFC activation was positively associated with intellectual function.

Conclusions

These results further elucidate the nature of reinforcement learning and decision-making in PSZ and healthy volunteers.

Keywords

Dopaminemotivationpsychosisreward prediction errorrostrolateral prefrontal cortexventral striatum
Type
Original Article
Information
Psychological Medicine , Volume 53 , Issue 14 , October 2023 , pp. 6600 - 6610
Check for updates
Copyright
Copyright © The Author(s), 2023. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Addington, D., Addington, J., Maticka-Tyndale, E., & Joyce, J. (1992). Reliability and validity of a depression rating scale for schizophrenics. Schizophrenia Research, 6(3), 201–208. https://doi.org/10.1016/0920-9964(92)90003-N.CrossRefGoogle ScholarPubMed
Andreasen, N. C. (1989). The Scale for the Assessment of Negative Symptoms (SANS): conceptual and theoretical foundations. The British journal of psychiatry, 155(S7), 49–52.Google Scholar
Andreasen, N. C., Pressler, M., Nopoulos, P., Miller, D., & Ho, B. C. (2010). Antipsychotic dose equivalents and dose-years: A standardized method for comparing exposure to different drugs. Biological Psychiatry, 67(3), 255–262. https://doi.org/10.1016/j.biopsych.2009.08.040.CrossRefGoogle ScholarPubMed
Badre, D., Doll, B. B., Long, N. M., & Frank, M. J. (2012). Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration. Neuron, 73(3), 595–607. https://doi.org/10.1016/J.NEURON.2011.12.025.CrossRefGoogle ScholarPubMed
Cavanagh, J. F., Figueroa, C. M., Cohen, M. X., & Frank, M. J. (2012). Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation. Cerebral Cortex, 22(11), 2575–2586. https://doi.org/10.1093/cercor/bhr332.CrossRefGoogle ScholarPubMed
Clark, L., Cools, R., & Robbins, T. W. (2004). The neuropsychology of ventral prefrontal cortex: Decision-making and reversal learning. Brain and Cognition, 55(1), 41–53. https://doi.org/10.1016/S0278-2626(03)00284-7.CrossRefGoogle ScholarPubMed
Culbreth, A. J., Westbrook, A., Daw, N. D., Botvinick, M., & Barch, D. M. (2016a). Reduced model-based decision-making in schizophrenia. Journal of Abnormal Psychology, 125(6), 777–787. https://doi.org/10.1037/abn0000164.CrossRefGoogle ScholarPubMed
Culbreth, A. J., Westbrook, A., Xu, Z., Barch, D. M., & Waltz, J. A. (2016b). Intact ventral striatal prediction error signaling in medicated schizophrenia patients. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 1(5), 474–483. https://doi.org/10.1016/j.bpsc.2016.07.007.Google ScholarPubMed
Dowd, E. C., Frank, M. J., Collins, A., Gold, J. M., & Barch, D. M. (2016). Probabilistic reinforcement learning in patients with schizophrenia: Relationships to anhedonia and avolition. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 1(5), 460–473. https://doi.org/10.1016/J.BPSC.2016.05.005.Google ScholarPubMed
First, M. B., & Gibbon, M., (2004). The Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I) and the Structured Clinical Interview for DSM-IV Axis II Disorders (SCID-II). In M. J. Hilsenroth & D. L. Segal (Eds.), Comprehensive handbook of psychological assessment, Vol. 2. Personality assessment (pp. 134–143). Hoboken, New Jersey: John Wiley & Sons, Inc.Google Scholar
Frank, M. J., Doll, B. B., Oas-Terpstra, J., & Moreno, F. (2009). Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nature Neuroscience, 12(8), 1062–1068. https://doi.org/10.1038/nn.2342.CrossRefGoogle ScholarPubMed
Gershman, S. J. (2019). Uncertainty and exploration. Decision, 6(3), 277–286. https://doi.org/10.1037/dec0000101.CrossRefGoogle ScholarPubMed
Gold, J. M., Waltz, J. A., Matveeva, T. M., Kasanova, Z., Strauss, G. P., Herbener, E. S., … Frank, M. J. (2012). Negative symptoms and the failure to represent the expected reward value of actions: Behavioral and computational modeling evidence. Archives of General Psychiatry, 69(2), 129–138. https://doi.org/10.1001/archgenpsychiatry.2011.1269.CrossRefGoogle ScholarPubMed
Gold, J. M., Waltz, J. A., Prentice, K. J., Morris, S. E., & Heerey, E. A. (2008). Reward processing in schizophrenia: A deficit in the representation of value. Schizophrenia Bulletin, 34(5), 835–847. https://doi.org/10.1093/schbul/sbn068.CrossRefGoogle ScholarPubMed
Kring, A. M., & Barch, D. M. (2014). The motivation and pleasure dimension of negative symptoms: Neural substrates and behavioral outputs. European Neuropsychopharmacology, 24(5), 725–736. https://doi.org/10.1016/j.euroneuro.2013.06.007.CrossRefGoogle ScholarPubMed
McClure, S. M., Berns, G. S., & Montague, P. R. (2003). Temporal prediction errors in a passive learning task activate human striatum. Neuron, 38(2), 339–346. https://doi.org/10.1016/S0896-6273(03)00154-5.CrossRefGoogle Scholar
Moustafa, A. A., Cohen, M. X., Sherman, S. J., & Frank, M. J. (2008). A role for dopamine in temporal decision making and reward maximization in Parkinsonism. Journal of Neuroscience, 28(47), 12294–12304. https://doi.org/10.1523/JNEUROSCI.3116-08.2008.CrossRefGoogle ScholarPubMed
Overall, J. E., & Gorham, D. R. (1962). The brief psychiatric rating scale. Psychological Reports, 10(3), 799–812. https://doi.org/10.2466/pr0.1962.10.3.799.CrossRefGoogle Scholar
Payzan-Lenestour, E., & Bossaerts, P. (2011). Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Computational Biology, 7(1), e1001048. https://doi.org/10.1371/journal.pcbi.1001048.CrossRefGoogle ScholarPubMed
Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith, C. D. (2006). Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature, 442(7106), 1042–1045. https://doi.org/10.1038/nature05051.CrossRefGoogle ScholarPubMed
Radua, J., Schmidt, A., Borgwardt, S., Heinz, A., Schlagenhauf, F., McGuire, P., & Fusar-Poli, P. (2015). Ventral striatal activation during reward processing in psychosis a neurofunctional meta-analysis. JAMA Psychiatry, 72(12), 1243–1251. https://doi.org/10.1001/jamapsychiatry.2015.2196.CrossRefGoogle ScholarPubMed
Schlagenhauf, F., Huys, Q. J. M., Deserno, L., Rapp, M. A., Beck, A., Heinze, H.-J., … Heinz, A. (2014). Striatal dysfunction during reversal learning in unmedicated schizophrenia patients. NeuroImage, 89, 171–180. https://doi.org/10.1016/J.NEUROIMAGE.2013.11.034.CrossRefGoogle ScholarPubMed
Strauss, G. P., Frank, M. J., Waltz, J. A., Kasanova, Z., Herbener, E. S., & Gold, J. M. (2011). Deficits in positive reinforcement learning and uncertainty-driven exploration are associated with distinct aspects of negative symptoms in schizophrenia. Biological Psychiatry, 69(5), 424–431. https://doi.org/10.1016/J.BIOPSYCH.2010.10.015.CrossRefGoogle ScholarPubMed
Waltz, J. A., Frank, M. J., Wiecki, T. V., & Gold, J. M. (2011). Altered probabilistic learning and response biases in schizophrenia: Behavioral evidence and neurocomputational modeling. Neuropsychology, 25(1), 86–97. https://doi.org/10.1037/a0020882.CrossRefGoogle ScholarPubMed
Waltz, J. A., & Gold, J. M. (2007). Probabilistic reversal learning impairments in schizophrenia: Further evidence of orbitofrontal dysfunction. Schizophrenia Research, 93(1–3), 296–303. https://doi.org/10.1016/J.SCHRES.2007.03.010.CrossRefGoogle ScholarPubMed
Waltz, J. A., Wilson, R. C., Albrecht, M. A., Frank, M. J., & Gold, J. M. (2020). Differential effects of psychotic illness on directed and random exploration. Computational Psychiatry, 4(0), 18. https://doi.org/10.1162/cpsy_a_00027.CrossRefGoogle ScholarPubMed
Weschler, D. (2001). Wechsler Test of Adult Reading (WTAR). The Psychological Corporation.Google Scholar
Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A., & Cohen, J. D. (2014). Humans use directed and random exploration to solve the explore-exploit dilemma. Journal of Experimental Psychology: General, 143(6), 2074–2081. https://doi.org/10.1037/a0038199.CrossRefGoogle ScholarPubMed
Young, R. C., Biggs, J. T., Ziegler, V. E., & Meyer, D. A. (1978). A rating scale for mania: Reliability, validity and sensitivity. British Journal of Psychiatry, 133(11), 429–435. https://doi.org/10.1192/bjp.133.5.429.CrossRefGoogle ScholarPubMed
Zajkowski, W. K., Kossut, M., & Wilson, R. C. (2017). A causal role for right frontopolar cortex in directed, but not random, exploration. ELife, 6. p.e27430. https://doi.org/10.7554/eLife.27430.CrossRefGoogle Scholar
Supplementary material: File

Culbreth et al. supplementary material

Culbreth et al. supplementary material

Download Culbreth et al. supplementary material(File)
File 1.2 MB