Hostname: page-component-77c89778f8-swr86 Total loading time: 0 Render date: 2024-07-21T17:55:37.115Z Has data issue: false hasContentIssue false

The influence of caregiver attitudes and socioeconomic group on formal and informal mental health service use among youth

Published online by Cambridge University Press:  10 June 2022

Cristiane Silvestre Paula*
Affiliation:
Programa de Pós-Graduação em Distúrbios do Desenvolvimento e Centro Mackenzie de Pesquisa sobre a Infância e Adolescência, Universidade Presbiteriana Mackenzie (UPM), Rua da Consolação 930, Edifício 28, São Paulo, Brazil
Carolina Ziebold
Affiliation:
Departamento de Psiquiatria, Universidade Federal de São Paulo (UNIFESP), São Paulo, Brazil
Wagner S. Ribeiro
Affiliation:
Care Policy and Evaluation Centre, London School of Economics and Political Science (LSE), London, United Kingdom
Pedro Mario Pan
Affiliation:
Laboratório de Neurociências Integrativas (LiNC), Departamento de Psiquiatria, Universidade Federal de São Paulo (UNIFESP), São Paulo, Brazil
Jair Jesus Mari
Affiliation:
Departamento de Psiquiatria, Universidade Federal de São Paulo (UNIFESP), São Paulo, Brazil
Rodrigo Bressan
Affiliation:
Laboratório de Neurociências Integrativas (LiNC), Departamento de Psiquiatria, Universidade Federal de São Paulo (UNIFESP), São Paulo, Brazil Instituto Ame Sua Mente, São Paulo, Brazil
Euripedes Constantino Miguel
Affiliation:
Departamento de Psiquiatria, Faculdade de Medicina da Universidade de São Paulo (FMUSP), São Paulo, Brazil
Luiz Augusto Rohde
Affiliation:
ADHD Outpatient Program & Developmental Psychiatry Program, Hospital de Clínicas de Porto Alegre, Federal University of Rio Grande do Sul, Porto Alegre, Brazil National Institute of Developmental Psychiatry, São Paulo, Brazil
Giovanni A. Salum
Affiliation:
National Institute of Developmental Psychiatry, São Paulo, Brazil Section on Negative Affect and Social Processes, Hospital de Clínicas de Porto Alegre, Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil
Sara Evans-Lacko
Affiliation:
Care Policy and Evaluation Centre, London School of Economics and Political Science (LSE), London, United Kingdom
*
* Author for correspondence: Cristiane Silvestre Paula, E-mail: cristiane.paula@mackenzie.br; csilvestrep09@gmail.com

Abstract

Background

Young people can receive mental health care from many sources, from formal and informal sectors. Caregiver characteristics/experiences/beliefs may influence whether young people get help and the type of care or support used by their child. We investigate facilitators/barriers to receiving formal and/or informal care, particularly those related to the caregiver’s profile.

Methods

We interviewed 1,400 Brazilian primary caregivers of young people (aged 10–19), participants of a high-risk cohort. Caregivers reported on young people’s formal/informal mental health care utilization, and associated barriers and facilitators to care. Data were also collected on youth mental health and its impact on everyday life; and caregiver characteristics—education, socioeconomics, ethnicity, mental health, and stigma. Logistic regression models were used to examine the relationship between caregiver and young people characteristics with formal/informal care utilization.

Results

Persistence and greater impact of youth mental health conditions were associated with a higher likelihood of care, more clearly for formal care. Caregiver characteristics, however, also played a key role in whether young people received any care: lower parental stigma was associated with greater formal service use, and lower socioeconomic class showed higher odds of informal care (mainly from religious leaders).

Conclusions

This study highlights the key role of the caregivers as gatekeepers to child treatment access, particularly parental stigma influencing whether young people received any mental health care, even in a low resource setting. These results help to map barriers for treatment access and delivery for young people, aiming to improve intervention efforts and mental health support.

Type
Research Article
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Copyright
© The Author(s), 2022. Published by Cambridge University Press on behalf of the European Psychiatric Association

Introduction

Mental health problems in young people are prevalent, and have a significant impact on their lives [Reference Polanczyk, Salum, Sugaya, Caye and Rohde1], but often they do not receive appropriate care [Reference Kieling, Baker-Henningham, Belfer, Conti, Ertem and Omigbodun2]. Support and treatment for such problems can arise from formal and informal sources of care such as religious leaders, non-health professionals, and self-help [Reference Kessler, Soukup, Davis, Foster, Wilkey and Van Rompay3]. Support often involves multiple, ideally coordinated, sources of care from health, education, and third-sector organizations [Reference MacDonald, Fainman-Adelman, Anderson and Iyer4]. Most studies investigating access to mental health care among youth, however, use clinical samples and focus on barriers to any service contacts rather than considering the context, type, and appropriateness or how the care meets the needs of individuals. Moreover, the range of sources of informal care makes systematic tracking of utilization difficult, and the types of support provided may not be recognized as care [Reference Brown, Evans-Lacko, Aschan, Henderson, Hatch and Hotopf5].

The receipt of adequate mental health by young people depends on a range of factors. First, youth are rarely able to seek and access care on their own; therefore, caregivers often represent key gatekeepers to care. Caregiver characteristics (e.g., social class and their own psychopathology) and experiences related to mental health (e.g., their knowledge, beliefs, and potential prejudice/stigma toward mental health) can influence their children’s pathway to care [Reference Ryan, Jorm, Toumbourou and Lubman6, Reference Gronholm, Ford, Roberts, Thornicroft, Laurens and Evans-Lacko7]. Second, characteristics of the youth (e.g., gender, age, and psychopathology) can influence caregiver perceptions of their children’s need for care, consequently influencing help-seeking [Reference Ryan, Jorm, Toumbourou and Lubman6]. In addition to caregiver and youth characteristics, structural barriers, such as cost, distance, and limitations of the appointment system, are important factors in the care pathway [Reference Paula, Bordin, Mari, Velasque, Rohde and Coutinho8, Reference Reardon, Harvey, Baranowska, O’Brien, Smith and Creswell9].

Therefore, characterizing formal and informal sources of support and understanding factors linked to care access and ongoing treatment can help policymakers to map barriers and facilitators for treatment access and delivery to have a more detailed picture of the needs of a population aiming at improving intervention efforts maximizing mental health provision.

This study aims: (a) to describe types of formal and informal mental health-related services used by youth in Brazil; (b) to investigate facilitators and barriers to any versus no contact with formal and/or informal care, particularly those related to caregivers’ and young people characteristics; and (c) given that barriers to any contact may differ from the barriers among those receiving some type of support, we also investigated perceived barriers to care among those who had at least some service contact.

We hypothesized that low socioeconomic group (SEG), the presence of caregiver mental health problems, and caregiver stigma would be associated with a lower likelihood of formal or informal child mental health service use.

Method

Sample

This study was nested within the Brazilian High-Risk Cohort Study for Mental Conditions (BHRCS) [Reference Salum, Gadelha, Pan, Moriyama, Graeff-Martins and Tamanaha10], which is a large school-based community cohort enriched for mental health conditions. The aim of the study was to understand the developmental trajectories of psychopathology and mental disorders using a two-stage design. Families were recruited from 22 schools in Porto Alegre and 35 schools in São Paulo. First, 9,937 parents were screened using the Family History Survey (FHS) on the registry day. The FHS is used to screen all family members (in 87% of cases, the mother was the primary informant) for the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV). An index of family load was computed for each of the potential eligible children based on the percentage of members in the family that screened positively for each of the disorders assessed, adjusted for relatedness. From this pool, we selected two subgroups: one randomly selected (n = 958) and one high-risk sample (n = 1,553), a subsample of children at increased risk of mental disorders based on FHS [Reference Salum, Gadelha, Pan, Moriyama, Graeff-Martins and Tamanaha10]. Because we would expect mental health service use to be relatively rare, this enriched sample, which included a high proportion of youth at-risk of mental health problems, is ideal. Moreover, as we wanted to understand the barriers among youth with mental health conditions who were not receiving care, we needed a nonclinical sample.

At baseline (2010–2011), 2,511 young people aged 6–14 years and their caregivers were interviewed. During the first follow-up (2014–2015), 2,010 parents/main guardians responded to a comprehensive assessment of mental health and mental health conditions related to their children (80% retention rate). However, for this study about use of services, we were only able to contact 94% of those who participated at the first follow-up (n = 1,881). Of those, 1,400 (74.4%) completed a further specific interview on mental health-related service use by their child (aged 10–19) and potential barriers and facilitators to using these services, 982 (70.1%) by telephone and 418 (29.9%) face-to-face. Of those, 1,400 (74.4%) completed a further specific interview on mental health-related service use by their child (aged 10–19) and potential barriers and facilitators to using these services, 982 (70.1%) by telephone and 418 (29.9%) face-to-face. Supplementary Figure S1 shows the sampling process flowchart. We compared characteristics of all baseline Brazilian High-Risk Cohort (BHRC) participants and those who completed the health-related service use study (Supplementary Table S1). Briefly, lower maternal education, lower socioeconomic status, Asian/indigenous ethnicity, female gender, and higher age at baseline increased the likelihood of attrition at follow-up.

All caregivers provided written consent, and all youth provided verbal assent. This study was approved by the Ethics committee of the UNIFESP (CAAE 06457219.9.0000.5505) and HC-Porto Alegre (CAAE 06457219.9.3001.5327).

Measures

Youth mental health service use: The Service Assessment for Children and Adolescents-SACA [Reference Stiffman, Horwitz, Hoagwood, Compton, Cottler and Bean11, Reference Horwitz, Hoagwood, Stiffman, Summerfeld, Weisz and Costello12] was used to ask caregivers about service contacts in the previous year due to concerns regarding their children’s emotions and behavior problems, including alcohol- and drug-related problems. The SACA assesses types of mental health services used, including treatments received, reasons for service use, and its quality, including barriers and facilitators for accessing the services. It covers 30 service settings grouped in three areas: inpatient, outpatient, and school.

Overall, concordance between parental reports and records, and the test–retest reliability for 12-month service use of the SACA is moderate to substantial [Reference Horwitz, Hoagwood, Stiffman, Summerfeld, Weisz and Costello12]. We received permission from the SACA developers to translate/adapt it for use in the Brazilian context in consultation with experts in the mental health system to ensure that we covered the relevant service types and settings.

For the current study, we first considered any formal or informal service use in the past 12 months and then subgroups of formal (health or education) and informal (religious/spiritual leader, self-help, and alternative therapies) mental health care (Supplementary Table S2).

Barriers to sufficient mental health care among youth who received services: As part of the SACA, caregivers who reported at least some service contact due to mental health problems in the past 12 months were asked: “Did you think the child needed any service other than the one she/he used?” Those who answered yes were asked about the reasons why the child had not received these services. The reasons included a list of 14 potential barriers and an open-ended question (“others”). Barriers were classified as: (a) structural (three items); (b) recognition and literacy (four items); (c) lack of trust and negative experiences with services/treatment (one item); and (d) stigma (one item; Supplementary Table S2).

Trajectory of psychiatric diagnosis: Psychiatric diagnoses were assessed at baseline and follow-up using the Brazilian-Portuguese version of the Development and Well-Being Assessment [Reference Goodman, Ford, Richards, Gatward and Meltzer13, Reference Fleitlich-Bilyk and Goodman14], a structured interview used to generate DSM-IV diagnoses conducted by trained psychologists. At baseline, diagnostic assessment and interviews were performed with the caregiver only. At follow-up, diagnostic assessment considered caregiver reports and additional information from interviews with the youth about internalizing conditions. All responses were evaluated by trained psychiatrists who determined final diagnostic status (for more details, please see [Reference Salum, Gadelha, Pan, Moriyama, Graeff-Martins and Tamanaha10]). Three categories of diagnostic trajectories were created: (a) no diagnosis (no diagnosis at baseline neither at follow-up); (b) transient mental health condition (presence of diagnosis at baseline or follow-up); and (c) persistent mental health condition (presence of diagnosis at both time points).

Impact of the mental health condition was measured using the “impact supplement” of the Strengths and Difficulties Questionnaire (SDQ), which assesses the impact of behavioral and emotional difficulties on the young person’s life according to parental reports. This supplement begins with a general question about difficulties. For those with no difficulties, a score of 0 is attributed. For those with difficulties, the areas in which the difficulties occur are then explored. A total score is generated by summing five items: one item about distress, and four on social impairment in: (a) family life, (b) friendships, (c) learning, and (d) leisure activities. Higher scores represent greater impact. The impact score has demonstrated internal consistency, cross-informant correlations, and stability across time [Reference Goodman15]. Moreover, other research suggests that the impact supplement adds information above and beyond symptoms [Reference Stringaris and Goodman16].

Caregiver education (“Primary” [less than secondary education], “Secondary” [any secondary education], and “Higher” [any complete/incomplete university education]).

SEG was evaluated using the Brazilian Association of Research Companies questionnaire that classifies familial socioeconomic status according to purchasing power and head of household education [17], one of the most widely used questionnaires in Brazil, classifying families into eight SEGs, which were dichotomised into two categories: Middle-High and Middle-Low/Low.

Caregiver ethnicity was categorized into white, black, mixed (between white and black), and other (including Asian and Indigenous). Due to low numbers of Asian (n = 3) and Indigenous (n = 2) participants, these two categories were collapsed.

Caregiver mental health was assessed using the Kessler distress scale (K6) [Reference Kessler, Green, Gruber, Sampson, Bromet and Cuitan18, Reference Kessler, Andrews, Colpe, Hiripi, Mroczek and Normand19], which is a self-administered scale assessing how frequently an adult experienced symptoms of psychological distress (e.g., how often did you feel hopeless?) in the past 30 days. It comprises six items and uses a five-point Likert scale, ranging from one (never) to five (all the time), where the higher the score, the greater the stress perceived by individual. The K6 has excellent internal consistency and reliability (Cronbach’s alpha = 0.89) [Reference Kessler, Andrews, Colpe, Hiripi, Mroczek and Normand19] and it was used in the current study to screen non-specific serious mental illnesses among caregivers. Based on previous studies, caregivers who scored above 13 points on the K6 in the past 12-months were considered to have a mental health condition [Reference Kessler, Barker, Colpe, Epstein, Gfroerer and Hiripi20].

Caregiver/parental stigma was assessed using the intended behavior subscale of the Brazilian version of the Reported and Intended Behavior Scale (RIBS-BP). The RIBS-BP assesses future intended stigmatizing behavior across four contexts: living with, working with, living nearby, and continuing a relationship with someone with a mental health condition [Reference Evans-Lacko, Rose, Little, Flach, Rhydderch and Henderson21, Reference Ribeiro, Gronholm, Silvestre de Paula, Hoffmann, Rojas Vistorte and Zugman22]. Items are summed into a total score where higher scores reflect less stigma. The RIBS-BP showed good to excellent construct validity [Reference Ribeiro, Gronholm, Silvestre de Paula, Hoffmann, Rojas Vistorte and Zugman22].

Data analysis

First, we calculated the prevalence of each type of service utilization (any, formal, and informal) among youth, overall, and by trajectories of mental health condition and according to sociodemographic characteristics.

Then, we used logistic regression models to assess the association between caregiver characteristics (education, SEG, ethnicity, mental health, and parental stigma) and youth characteristics (gender, age, trajectory, and impact of the mental health condition) with utilization of formal and informal care. Unadjusted and adjusted odds ratios and 95% confidence intervals are presented.

Among those who received sufficient/insufficient mental health care, as defined above, we describe the additional types of barriers (structural, lack of recognition and illiteracy, lack of trust and negative experiences in services/treatment, anticipated stigma, and each of their subcategories), reported by the sample overall and according to trajectories of mental health condition (i.e., none, transient, and persistent). We then used logistic regression models to examine the association between factors related with reporting any additional barriers to sufficient mental health care. Due to the small sample size, we limited the number of variables included in this model to key caregiver (education, SEG, and mental health condition) and young person (gender, age, trajectory, and impact of mental health condition) characteristics. Exploratory analyses according to the type of barrier are presented in Supplementary Table S5. Data were analyzed using STATA/SE 16.0, version 16.

Results

Participant characteristics are described in Supplementary Table S3. Table 1 presents the types of services used overall and by the trajectory of mental health condition. Across all mental health trajectory groups, formal service utilization was 6–7 times more common than informal care during the previous year. Most youth using informal care did so in combination with formal care, particularly among those with a persistent psychiatric diagnosis (83.3%).

Table 1. Mental health-related service use in the past 12 months among Brazilian young people, overall, and by the trajectory of mental health condition (n = 1,400).

a One missing data.

Among formal service users, health services were used 3–4 times more often than educational services, across all mental health trajectory groups. The most frequent type of informal care was religious support (79.2%; Table 1).

After adjusting for youth mental health characteristics and caregiver characteristics, we found that parental stigma was the only caregiver characteristic associated with formal service use. Lower parental stigma was associated with higher odds of formal service use (i.e., each point on the RIBS-BP scale, increased the odds of formal service use by 12% [OR 95% CI: 1.04–1.91; p-value < 0.01]). Among young person characteristics, persistence of mental health conditions (OR: 3.15; 95% CI: 1.77–5.60; p-value < 0.01) and greater impact of mental health conditions on everyday life were associated with a greater likelihood of formal service use (i.e., each point on the SDQ impact scale increased odds of formal service use by 30% [OR 95% CI: 1.17–1.44; p < 0.01]; Table 2).

Table 2. Caregiver and child/adolescent characteristics associated with formal service use due to the trajectory of mental health condition in the past 12 months (bivariate and multivariable analyses; n = 1,400).

a Adjusted model with 1,385 participants; town of residence and data collection method (face-to-face or phone interview) were controlled.

SEG was the only caregiver variable associated with informal care: families from lower SEG (low-middle or low) had higher odds of informal care compared to families from higher SEG (OR: 4.51; 95% CI: 1.28–15.91; p-value = 0.02; Table 3).

Table 3. Caregiver and child/adolescent characteristics associated with informal service use due to the trajectory of mental health condition in the past 12 months (bivariate and multivariable analyses; n = 1,400).

a Adjusted model with 1,385 participants; town of residence and data collection method (face-to-face or phone interview) were controlled.

Although not the main aim of our article, we explored whether there were any differences in service utilization among those with externalizing versus internalizing conditions. We found that having either an internalizing or externalizing diagnosis increased the odds of contact with specialized services compared with no diagnosis; however, there was no difference in contact among those with internalizing problems versus externalizing problems (Supplementary Table S4).

Among the 133 participants who used some type of service during the previous year, 63 (47.4%) reported that they wanted or needed additional support for their youth’s mental health conditions, but could not access the support (defined as sufficient care). In terms of barriers to sufficient support (Table 4), youth with persistent psychiatric diagnoses experienced more barriers to additional support (65.8%) in comparison with those with transient mental health conditions (44.8%) or no mental health condition (32.4%). Most caregivers who reported experiencing barriers to additional support described multiple barriers impeding their access to care (71.4%). Considering the four broad barrier categories, structural barriers were the most common barrier to receiving additional support (43.6%), and this was mostly due to the cost of treatment. Recognition/literacy was the second most common barrier (27.1%), and lack of knowledge about where to go or whom to trust to ask for help was the most common reason for this type of barrier. About one-quarter (25.4%) of caregivers reported lack of trust/negative experiences as an impediment to additional support, and this was mainly due to “having a previous bad experience with professionals.” Nine percent of the caregivers faced anticipated stigma experienced in this subsample (Table 4).

Table 4. Prevalence of barriers to receiving sufficient mental health care and support among young people who used some type of service in the past 12 months (n = 133a).

a Analysis with 133 out of the total 155 participants who used some type of service.

We examined associations between caregiver and youth characteristics with reporting any barrier related to receiving additional support among the subsample of service users. No caregiver or youth characteristics were associated with greater likelihood of receiving sufficient mental health care among those participants using services (Table 5).

Table 5. Caregiver and young person characteristics associated with barriers to sufficient mental health care among those who received at least on type of service in the past 12 months (bivariate and multivariable analyses; n = 133).

a Adjusted model with 131 participants; town of residence and data collection method (face-to-face or phone interview) were controlled.

We further analyzed associations between caregiver and youth characteristics with the three most common types of barriers (structural, recognition/literacy, and lack of trust/negative experiences in services/treatments). Among those using services, three characteristics were found to be associated with respondents reporting “lack of trust/negative experiences in services/treatments” as a barrier to sufficient care: caregivers’ mental health problems (OR: 2.27; 95% CI: 0.84–6.16; p-value = 0.03), youth’s persistent psychopathology (OR: 3.15; 95% CI: 1.77–5.60; p-value < 0.01), and its greater impact on their everyday life (OR: 1.26; 95% CI: 1.06–1.49; p-value < 0.01). Due to small sample sizes, we consider these analyses exploratory and report them in the Supplementary Material only (Supplementary Table S5).

Discussion

We confirmed the main hypothesis that low socioeconomic status and caregiver stigma were associated with a lower likelihood of formal or informal child mental health service use. We did not confirm that the presence of caregiver mental health problems was associated with use of the same formal/informal services.

We found that persistence of youth mental health conditions and their impact on daily life were associated with higher odds of formal care. Importantly, caregiver characteristics and beliefs were also a key influence in the care process: lower parental stigma was associated with greater formal service use, and lower SEG was associated with greater odds of informal care.

From the literature, it was expected that youth psychopathology, and caregivers’ sociodemographic characteristics, such as SEG, would represent significant barriers to care [Reference Hahm, Cook, Ault-Brutus and Alegría23], particularly in LMICs such as Brazil [Reference Evans-Lacko, Aguilar-Gaxiola, Al-Hamzawi, Alonso, Benjet and Bruffaerts24].

The Brazilian health care system provides universal care for the entire population, with the mental health system being fully integrated into it. The Psychosocial Community Care Centers are treatment facilities staffed by multidisciplinary teams, and are the main source of mental health specialty care. Any Brazilian individual can seek mental health care directly from these units [25, Reference Paula, Lauridsen-Ribeiro, Wissow, IaSIAS and Evans-Lacko26]. Traditionally, most of the health funding is provided by the federal government, but in the recent years, there has been an increase in the participation of the private health system. For example, between 2015 and 2019, federal allocation decreased from 44.8 to 42.2%, whereas private health insurance (employer-subsidized plans) increased from 30.1 to 32.1%, and people themselves paying for care increased from 25.1 to 25.7% [27]. It should also be mentioned that like most LMICs and other countries in Latin America, mental health expenditure per person, as well as the number of mental health professionals, is below the recommended WHO standard [28].

It is interesting that our study found caregiver stigma to be the main barrier to access formal care for their child. Stigma associated with mental health problems represents a substantial burden to individuals, families, and society, being described as one of the most relevant barriers to care in different countries and cultures [Reference Evans-Lacko, Tachimori, Kovess-Masféty, Chatterji, Thornicroft, Stein, Scott, de Jonge and Kessler29]. Parental attitudes and beliefs are particularly relevant due to their role as a key gatekeeper to their child’s treatment access. Therefore, caregivers’ perceptions may influence concerns around labeling related to their child’s condition, and their perceptions of treatment options, impacting the help-seeking trajectory [Reference Gronholm, Ford, Roberts, Thornicroft, Laurens and Evans-Lacko7, Reference Reardon, Harvey, Baranowska, O’Brien, Smith and Creswell9].

At the same time, there are several effective interventions which can reduce help-seeking stigma, particularly encouraging are those involving different forms of social contact [Reference Thornicroft, Mehta, Clement, Evans-Lacko, Doherty and Rose30]. Despite this, most studies were carried out in high-income countries and with adults [Reference Xu, Huang, Kösters, Staiger, Becker and Thornicroft31]. Moreover, there is a paucity of research looking at the role of parental stigma on youth mental health service use. One study from the United Kingdom [Reference Gronholm, Ford, Roberts, Thornicroft, Laurens and Evans-Lacko7] found that the odds of service use were greater among caregivers with less intended stigmatizing behaviors. Another study in the United States identified higher social distance among parents with lower mental health problem recognition [Reference Villatoro, DuPont-Reyes, Phelan, Painter and Link32]. More recently, a cohort of young people from the United Kingdom identified that their persistent psychopathology, family socioeconomic disadvantage, and low caregiver intended stigma-related behavior were associated with increased likelihood of youth service use, whereas young people older age and socioeconomic disadvantage were associated with increased costs [Reference Ribeiro, Romeo, King, Owens, Gronholm and Fisher33].

Although the number of participants reporting informal service use was small in our sample, it represents a potentially important source of support and an area where little research has been conducted. We found this result surprisingly rare, knowing that families usually experience a wide variety of pathways to care which often includes informal care in regions and countries with fewer services [Reference MacDonald, Fainman-Adelman, Anderson and Iyer4]. Among those who used informal care, the vast majority with a mental health condition (almost 90%) also reported formal service use. This suggests that informal care did not prevent access to formal care [Reference Lamb, Bower, Rogers, Dowrick and Gask34], although we do not know if informal care led to formal care [Reference Rüdell, Bhui and Priebe35].

The type of informal care reported in our sample is similar to that reported in other studies, with religious services being the most frequently stated by Latino samples [Reference Givens, Houston, Van Voorhees, Ford and Cooper36]. Moreover, our rates of contact with religious providers for mental health care and complementary contacts with formal services were similar to those found among adult participants of the World Mental Health Survey [Reference Kovess-Masfety, Evans-Lacko, Williams, Andrade, Benjet and Ten Have37].

We also found that low SEG was the only significant predictor of greater informal care utilization. Caregivers from low SEGs might have easier access to and comfort with using informal care compared to formal mental health services since religious and other informal sources are usually free of charge [Reference Kovess-Masfety, Evans-Lacko, Williams, Andrade, Benjet and Ten Have37, Reference Yap, Reavley and Jorm38] and quite available in Brazil, a strongly religious country. These types of services may also be seen as less stigmatizing than more traditional formal mental health services. Moreover, in addition to mental health problems, these individuals may be more likely to experience social problems and complex circumstances, which may limit accessibility to formal care, including out of pocket payments, lack of health insurance, and difficulties in getting to the service [Reference Reardon, Harvey, Baranowska, O’Brien, Smith and Creswell9, Reference Martinez, Co, Lau and Brown39].

The low informal care utilization in the current study may also have other explanations. It is possible that caregivers use informal care, but do not identify these sources of care as providing mental health support, or they may not consider these types of informal providers as capable of dealing with mental health problems and thus prefer formal care. Additionally, our definition of informal care was focused on nontraditional providers which could deliver some services and support, but did not include family and friends which can also be an informal source of help [Reference Lamb, Bower, Rogers, Dowrick and Gask34, Reference Rüdell, Bhui and Priebe35]. Although our results are novel, the numbers are very low (only 24 youth used informal care); thus, further studies are necessary to understand factors related to informal care.

No other youth characteristics (age or gender) impacted informal or formal service utilization in our sample. Previous studies from Brazil and other countries suggest that service use may vary according to gender, age, and type of condition. For example, boys with externalizing symptoms access services more often during childhood, whereas girls with internalizing problems receive more help during adolescence [Reference Zwaanswijk, Verhaak, Bensing, van der Ende and Verhulst40Reference Georgiades, Duncan, Wang, Comeau and Boyle42]. There is no consensus in the field about these relationships as they also depend on sociocultural context; however, we also did not have the power to explore the interactions between these different variables and how they may change as the child ages and we recognize this is a limitation of our study.

As described above, at least in high-income countries, there is good evidence about barriers related to nonattendance [Reference Sayal43] as well as treatment dropout [Reference Reid, Cunningham, Tobon, Evans, Stewart and Brown44], but less is known about barriers to care among those using some type of service [Reference Reid, Cunningham, Tobon, Evans, Stewart and Brown44, Reference Kazdin and Wassell45]. The next step of our analysis was to understand the main barriers among the subsample of youth already using services and needing additional care. While just under one-third of those with persistent and impactful problems received any care, the majority of those in care reported needing more support and experienced barriers to sufficient care. Thus, stigma was important for getting people into care, while structural barriers were the most common obstacle to achieve sufficient care.

Among those using services, the majority did not feel that the care was adequate and structural barriers (mainly high cost, distance, and waiting lists) were the most significant impediments to receiving sufficient care. Hence, stigma was less important, suggesting that after families were able to access treatment, stigma decreased its power as a barrier. Another hypothesis is that people already consulting services had low stigma; therefore, the other types of barriers were more relevant for them.

Structural barriers are well known as being one of the main problems in respect of accessing mental health care. Increased financial investments, better coordination of the existing resources, the training of health providers, and empowering professionals and parents are some of the strategies recommended to improve the quality of youth mental health provision [Reference Paula, Bordin, Mari, Velasque, Rohde and Coutinho8, Reference Reardon, Harvey, Baranowska, O’Brien, Smith and Creswell9, Reference Paula, Lauridsen-Ribeiro, Wissow, IaSIAS and Evans-Lacko26, Reference Patel, Kieling, Maulik and Divan46].

Our study did not identify any barriers to sufficient mental health care related to caregiver or the characteristics of the young people. Our exploratory analysis, however, identified weak evidence of two potential barriers which may require further investigation: the presence of caregiver mental health problems, and mental health conditions that have a significant effect on the everyday lives of the youths. Families with a child with more severe dysfunction and the presence of parent psychopathology might result in more complex needs due to a higher number of problems [Reference Ryan, Jorm, Toumbourou and Lubman6, Reference Reid, Cunningham, Tobon, Evans, Stewart and Brown44, Reference Kazdin and Wassell45]. Moreover, potential burden associated with the child’s problems may be amplified in the presence of a caregiver’s own mental health problems [Reference Ryan, Jorm, Toumbourou and Lubman6]. Given the shortage of mental health specialists everywhere [Reference Wang, Aguilar-Gaxiola, Alonso, Angermeyer, Borges and Bromet47], including in Brazil [Reference Paula, Bordin, Mari, Velasque, Rohde and Coutinho8, Reference Paula, Lauridsen-Ribeiro, Wissow, IaSIAS and Evans-Lacko26], access to sufficient care is more challenging for complex cases who need more intense and specialized help [Reference Evans-Lacko, Tachimori, Kovess-Masféty, Chatterji, Thornicroft, Stein, Scott, de Jonge and Kessler29, Reference Reid, Cunningham, Tobon, Evans, Stewart and Brown44]. Further studies with larger sample sizes are required to verify these hypotheses.

The current study has some limitations. There were some differences between the original sample and the final one at the follow-up due to attrition rates. Even though not ideal, this does not seem to be a major issue because our study is not a prevalence or incidence study, where representative samples would be mandatory. Although we used a validated instrument which was adapted to the Brazilian context, data related to service use were limited to caregiver reports. However, the concordance between parent report and records of service use on the parental version of SACA is strong [Reference Hoagwood, Horwitz, Stiffman, Weisz, Bean and Rae48]. There may also be underreporting of informal care, given the interviewers could be perceived as scientists or mental health professionals. Parental stigma was assessed after service contact; therefore, it is difficult to establish the direction of the effect of stigma on service use. The frequency of service utilization was small; therefore, results should be taken with caution and additional studies are needed to replicate these findings, especially from informal care and related to barriers to sufficient care. Finally, the young person’s mental health condition at baseline was reported exclusively by their caregivers; the inclusion of the young person’s own perspective in respect of their mental health would have increased the accuracy of our data.

Even with these limitations, there are few studies with detailed data on barriers to accessing youth mental health services, using validated measures of service use covering a wide range of types of formal and informal care among a large community sample. It is noteworthy that this is one of the few studies looking at youth mental health service use in an LMIC.

In conclusion, our results reinforce the relevance of caregivers on the help-seeking process to access services for youth with mental health problems. More specifically, the role of parental stigma was a key barrier to formal care and low socioeconomic position to informal care. Thus, generating data about effective interventions to reduce stigma and promote access to coordinated care are important priorities worldwide, and particularly among vulnerable populations.

Supplementary Materials

To view supplementary material for this article, please visit http://doi.org/10.1192/j.eurpsy.2022.24.

Data Availability Statement

Data were provided by the Brazilian High-Risk Cohort Study and are available upon request in the Open Science Framework public repository (https://osf.io/ktz5h/).

Authors contributions

Conceptualizartion: CSP, SEL. Data curation: SEL, ECM, LAR, JJM, RB, GS, PMP, WSR. Formal analysis: CSP, SEL, CZ. Funding acquisition: ECM, LAR, RB, SEL, JJM, GS, PMP. Methodology: CSP, SEL, CZ, WSR. Project Administration: SEL, WSR, GS, PMP. Writing- review and editing: All authors.

Financial Support

The research presented in this article was funded by the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007-2013)/ERC Grant Agreement No. 337673 (APPLAUSE study), and supported by the National Institute of Developmental Psychiatry for Children and Adolescents, a science and technology institute funded by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq; National Council for Scientific and Technological Development; Grant Numbers 573974/2008-0 and 465550/2014-2) and the Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP Grant Numbers 2008/57896-8 and 2014/50917-0; BHRCS). C.S.P., J.J.M., R.B., E.C.M., L.A.R., and G.A.S. are researchers from the National Research Council (CNPQ). C.S.P. receives funding from CAPES/PRINT Grant Number 88887.310343/2018-00, FAPESP Grant Number 2018/12747-7 (Research Fellowship Abroad—Bolsa de Pesquisa no Exterior [BPE]), and Fundo Mackpesquisa. C.Z. receives a Young Talent Research Scholarship by the Coordination for the Improvement of Higher Education Personnel (CAPES [Brazil] Grant Number 88887.575201/2020-00).

Conflicts of Interest

On behalf of the other authors, the corresponding author states that there is no conflict of interest to declare. P.M.P. has received payment or honoraria for lectures and presentations in educational events for Sandoz, Daiichi Sankyo, Eurofarma, Abbot, Libbs, Instituto Israelita de Pesquisa e Ensino Albert Einstein, and Instituto D’Or de Pesquisa e Ensino. R.B. reports personal fees from Torrent, Ludbeck, and Ache, and personal fees and nonfinancial support from Janssen, outside the submitted work; has received grants from the European Research Council, Medical Research Council (UK), and Cyted—Programa Iberoamericano de Ciencia y Tecnología para el Desarrollo; and has received grants from CNPq, FAPESP, and CAPES (Brazil). L.A.R. has received grant or research support from, served as a consultant to, and served on the speakers’ bureau of Aché, Bial, Medice, Novartis/Sandoz, Pfizer/Upjohn, and Shire/Takeda in the last 3 years. The ADHD and Juvenile Bipolar Disorder Outpatient Programs chaired by L.A.R. have received unrestricted educational and research support from the following pharmaceutical companies in the last 3 years: Novartis/Sandoz and Shire/Takeda. L.A.R. has received authorship royalties from Oxford Press and ArtMed.

References

Polanczyk, GV, Salum, GA, Sugaya, LS, Caye, A, Rohde, LA. Annual research review: a meta-analysis of the worldwide prevalence of mental disorders in children and adolescents. J Child Psychol Psychiatry. 2015;56(3):345–65.CrossRefGoogle ScholarPubMed
Kieling, C, Baker-Henningham, H, Belfer, M, Conti, G, Ertem, I, Omigbodun, O, et al. Child and adolescent mental health worldwide: evidence for action. Lancet. 2011;378(9801):1515–25.CrossRefGoogle ScholarPubMed
Kessler, RC, Soukup, J, Davis, RB, Foster, DF, Wilkey, SA, Van Rompay, MI, et al. The use of complementary and alternative therapies to treat anxiety and depression in the United States. Am J Psychiatry. 2001;158(2):289–94.CrossRefGoogle ScholarPubMed
MacDonald, K, Fainman-Adelman, N, Anderson, KK, Iyer, SN. Pathways to mental health services for young people: a systematic review. Soc Psychiatry Psychiatr Epidemiol. 2018;53(10):1005–38.CrossRefGoogle ScholarPubMed
Brown, JSL, Evans-Lacko, S, Aschan, L, Henderson, MJ, Hatch, SL, Hotopf, M. Seeking informal and formal help for mental health problems in the community: a secondary analysis from a psychiatric morbidity survey in South London. BMC Psychiatry. 2014;14(1):275.CrossRefGoogle ScholarPubMed
Ryan, SM, Jorm, AF, Toumbourou, JW, Lubman, DI. Parent and family factors associated with service use by young people with mental health problems: a systematic review. Early Interv Psychiatry. 2015;9(6):433–46.CrossRefGoogle ScholarPubMed
Gronholm, PC, Ford, T, Roberts, RE, Thornicroft, G, Laurens, KR, Evans-Lacko, S. Mental health service use by young people: the role of caregiver characteristics. PLoS One. 2015;10(3):e0120004.CrossRefGoogle ScholarPubMed
Paula, CS, Bordin, IAS, Mari, JJ, Velasque, L, Rohde, LA, Coutinho, ESF. The mental health care gap among children and adolescents: data from an epidemiological survey from four Brazilian regions. PLoS One. 2014;9(2):e88241.CrossRefGoogle ScholarPubMed
Reardon, T, Harvey, K, Baranowska, M, O’Brien, D, Smith, L, Creswell, C. What do parents perceive are the barriers and facilitators to accessing psychological treatment for mental health problems in children and adolescents? A systematic review of qualitative and quantitative studies. Eur Child Adolesc Psychiatry. 2017;26(6):623–47.CrossRefGoogle ScholarPubMed
Salum, GA, Gadelha, A, Pan, PM, Moriyama, TS, Graeff-Martins, AS, Tamanaha, AC, et al. High risk cohort study for psychiatric disorders in childhood: rationale, design, methods and preliminary results. Int J Methods Psychiatr Res. 2015;24(1):5873.CrossRefGoogle ScholarPubMed
Stiffman, AR, Horwitz, SM, Hoagwood, K, Compton, W, Cottler, L, Bean, DL, et al. The Service Assessment for Children and Adolescents (SACA): adult and child reports. J Am Acad Child Adolesc Psychiatry. 2000;39(8):1032–9.CrossRefGoogle ScholarPubMed
Horwitz, SM, Hoagwood, K, Stiffman, AR, Summerfeld, T, Weisz, JR, Costello, EJ, et al. Reliability of the Services Assessment for Children and Adolescents. Psychiatr Serv. 2001;52(8):1088–94.CrossRefGoogle ScholarPubMed
Goodman, R, Ford, T, Richards, H, Gatward, R, Meltzer, H. The Development and Well-Being Assessment: description and initial validation of an integrated assessment of child and adolescent psychopathology. J Child Psychol Psychiatry. 2000;41(5):645–55.CrossRefGoogle ScholarPubMed
Fleitlich-Bilyk, B, Goodman, R. Prevalence of child and adolescent psychiatric disorders in Southeast Brazil. J Am Acad Child Adolesc Psychiatry. 2004;43(6):727–34.CrossRefGoogle ScholarPubMed
Goodman, R. Psychometric properties of the Strengths and Difficulties Questionnaire. J Am Acad Child Adolesc Psychiatry. 2001;40(11):1337–45.CrossRefGoogle ScholarPubMed
Stringaris, A, Goodman, R. The value of measuring impact alongside symptoms in children and adolescents: a longitudinal assessment in a community sample. J Abnorm Child Psychol. 2013;41(7):1109–20.CrossRefGoogle Scholar
Associaçao Brasileira de Pesquisa de Mercado A. Critério de classificação econômica Brasil; 2010.Google Scholar
Kessler, RC, Green, JG, Gruber, MJ, Sampson, NA, Bromet, E, Cuitan, M, et al. Screening for serious mental illness in the general population with the K6 screening scale: results from the WHO World Mental Health (WMH) survey initiative. Int J Methods Psychiatr Res. 2010;19(Suppl 1):4–22.CrossRefGoogle ScholarPubMed
Kessler, RC, Andrews, G, Colpe, LJ, Hiripi, E, Mroczek, DK, Normand, SLT, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med. 2002;32(6):959–76.CrossRefGoogle ScholarPubMed
Kessler, RC, Barker, PR, Colpe, LJ, Epstein, JF, Gfroerer, JC, Hiripi, E, et al. Screening for serious mental illness in the general population. Arch Gen Psychiatry. 2003;60(2):184.CrossRefGoogle ScholarPubMed
Evans-Lacko, S, Rose, D, Little, K, Flach, C, Rhydderch, D, Henderson, C, et al. Development and psychometric properties of the Reported and Intended Behaviour Scale (RIBS): a stigma-related behaviour measure. 2011;20:263–71.CrossRefGoogle Scholar
Ribeiro, WS, Gronholm, PC, Silvestre de Paula, C, Hoffmann, MS, Rojas Vistorte, AO, Zugman, C, et al. Development and validation of the Brazilian Portuguese version of the Reported and Intended Behaviour Scale (RIBS-BP). Stigma and Health. 2021;6(2):163–72.CrossRefGoogle Scholar
Hahm, HC, Cook, BL, Ault-Brutus, A, Alegría, M. Intersection of race-ethnicity and gender in depression care: screening, access, and minimally adequate teatment. Psychiatric Serv. 2015;66(3):258–64.CrossRefGoogle Scholar
Evans-Lacko, S, Aguilar-Gaxiola, S, Al-Hamzawi, A, Alonso, J, Benjet, C, Bruffaerts, R, et al. Socio-economic variations in the mental health treatment gap for people with anxiety, mood, and substance use disorders: results from the WHO World Mental Health (WMH) surveys. Psychol Med. 2018;48(9):1560–71.CrossRefGoogle ScholarPubMed
Ministério da Saúde (BR). Saúde mental no SUS: OS Centros de Atencão Psicossocial. Brasilia: Ministério da Saúde; 2004 [accessed 27 January, 2022], https://www.nescon.medicina.ufmg.br/biblioteca/imagem/1212.pdfGoogle Scholar
Paula, CSCS, Lauridsen-Ribeiro, E, Wissow, L, IaSIAS, Bordin, Evans-Lacko, S. How to improve the mental health care of children and adolescents in Brazil: actions needed in the public sector. Rev Bras Psiquiatr. 2012;34(3):334–41.CrossRefGoogle ScholarPubMed
Ministério da Saúde (BR). Contas de saúde na perspectiva da contabilidade internacional: conta SHA para o Brasil, 2015 a 2019. Brasília: Instituto de Pesquisa Econômica Aplicada (IPEA); 2022. https://www.ipea.gov.br/portal/images/stories/PDFs/livros/livros/220202_livro_contas_de_saude.pdf May 2nd 2022.Google Scholar
World Health Organization. Mental health atlas 2017. Geneva: World Health Organization; 2018. https://www.who.int/publications/i/item/9789241514019 May 2nd 2022.Google Scholar
Evans-Lacko, S, Tachimori, H, Kovess-Masféty, V, Chatterji, S, Thornicroft, G. Service use. In: Stein, DJ, Scott, KM, de Jonge, P, Kessler, RC, editors. Mental disorders around the world: facts and figures from the WHO World Mental Health surveys. Cambridge: Cambridge University Press; 2018, p. 314–23.Google Scholar
Thornicroft, G, Mehta, N, Clement, S, Evans-Lacko, S, Doherty, M, Rose, D, et al. Evidence for effective interventions to reduce mental-health-related stigma and discrimination. Lancet. 2016;387(10023):1123–32.CrossRefGoogle ScholarPubMed
Xu, Z, Huang, F, Kösters, M, Staiger, T, Becker, T, Thornicroft, G, et al. Effectiveness of interventions to promote help-seeking for mental health problems: systematic review and meta-analysis. Psychol Med. 2018;48(16):2658–67.CrossRefGoogle ScholarPubMed
Villatoro, AP, DuPont-Reyes, MJ, Phelan, JC, Painter, K, Link, BG. Parental recognition of preadolescent mental health problems: does stigma matter? Soc Sci Med. 2018;216(5):8896.CrossRefGoogle ScholarPubMed
Ribeiro, WS, Romeo, R, King, D, Owens, S, Gronholm, PC, Fisher, HL, et al. Influence of stigma, sociodemographic and clinical characteristics on mental health-related service use and associated costs among young people in the United Kingdom. Eur Child Adolesc Psychiatry. 2022 DOI: 10.1007/s00787-022-01947-2.CrossRefGoogle ScholarPubMed
Lamb, J, Bower, P, Rogers, A, Dowrick, C, Gask, L. Access to mental health in primary care: a qualitative meta-synthesis of evidence from the experience of people from “hard to reach” groups. Health. 2012;16(1):76–104.CrossRefGoogle Scholar
Rüdell, K, Bhui, K, Priebe, S. Do “alternative” help-seeking strategies affect primary care service use? A survey of help-seeking for mental distress. BMC Public Health. 2008;8(1):207.CrossRefGoogle ScholarPubMed
Givens, JL, Houston, TK, Van Voorhees, BW, Ford, DE, Cooper, LA. Ethnicity and preferences for depression treatment. Gen Hosp Psychiatry. 2007;29(3):182–91.CrossRefGoogle ScholarPubMed
Kovess-Masfety, V, Evans-Lacko, S, Williams, D, Andrade, LH, Benjet, C, Ten Have, M, et al. The role of religious advisors in mental health care in the World Mental Health surveys. Soc Psychiatry Psychiatr Epidemiol. 2017;52(3):353–67.CrossRefGoogle ScholarPubMed
Yap, MBH, Reavley, N, Jorm, AF. Where would young people seek help for mental disorders and what stops them? Findings from an Australian national survey. J Affect Disord. 2013;147(1–3):255–61.CrossRefGoogle ScholarPubMed
Martinez, AB, Co, M, Lau, J, Brown, JSL. Filipino help-seeking for mental health problems and associated barriers and facilitators: a systematic review. Soc Psychiatry Psychiatr Epidemiol. 2020;55(11):1397–413.CrossRefGoogle ScholarPubMed
Zwaanswijk, M, Verhaak, PF, Bensing, JM, van der Ende, J, Verhulst, FC. Help seeking for emotional and behavioural problems in children and adolescents: a review of recent literature. Eur Child Adolesc Psychiatry. 2003;12(4):153–61.CrossRefGoogle ScholarPubMed
Ceballos, GY, Paula, CS, Ribeiro, EL, Santos, DN. Child and adolescent psychosocial care center service use profile in Brazil: 2008 to 2012. Braz J Psychiatry. 2019;41(2):138–47.CrossRefGoogle ScholarPubMed
Georgiades, K, Duncan, L, Wang, L, Comeau, J, Boyle, MH, Ontario Child Health Study Team. Six-month prevalence of mental disorders and service contacts among children and youth in Ontario: evidence from the 2014 Ontario child health study. Can J Psychiatry. 2019;64(4):246–55.CrossRefGoogle ScholarPubMed
Sayal, K. Annotation: pathways to care for children with mental health problems. J Child Psychol Psychiatry. 2006;47(7):649–59.CrossRefGoogle ScholarPubMed
Reid, GJ, Cunningham, CE, Tobon, JI, Evans, B, Stewart, M, Brown, JB, et al. Help-seeking for children with mental health problems: parents’ efforts and experiences. Admin Pol Ment Health. 2011;38(5):384–97.CrossRefGoogle ScholarPubMed
Kazdin, AE, Wassell, G. Predictors of barriers to treatment and therapeutic change in outpatient therapy for antisocial children and their families. Ment Health Serv Res. 2000;2(1):2740.CrossRefGoogle ScholarPubMed
Patel, V, Kieling, C, Maulik, PK, Divan, G. Improving access to care for children with mental disorders: a global perspective. Arch Dis Child. 2013;98(5):323–7.CrossRefGoogle ScholarPubMed
Wang, PS, Aguilar-Gaxiola, S, Alonso, J, Angermeyer, MC, Borges, G, Bromet, EJ, et al. Use of mental health services for anxiety, mood, and substance disorders in 17 countries in the WHO World Mental Health surveys. Lancet. 2007;370(9590):841–50.CrossRefGoogle ScholarPubMed
Hoagwood, K, Horwitz, S, Stiffman, A, Weisz, J, Bean, D, Rae, D, et al. Concordance between parent reports of children’s mental health services and service records: the Services Assessment for Children and Adolescents (SACA). J Child Fam Stud. 2000;9(3):315–31.CrossRefGoogle Scholar
Figure 0

Table 1. Mental health-related service use in the past 12 months among Brazilian young people, overall, and by the trajectory of mental health condition (n = 1,400).

Figure 1

Table 2. Caregiver and child/adolescent characteristics associated with formal service use due to the trajectory of mental health condition in the past 12 months (bivariate and multivariable analyses; n = 1,400).

Figure 2

Table 3. Caregiver and child/adolescent characteristics associated with informal service use due to the trajectory of mental health condition in the past 12 months (bivariate and multivariable analyses; n = 1,400).

Figure 3

Table 4. Prevalence of barriers to receiving sufficient mental health care and support among young people who used some type of service in the past 12 months (n = 133a).

Figure 4

Table 5. Caregiver and young person characteristics associated with barriers to sufficient mental health care among those who received at least on type of service in the past 12 months (bivariate and multivariable analyses; n = 133).

Supplementary material: File

Paula et al. supplementary material

Paula et al. supplementary material

Download Paula et al. supplementary material(File)
File 67.8 KB
Submit a response

Comments

No Comments have been published for this article.