Hostname: page-component-586b7cd67f-2brh9 Total loading time: 0 Render date: 2024-12-04T21:46:49.779Z Has data issue: false hasContentIssue false

Methods and tools to assess implementation of mental health policies and plans: A systematic review

Published online by Cambridge University Press:  17 March 2023

Akmal Alikhan Aliev*
Affiliation:
Department of Public Mental Health, National Institute of Mental Health, Klecany, Czechia WHO Collaborating Centre for Public Mental Health Research and Service Development, Klecany, Czechia
Hana Tomaskova
Affiliation:
Department of Public Mental Health, National Institute of Mental Health, Klecany, Czechia WHO Collaborating Centre for Public Mental Health Research and Service Development, Klecany, Czechia Department of Psychology, Charles University, Prague, Czechia
Petr Winkler
Affiliation:
Department of Public Mental Health, National Institute of Mental Health, Klecany, Czechia WHO Collaborating Centre for Public Mental Health Research and Service Development, Klecany, Czechia Health Service and Population Research Department, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, UK
Yongjie Yon
Affiliation:
World Health Organization Regional Office for Europe, Copenhagen, Denmark
Anna Kagstrom
Affiliation:
Department of Public Mental Health, National Institute of Mental Health, Klecany, Czechia WHO Collaborating Centre for Public Mental Health Research and Service Development, Klecany, Czechia
Zoe Guerrero
Affiliation:
Department of Public Mental Health, National Institute of Mental Health, Klecany, Czechia WHO Collaborating Centre for Public Mental Health Research and Service Development, Klecany, Czechia
Ledia Lazeri
Affiliation:
World Health Organization Regional Office for Europe, Copenhagen, Denmark
Marge Reinap
Affiliation:
World Health Organization Regional Office for Europe, Copenhagen, Denmark
Cassie Redlich
Affiliation:
World Health Organization Regional Office for Europe, Copenhagen, Denmark
Ana Maria Tijerino Inestroza
Affiliation:
World Health Organization Regional Office for Europe, Copenhagen, Denmark
Jason Maurer
Affiliation:
World Health Organization Regional Office for Europe, Copenhagen, Denmark
*
Author for correspondence: Akmal Alikhan Aliev, Email: akmal.aliev@nudz.cz
Rights & Permissions [Opens in a new window]

Abstract

Mental health policies and plans (MHPPs) are important policy instruments and powerful tools to facilitate development of mental health systems and services across the world. We aimed to map and analyse methods and tools used to assess the extent, process and impact of implementing MHPPs. We systematically searched peer-reviewed and grey literature across seven scientific databases. We extracted and analysed the data on a) the characteristics of included studies (e.g., policy areas, region of origin, income setting) and b) the methodology and evaluation tools applied to assess the extent and process of implementation. We included 48 studies in the analyses. Twenty-six of these studies employed only qualitative methods (e.g., semi-structured interviews, focus group discussions, desk review, stakeholder consultations); 12 studies used quantitative methods (e.g., trend analysis, survey) and 10 used mixed-methods approaches. Generally, methods and tools used for assessment were described poorly with less than half of the studies providing partial or full details about them. Only three studies provided assessment of full policies. There is a lack of rigorous research to assess implementation MHPPs. Assessments of the implementation of entire MHPPs are almost non-existent. Strategies to assess the implementation of MHPPs should be an integral part of MHPPs.

Topics structure

Type
Review
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - SA
This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike licence (http://creativecommons.org/licenses/by-nc-sa/4.0), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the same Creative Commons licence is used to distribute the re-used or adapted article and the original article is properly cited. The written permission of Cambridge University Press must be obtained prior to any commercial use.
Copyright
© The Author(s), 2023. Published by Cambridge University Press

Impact statement

This review highlights a lack of both quantity and rigorousness of assessments of implementation of mental health policies and plans as reported across the globe. While studies included in the review often addressed in detail the posed research questions and assessment objectives, they rarely presented clearly their methods and lacked sufficient descriptions of tools used in the evaluation, thus making them hardly interpretable and reproducible in other research contexts. Reports of assessments of entire policies were scarce. Instead, studies largely focused on assessment of certain policy objectives or tried to investigate questions of interest in relation to the implementation process. Thus, the review reveals gaps within implementation science in global mental health and calls for future efforts to better assess the impact of mental health policies and so to enable learning from the lessons made.

Introduction

The need for urgent improvements in mental healthcare systems across the globe has been recognised for a long time (WHO, 2001). The global burden of mental disorders is ever increasing, and the treatment gap still prevails across all income settings. Mental health and well-being have been further compromised by the COVID-19 pandemic, and forthcoming challenges, such as climate change and associated migration and population displacement, are likely to exacerbate the existing burden. Mental health policies and plans (MHPPs) are important policy instruments to spark and concert action for change, yet the methodologies and tools to assess the extent of implementation of MHPPs have not been properly examined.

Since the launch of the World Health Organisation (WHO) Global and European Mental Health Action Plans in 2013, many countries introduced new national MHPPs. Two-thirds of countries in the WHO European region have either developed or updated their national mental health policies or laws since then (WHO, 2018). While this is a welcome development, it is important to ensure that these newly developed or updated MHPPs have a real and important impact on the mental health and well-being of populations.

However, there are a number of challenges that both high-income (HICs) and low- and middle-income countries (LMICs) face in implementing their MHPPs (Zhou et al., Reference Zhou, Yu, Yang, Chen and Xiao2018), such as limited access to financial and human resources (WHO, 2015) and low public mental health literacy (Campion and Knapp, Reference Campion and Knapp2018). In LMICs, challenges including a lack of professional training among healthcare workers, opposition from key stakeholders and resistance to decentralisation of mental health services are reported as more pronounced than in HICs (Saraceno et al., Reference Saraceno, van Ommeren, Batniji, Cohen, Gureje, Mahoney, Sridhar and Underhill2007). The processes countries take to implement MHPPs, including identifying the bottlenecks and facilitating factors, are largely unknown due to the lack of implementation and evaluation studies (Zhou et al., Reference Zhou, Yu, Yang, Chen and Xiao2018). For example, the Strategy of Psychiatric Care Reform from Czechia that was launched in 2017/2018 and has ended in 2022 contains measurable indicators for each of the 10 implementation projects. However, to date no large-scale evaluation has been conducted to assess its implementation. Against this context, and while taking into consideration that many existing national mental health strategies, policies and action plans in the WHO European Region and beyond are now close to their expiration, we aimed to map and analyse tools and methodologies used to assess the extent and process of implementing national or regional MHPPs. This mapping review intends to inform policy development, implementation and evaluation in the WHO European Region.

Methods

We conducted a systematic search of peer-reviewed and grey literature to identify assessments of MHPPS. We followed the protocol recommended by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (https://www.prisma-statement.org/) to report the screening process and findings (Moher et al., Reference Moher, Liberati, Tetzlaff and Altman2010; Page et al., Reference Page, Moher, Bossuyt, Boutron, Hoffmann, Mulrow, Shamseer, Tetzlaff, Akl, Brennan, Chou, Glanville, Grimshaw, Hróbjartsson, Lalu, Li, Loder, Mayo-Wilson, McDonald, McGuinness, Stewart, Thomas, Tricco, Welch, Whiting and McKenzie2021) and we registered the protocol in the Prospero Database for systematic reviews (Registration #CRD42022290862).

Search strategy and selection criteria

We searched the following bibliographic databases: Global Health, Medline, Embase, Web of Science, Global Index Medicus, WHO MindBank and Open Grey using four sets of search terms to identify relevant studies. These search terms included i) reform or policy or strategy or plan, ii) mental health or psych* or suicide or dementia, iii) implementation and iv) national or government*. Full search strategy is available in the Appendix. Additionally, we screened the lists of references of studies included from the main search as well as from the systematic review on mental health policies conducted by Zhou et al. (Reference Zhou, Yu, Yang, Chen and Xiao2018). The previous review by Zhou et al. (Reference Zhou, Yu, Yang, Chen and Xiao2018) provided an excellent foundation to understand the challenges in implementing mental health policies; however, to the knowledge of the authors of this systematic review, our study is the first of its kind to map existing methodologies and tools to assess the implementation of MHPPs.

We included studies that covered MHPPs, as well as policies covering specific mental health areas, including child and adolescent mental health, suicide prevention or dementia. These priority areas are included in the WHO European Framework for Action on Mental Health (2021–2025). We used WHO definitions of a mental health policy, which is referred to ‘an organized set of values, principles and objectives for improving mental health and reducing the burden of mental disorders in a population and defines a vision for future action’. A mental health plan is defined as ‘a detailed scheme for implementing strategic actions that addresses the promotion of mental health, the prevention of mental health conditions, and treatment and rehabilitation’.

We excluded studies that were 1) not focused on implementation of policies at a national or regional level; 2) policies that fall outside of the priority areas of the WHO European Framework for Action on Mental Health (e.g., substance use disorders, depression); 3) did not evaluate implementation of mental health policy or 4) did not describe any methods of evaluation. Conference abstracts, study protocols, opinion papers and editorials were also excluded. The study selection was not limited by year of publication or country of origin. Multiple languages (English, French, Spanish and Russian) were searched to ensure relevant studies are identified and captured.

All references identified through databases were imported into Rayyan’s online reference manager. After deleting duplicates, AA and HT independently screened titles and abstracts, following full-text examination of included articles. All disagreements were resolved by discussion.

Data extraction and analysis

We extracted and analysed data relevant to both characteristics of included studies and methods of assessment. Characteristics of the studies included the WHO Region of the studied country; country income classification according to World Bank; study objectives; policy period; policy area (e.g., such as suicide prevention, dementia prevention or mental healthcare development) and scope of evaluation of policy implementation, which we further divided into three categories: 1) progress and 2) process of implementation and impact 3) of MHPPs. We defined

  • progress of implementation as a measure of extent to which MHPPs were implemented;

  • process of implementation as referring to assessing barriers and facilitators, active ingredients, drivers, cultures, structures, ethics, pace and timing, or other related factors influencing implementation of mental health policies and

  • impact of policy as referring to achievements resulting from implementation of MHPPs.

We also extracted information about methods of evaluation including study design (quantitative, qualitative; mixed methods), aspects of evaluation (such as quantitative indicators), theoretical frameworks used for evaluation and tools used to assess the extent and process of implementation of MHPPs (e.g., questionnaire, interview guide).

We distinguished studies that provided description of the tools used in evaluation (e.g., for interviews and focus group discussions, providing an interview guide showing the areas covered by the interviewer) and articles that were limited to simply mentioning the tool. When possible, we contacted authors to provide examples of the tools they had used, but not if these had not been reported in detail. We also highlighted studies that indicated that tools were pretested and those that supplemented detailed instruction or guides for using the methods and tools and interpreting results. Additionally, we wanted to distinguish studies that aimed at extensive assessment of all policy objectives from those that implemented specific programmes or interventions or certain parts of a policy.

In reporting the results, we adopted a structure used in the review of methods and tools to assess food and health policies by Phulkerd et al. (Reference Phulkerd, Lawrence, Vandevijvere, Sacks, Worsley and Tangcharoensathien2015). Findings are presented separately by progress, process and impact. Each section includes analysis of policy areas of the studies, measured indicators and finally tools and methods used to assess policy implementation.

Results

The electronic database search identified 7,298 studies. After removing duplicates, 3,120 unique items were left. Unrelated abstracts were excluded based on title/abstract screening leaving 88 full texts for further screening, of which 22 studies were eligible for inclusion. Another 26 studies were selected from either reference lists of included articles or publications known to authors including 7 grey literature publications. In total, 48 studies were included (Figure 1).

Figure 1. Screening and selection of articles.

Most countries represented were HIC (n = 32) followed by five studies on upper-middle and four on lower-middle-income countries (UMIC and LMIC). Six articles focused on low-income countries (LIC). The European Region was the region with the most publications (n = 22), followed by African (n = 12) and Western Pacific (n = 9) regions; three studies were from the Americas, and one from South-East Asia and Eastern Mediterranean Region each (Table 1).

Table 1. Characteristics of included studies

We found 40 studies that assessed implementation of mental health policies, seven studies assessed implementation of suicide prevention strategies in Australia, Japan (n = 2), Northern Ireland, UK Scotland, UK (n = 2) and United States of America and only one study assessed dementia prevention policy in three European countries (Denmark, Germany, Italy). We found no study assessing the implementation of child and adolescent mental health policy. All studies, with the exception of one, focused on policies implemented at the national level. One study assessed the policy at both national and district levels (Doku et al., Reference Doku, Ofori-Atta, Akpalu, Read, Osei, Ae-Ngibise, Awenva, Lund, Flisher, Petersen, Bhana, Bird, Drew, Faydi, Funk, Green and Omar2008). Only three studies presented results of assessment of entire MHPPs (Australian Health Ministers Advisory Council, 1997; Department of Health, Social Services and Public Safety, 2012; Loukidou et al., Reference Loukidou, Mastroyannakis, Power, Thornicroft, Craig and Bouras2013). More details on each study are provided in Tables 24.

Table 2. Summary of identified studies assessing the progress of implementation

Table 3. Summary of identified studies assessing the process of implementation

Table 4. Summary of identified studies assessing the impact of mental health policies and plans

Progress of policy implementation

Overview

We identified 14 studies that assessed progress of implementation, usually together with process assessment (n = 3) (Mwanza et al., Reference Mwanza, Sikwese, Mwanza, Mayeya, Lund, Bird, Drew, Faydi, Funk and Green2008; Draper et al., Reference Draper, Lund, Kleintjes, Funk, Omar and Flisher2009; Omar et al., Reference Omar, Green, Bird, Mirzoev, Flisher, Kigozi, Lund, Mwanza and Ofori-Atta2010) or impact assessment (n = 5) (Australian Health Ministers´ Advisory Council, 1997; Hickie and Groom, Reference Hickie and Groom2004; Loukidou et al., Reference Loukidou, Mastroyannakis, Power, Thornicroft, Craig and Bouras2013; Nakanishi et al., Reference Nakanishi, Yamauchi and Takeshima2015; Nakanishi and Endo, Reference Nakanishi and Endo2017) or both (n = 2) (Reid Howie Associates, 2006; Doku et al., Reference Doku, Ofori-Atta, Akpalu, Read, Osei, Ae-Ngibise, Awenva, Lund, Flisher, Petersen, Bhana, Bird, Drew, Faydi, Funk, Green and Omar2008). Two studies assessed solely progress of policy implementation (Dlouhy, Reference Dlouhy2014; Sheehan et al., Reference Sheehan, Griffiths, Rickwood and Carron-Arthur2015). Nine studies assessed policy implementation in HICs, four in UMICs, one in LMICs and three in LICs. Most (86%) of these studies were single-country focused, and two were multinational. Quantitative (e.g., surveys, questionnaires) and qualitative methods (e.g., interviews, focus group discussions) or a combination of both were used, with qualitative methods substantially prevailing.

Policy areas

Five studies, from HICs, focused on suicide prevention policies. The rest focused on mental health policies, which mostly included overarching mental healthcare development strategies. The contents of these overarching policies included mental health promotion and prevention of mental health problems (Hickie and Groom, Reference Hickie and Groom2004; Dlouhy, Reference Dlouhy2014; Nakanishi and Endo, Reference Nakanishi and Endo2017), improving quality of care (Hickie and Groom, Reference Hickie and Groom2004; Doku et al., Reference Doku, Ofori-Atta, Akpalu, Read, Osei, Ae-Ngibise, Awenva, Lund, Flisher, Petersen, Bhana, Bird, Drew, Faydi, Funk, Green and Omar2008; Nakanishi and Endo, Reference Nakanishi and Endo2017), strengthening research (Hickie and Groom, Reference Hickie and Groom2004) and deinstitutionalisation and development of community care (Doku et al., Reference Doku, Ofori-Atta, Akpalu, Read, Osei, Ae-Ngibise, Awenva, Lund, Flisher, Petersen, Bhana, Bird, Drew, Faydi, Funk, Green and Omar2008; Loukidou et al., Reference Loukidou, Mastroyannakis, Power, Thornicroft, Craig and Bouras2013). One study specifically looked at the integration of mental health services into primary healthcare (Draper et al., Reference Draper, Lund, Kleintjes, Funk, Omar and Flisher2009).

Aspects measured

Progress of policy implementation was expressed as a) existence of policy implementation at all or b) the degree of its implementation. Assessment of existence was expressed in qualitative or survey questions on whether any implementation activities were carried out. Eight studies examined the existence of policy implementation (Doku et al., Reference Doku, Ofori-Atta, Akpalu, Read, Osei, Ae-Ngibise, Awenva, Lund, Flisher, Petersen, Bhana, Bird, Drew, Faydi, Funk, Green and Omar2008; Mwanza et al., Reference Mwanza, Sikwese, Mwanza, Mayeya, Lund, Bird, Drew, Faydi, Funk and Green2008; Draper et al., Reference Draper, Lund, Kleintjes, Funk, Omar and Flisher2009; Omar et al., Reference Omar, Green, Bird, Mirzoev, Flisher, Kigozi, Lund, Mwanza and Ofori-Atta2010; Dlouhy, Reference Dlouhy2014; Nakanishi et al., Reference Nakanishi, Yamauchi and Takeshima2015; Nakanishi and Endo, Reference Nakanishi and Endo2017; Substance Abuse and Mental Health Services Administration, 2017). Degree of implementation was measured in a variety of ways in eight studies, seven of which focused on suicide prevention policies. For example, one study assessed perceptions of psychiatrists about degrees to which the key aspects of reform were implemented (Hickie and Groom, Reference Hickie and Groom2004), others measured progress by assessing whether a tangible outcome was produced (i.e., specific output) according to an implementation plan (e.g., completion of a report on training programmes) or the number or percentage of activities completed (e.g., number of training programmes provided, number of staff attending an event) (Department of Health, Social Services and Public Safety, 2012; Loukidou et al., Reference Loukidou, Mastroyannakis, Power, Thornicroft, Craig and Bouras2013; Sheehan et al., Reference Sheehan, Griffiths, Rickwood and Carron-Arthur2015; Substance Abuse and Mental Health Services Administration, 2017). One study examined level of usage of projects expressed in number of referrals to the programme or time users spent in the programme (Reid Howie Associates, 2006). Additionally, another study investigated patterns of implementation, that is, the frequency of types of suicide prevention programmes authorities chose to implement in different prefectures (Nakanishi et al., Reference Nakanishi, Yamauchi and Takeshima2015). Yet, other study compared areas of focus (e.g., addictions, unemployment) that were addressed by different authorities across the country in implementation of a national suicide prevention strategy (Nakanishi and Endo, Reference Nakanishi and Endo2017).

Methods and tools

Various methods were employed to measure progress of implementation. Four studies used quantitative methods: surveys or questionnaires (Hickie and Groom, Reference Hickie and Groom2004; Dlouhy, Reference Dlouhy2014; Nakanishi et al., Reference Nakanishi, Yamauchi and Takeshima2015; Sheehan et al., Reference Sheehan, Griffiths, Rickwood and Carron-Arthur2015). Three studies used qualitative semi-structured interviews (Doku et al., Reference Doku, Ofori-Atta, Akpalu, Read, Osei, Ae-Ngibise, Awenva, Lund, Flisher, Petersen, Bhana, Bird, Drew, Faydi, Funk, Green and Omar2008; Draper et al., Reference Draper, Lund, Kleintjes, Funk, Omar and Flisher2009; Omar et al., Reference Omar, Green, Bird, Mirzoev, Flisher, Kigozi, Lund, Mwanza and Ofori-Atta2010), and one study complemented qualitative interviews with literature review (Draper et al., Reference Draper, Lund, Kleintjes, Funk, Omar and Flisher2009). Five studies applied mixed-methods approaches (Mwanza et al., Reference Mwanza, Sikwese, Mwanza, Mayeya, Lund, Bird, Drew, Faydi, Funk and Green2008; Department of Health, Social Services and Public Safety, 2012; Loukidou et al., Reference Loukidou, Mastroyannakis, Power, Thornicroft, Craig and Bouras2013; Nakanishi and Endo, Reference Nakanishi and Endo2017; Substance Abuse and Mental Health Services Administration, 2017) with combination of qualitative (semi-structured interviews, focus group discussions, desk review, stakeholder consultations) methods and quantitative surveys or statistical analysis. With the exception of one, all studies used surveys that were specially designed ad hoc self-administered questionnaires, with the WHO-AIMS Instrument and Survey Checklist being the only standardised tool used (Mwanza et al., Reference Mwanza, Sikwese, Mwanza, Mayeya, Lund, Bird, Drew, Faydi, Funk and Green2008). Study participants varied across studies and included, depending on the context, users and families, service providers, traditional healers, agencies and organisations involved in the implementation, government and international policymakers.

Five studies used a single method such as survey or semi-structured interviews with key stakeholders, while the rest employed more than one method with a combination of following methods: questionnaires, qualitative interviews and focus group discussions, semi-structured discussions, literature and document review, and quantitative trends for suicide data. Only nine studies provided details on the content of their tools, of which two provided their interview and focus group guide templates in the Supplementary Materials (Doku et al., Reference Doku, Ofori-Atta, Akpalu, Read, Osei, Ae-Ngibise, Awenva, Lund, Flisher, Petersen, Bhana, Bird, Drew, Faydi, Funk, Green and Omar2008; Draper et al., Reference Draper, Lund, Kleintjes, Funk, Omar and Flisher2009).

Process of policy implementation

Overview

Overall, 27 studies assessed the process of policy implementation. Of these, 16 assessed only the implementation process; seven studies provided an assessment of the implementation process combined with an evaluation of the progress or impact of implementation and four studies assessed all three (progress, process, impact). Twenty-three were single-country studies and four evaluated more than one country. Nearly half (n = 13) of the studies were conducted in HICs; five studies by UMICs; four by LMICs and LICs each and one multinational study assessed both UMIC and LIC.

Policy areas

Of all 27 studies, 22 assessed mental health policies where four studies assessed suicide prevention strategies (Reid Howie Associates, 2006; Mackenzie et al., Reference Mackenzie, Blamey, Halliday, Maxwell, McCollam, McDaid, MacLean, Woodhouse and Platt2007; Department of Health, Social Services and Public Safety, 2012; Substance Abuse and Mental Health Services Administration, 2017) and one study focused on national dementia prevention policies (Boeree et al., Reference Boeree, Zoller and Huijsman2021). Most of the mental health policies were national development strategies that aimed at decentralisation and deinstitutionalisation of mental healthcare and establishing community-based systems. Other studies investigated policies on mental health system funding change (Aviram and Azary-Viesel, Reference Aviram and Azary-Viesel2018) or looked at the implementation process of multiple consecutive deinstitutionalisation reforms (from 1950 onwards) (Jones, Reference Jones2000). One study focused on a new policy framework that would expand the understanding of mental ill health within the country’s welfare system and investigated how the process was dealt with by implementing agencies (Fjellfeldt, Reference Fjellfeldt2020). Two studies specifically looked at policies centred around the service development process (Stanley-Clarke et al., Reference Stanley-Clarke, Sanders and Munford2014) and the introduction of a new model of care (Park et al., Reference Park, Lencucha, Mattingly, Zafran and Kirmayer2015).

Aspects measured

Most studies assessed challenges or barriers and/or facilitating factors on policy implementation (n = 24). Specific implementation determinants measured included context-dependant features of the policy implementation process such as public and political level of support for mental health reform and pace of its implementation (Ryan et al., Reference Ryan, Nwefoh, Aguocha, Ode, Okpoju, Ocheche, Woyengikuro, Abdulmalik and Eaton2020); ethical tensions arising during policy implementation (Park et al., Reference Park, Lencucha, Mattingly, Zafran and Kirmayer2015); key informants’ thoughts and feelings associated with the implementation process of a new policy framework (Fjellfeldt, Reference Fjellfeldt2020) and policy levers through which the mental health system reform was to be implemented (Grace et al., Reference Grace, Meurk, Head, Hall, Carstensen, Harris and Whiteford2015). Two studies used a theoretical framework on drivers and constraints that affect policy development and implementation as a conceptual background to the methodology to guide evaluation. Boeree et al. (Reference Boeree, Zoller and Huijsman2021) described the primary drivers for implementation as follows: 1) planning and infrastructure; 2) individual, group, organisational and systemic factors, as well as contextual factors; 3) the underlying theory and process of change involving all partners and 4) performance measures and evaluation Doku et al. (Reference Doku, Ofori-Atta, Akpalu, Read, Osei, Ae-Ngibise, Awenva, Lund, Flisher, Petersen, Bhana, Bird, Drew, Faydi, Funk, Green and Omar2008) described three major constraints for effective implementation: 1) lack of strategic planning; 2) inappropriate health system to support the policy and 3) lack of support or resistance to implementation, partly due to stigma associated with mental illness.

Other evaluation frameworks were more comprehensive and included various elements such as context, content and process or stream (such as in Kingdon’s conceptualisation of policy or Walt’s analytical framework), and the various actors involved in mental health policy (De Vries and Klazinga, Reference De Vries and Klazinga2006; Omar et al., Reference Omar, Green, Bird, Mirzoev, Flisher, Kigozi, Lund, Mwanza and Ofori-Atta2010; Grace et al., Reference Grace, Meurk, Head, Hall, Carstensen, Harris and Whiteford2015).

One study used a manualised case study methodology to organise and integrate data from various sources across domains of interest. The collection of data was accompanied by the overarching research questions: ‘Is this programme working? Why or why not?’ along with a description of strengths, weaknesses, opportunities and threats (Ryan et al., Reference Ryan, Nwefoh, Aguocha, Ode, Okpoju, Ocheche, Woyengikuro, Abdulmalik and Eaton2020). Similarly, in a study on ethical tensions that may arise during policy implementation, a specific ethical framework with three analytic levels: i) person-focused, ii) event-focused and iii) discursive practices was developed to capture the experiences of participants involved in the programme (Park et al., Reference Park, Lencucha, Mattingly, Zafran and Kirmayer2015).

Methods and tools

Studies used qualitative (n = 22) or mixed-methods (n = 5) techniques for investigation of implementation process. Every study in this review employed at least some form of qualitative methods: key informant interview, focus group discussion, other communications with stakeholders (discussions, meetings, forums, Theory of Change workshop), ethnography or observation or documents review. One study used a free listing technique with broad open-ended questions to elicit a comprehensive list of implementation barriers (Abdulmalik et al., Reference Abdulmalik, Kola and Gureje2016). Quantitative tools included self-reported surveys (e.g., to measure knowledge by health workers about the content of mental health policy). A quantitative survey provided primary healthcare staff with a list of challenges to integrate mental health into primary healthcare, which included dichotomous answers (yes/no or agree/disagree) on potential challenges/barriers that were reported during qualitative interviews (Abdulmalik et al., Reference Abdulmalik, Kola and Gureje2016). Other quantitative methods measured impact of implementation (see below for more details). Participants varied from study to study and included service users, service providers, healthcare managers, media representatives and policymakers at macro level.

Nine studies used single methods such as semi-structured qualitative interview (n = 7), document review (n = 1) or a workshop with stakeholders (n = 1), while the rest used more than one method. These were various combinations of qualitative interviews, focus group discussions, document reviews, meetings with stakeholders, observations and quantitative surveys. Only 10 studies provided rationale or details about the tools they employed of which three were available (Doku et al., Reference Doku, Ofori-Atta, Akpalu, Read, Osei, Ae-Ngibise, Awenva, Lund, Flisher, Petersen, Bhana, Bird, Drew, Faydi, Funk, Green and Omar2008; Draper et al., Reference Draper, Lund, Kleintjes, Funk, Omar and Flisher2009; Bikker et al., Reference Bikker, Lesmana and Tiliopoulos2020) (see Supplementary Material).

Impact

Overview

Of the 27 studies assessing the impact of policy implementation, 16 evaluated only impact, five evaluated impact together with progress, four evaluated impact together with process and two evaluated all three. Twenty-three studies were published in HICs and only four in LMICs. With the exception of two, all studies investigated single countries only. Eight studies applied qualitative methods (e.g., interviews, focus group discussions, document review), 11 studies used quantitative methods (e.g., survey) and eight studies employed a mixed-methods approach.

Policy areas

Most studies assessing the impact of policy implementation focused on general mental health policy (n = 22) where four studies assessed suicide prevention strategies and one study assessed dementia policies. Most mental health policies encompassed the goals of deinstitutionalisation with provision of more community services (n = 20), of which six studies focused on mental health reform in Italy (Lovell, Reference Lovell1986; Williams et al., Reference Williams, Salvia and Tansella1986; Palermo, Reference Palermo1991; Barbato, Reference Barbato1998; De Girolamo and Cozza, Reference de Girolamo and Cozza2000; Munizza et al., Reference Munizza, Gonella, Pinciaroli, Rucci, Picci and Tibaldi2011) and four studies investigated mental health policy in Greece (Madianos, Reference Madianos2002; Karastergiou et al., Reference Karastergiou, Mastrogianni, Georgiadou, Kotrotsios and Mauratziotou2005; Madianos and Christodoulou, Reference Madianos and Christodoulou2007; Loukidou et al., Reference Loukidou, Mastroyannakis, Power, Thornicroft, Craig and Bouras2013). Two other policies included a study where a mental health policy in the United Kingdom policy sought to introduce ‘tiered’ prioritisation of patients (Bindman et al., Reference Bindman, Beck, Glover, Thornicroft, Knapp, Leese and Szmukler1999) and a study in Australia introducing new ‘priority themes’ for mental healthcare development (Hickie and Groom, Reference Hickie and Groom2004). Other goals covered by policies included increase of public–private partnerships to deliver mental health services (Ryan et al., Reference Ryan, Nwefoh, Aguocha, Ode, Okpoju, Ocheche, Woyengikuro, Abdulmalik and Eaton2020), promote integrated primary mental healthcare (Petersen et al., Reference Petersen, Ssebunnya, Bhana and Baillie2011), improve quality of care (Whiteford et al., Reference Whiteford, Buckingham and Manderscheid2002; Hickie and Groom, Reference Hickie and Groom2004; Sharkey, Reference Sharkey2017; Winkler et al., Reference Winkler, Formánek, Mladá and Evans Lacko2021), ensure research evidence translation into practice (Hickie and Groom, Reference Hickie and Groom2004;Sharkey, Reference Sharkey2017; Winkler et al., Reference Winkler, Formánek, Mladá and Evans Lacko2021), raise awareness and reduce mental illness stigma (Sharkey, Reference Sharkey2017; Winkler et al., Reference Winkler, Formánek, Mladá and Evans Lacko2021), promote mental health and prevent mental disorders (Australian Health Ministers´ Advisory Council, 1997; Hickie and Groom, Reference Hickie and Groom2004) and protect consumer rights (Australian Health Ministers´ Advisory Council, 1997; Winkler et al., Reference Winkler, Formánek, Mladá and Evans Lacko2021).

Aspects measured

The identified studies mainly examined the effectiveness of policies and their appropriateness. Two studies measured appropriateness of policy implementation (Australian Health Ministers´ Advisory Council, 1997; Substance Abuse and Mental Health Services Administration, 2017), which was defined as whether the strategy’s goals and actions remain relevant and suitable to the implementation that is being carried out.

In terms of effectiveness, aspects measured varied substantially between the studies depending on the policy goals and planned deliverables. For example, deinstitutionalisation policies measured changes in psychiatric beds capacities in various settings (Williams et al., Reference Williams, Salvia and Tansella1986; De Girolamo and Cozza, Reference de Girolamo and Cozza2000; Vázquez-Barquero et al., Reference Vázquez-Barquero, García and Torres-González2001; Madianos, Reference Madianos2002; Whiteford et al., Reference Whiteford, Buckingham and Manderscheid2002; Madianos and Christodoulou, Reference Madianos and Christodoulou2007), number of referrals to community services (De Girolamo and Cozza, Reference de Girolamo and Cozza2000; Vázquez-Barquero et al., Reference Vázquez-Barquero, García and Torres-González2001; Sharkey, Reference Sharkey2017), number of deinstitutionalised patients (Loukidou et al., Reference Loukidou, Mastroyannakis, Power, Thornicroft, Craig and Bouras2013), ratio of psychiatric patients identified by a GP (Vázquez-Barquero et al., Reference Vázquez-Barquero, García and Torres-González2001), transinstitutionalisation of patients to other facilities (Lovell, Reference Lovell1986; Barbato, Reference Barbato1998), prevalence of homeless people with mental disorders and criminalisation of the mentally ill (Barbato, Reference Barbato1998), development of various decentralised services (Lovell, Reference Lovell1986; Williams et al., Reference Williams, Salvia and Tansella1986; De Girolamo and Cozza, Reference de Girolamo and Cozza2000; Madianos and Christodoulou, Reference Madianos and Christodoulou2007; Munizza et al., Reference Munizza, Gonella, Pinciaroli, Rucci, Picci and Tibaldi2011), epidemiological data on incidence and treated prevalence (Munizza et al., Reference Munizza, Gonella, Pinciaroli, Rucci, Picci and Tibaldi2011), change in suicide trends before and after the psychiatric reform (Williams et al., Reference Williams, Salvia and Tansella1986; Barbato, Reference Barbato1998; De Girolamo and Cozza, Reference de Girolamo and Cozza2000) and changes in the quality of care (De Girolamo and Cozza, Reference de Girolamo and Cozza2000; Rey et al., Reference Rey, Walter and Giuffrida2004), in psychiatric practice and in access to care (Rey et al., Reference Rey, Walter and Giuffrida2004). Other measures included change in service expenditures (Whiteford et al., Reference Whiteford, Buckingham and Manderscheid2002), involvement of service users and carers (Whiteford et al., Reference Whiteford, Buckingham and Manderscheid2002), clinical outcomes in hospital residents and community patients and quality of life of people with mental disorders (De Girolamo and Cozza, Reference de Girolamo and Cozza2000); change in public stigma (Vázquez-Barquero et al., Reference Vázquez-Barquero, García and Torres-González2001; Winkler et al., Reference Winkler, Formánek, Mladá and Evans Lacko2021) and change in number of published scientific articles (to measure the research potential) (Sharkey, Reference Sharkey2017). Some studies used subjective measures such as personal opinion from stakeholders about the policy implementation (Boeree et al., Reference Boeree, Zoller and Huijsman2021). Studies on specific programmes related to policy implementation measured the number of clients enrolled in a programme (Bindman et al., Reference Bindman, Beck, Glover, Thornicroft, Knapp, Leese and Szmukler1999; Ryan et al., Reference Ryan, Nwefoh, Aguocha, Ode, Okpoju, Ocheche, Woyengikuro, Abdulmalik and Eaton2020).

Studies on suicide prevention strategies measured incidence of suicide across prefectures in Japan where different prevention programmes were implemented (Nakanishi et al., Reference Nakanishi, Yamauchi and Takeshima2015) or the perceived level of support by bereaved families to assess effectiveness of support programmes and resources allocated on implementation of suicide prevention strategy in the Northern Ireland (Department of Health, Social Services and Public Safety, 2012).

An Australian study used a framework with four evaluation focus areas: rights of consumers and carers, mixed services, linkages between mental health services with other sectors and promotion and prevention. Each focus area contained a number of questions to be answered during the evaluation (Australian Health Ministers´ Advisory Council, 1997). Similarly, a Northern Ireland (UK) study, for each predefined evaluation question, defined an evaluation area as well as what to measure and recommended methods and tools for evaluation. For example, effectiveness and impact for individuals and families were defined as an evaluation area and for that particular area evidence base and families service use experiences were investigated by using a pre-developed survey, focus groups with families and reviewing available published evaluations of already completed initiatives (Department of Health, Social Services and Public Safety, 2012).

Methods and tools

Nine studies applied qualitative methods, 11 used quantitative and another seven used mixed-methods approach. Qualitative semi-structured interviews and document reviews were most commonly used with each method mentioned in six and seven studies, respectively. Other qualitative methods included focus group discussions, consultations with experts, meetings with officials and observation.

Among the quantitative methods, statistical analyses of epidemiological data (n = 7) were used most commonly. Other methods included survey questionnaire (n = 9) and using specific scales such as Community Attitudes towards Mental Illness (Taylor and Dear, Reference Taylor and Dear1981) and Reported and Intended Behaviour Scale (Evans-Lacko et al., Reference Evans-Lacko, Rose, Little, Flach, Rhydderch, Henderson and Thornicroft2011) to measure public stigma (Winkler et al., Reference Winkler, Formánek, Mladá and Evans Lacko2021). Quantitative surveys asked consumers and carers to rate their experience of healthcare (Hickie and Groom, Reference Hickie and Groom2004), to choose whether certain aspects of psychiatric practice had increased, remained the same or decreased (e.g., income, satisfaction, patients’ illness severity) or whether the perceived quality of care has improved or deteriorated (Rey et al., Reference Rey, Walter and Giuffrida2004).

Twelve studies used only one method while the rest employed a combination of the above-mentioned methods such as survey and statistical analysis, literature review and meeting with officials. Nine studies described the content of their tools, of which only three are available (see Supplementary Material).

Discussion

To our knowledge, this is the first study to systematically map and analyse methods and tools used to assess the implementation of MHPPs. We found no comprehensive, high-quality, peer-reviewed assessment of implementation of MHPPs as such. Given that MHPPs are important instruments to improve mental healthcare and well-being of populations, rigorous peer-reviewed assessment of their implementation is crucial so important lessons can be learned and mental health systems improved. Studies included in our review placed emphasis on the presentation of the results but lacked rigorous methodological description, which makes their tools and methods unclear. Less than half of the included studies provided details about the tools they used for data collection (Munizza et al., Reference Munizza, Gonella, Pinciaroli, Rucci, Picci and Tibaldi2011; Bikker et al., Reference Bikker, Lesmana and Tiliopoulos2020; Winkler et al., Reference Winkler, Formánek, Mladá and Evans Lacko2021). Only three studies pretested or piloted the tools. Very few publications provided a full description of their tools. For instance, although a substantial majority of studies employed interview and focus group guides and questionnaires that were specifically tailored to the evaluation purposes, they failed to provide samples of the questions they used. Most tools were not commonly standardised, which is likely due to the broad nature of MHPPs and the comprehensiveness of their specific policy area and the diversity of contexts in which they have been implemented.

Only three studies assessed all three categories of implementation (progress, process and impact) and they were all non-academic assessments of suicide prevention strategies. Similarly, assessments of entire MHPPs, as opposed to only certain parts of them, are rare. Clearly, a full and comprehensive assessment of an overarching policy like a mental health reform might be a lengthy (impact can be measured after decades of reform) and resource-demanding process. In contrast, we found that studies focused on specific evaluation questions related to MHPPs implementation (e.g., challenges associated with the reform in various contexts or opinions about the implementation progress; development of public stigma; changes in suicide rates, etc.) were mostly published in academic journals. Such assessment with the primary focus on only one or several aspects of MHPPs provided that rigorousness and transparency of reported methods and results are ensured certainly is a valid alternative strategy. However, these smaller assessments of an MHPP have to be put together into comprehensive reports of MHHPs implementation and made accessible to readers. In any case, evaluation strategy should be an integral part of the MHPPs.

Given the broad nature and complexity of MHPPs, it is likely that there could be a publication bias where studies with narrower research questions get published in academic journals, whereas extensive assessments and evaluations might have been published only as project reports or policy papers. We identified seven such reports, which were extensive national evaluation reports of countries’ MHPPs or suicide prevention strategies (n = 5), one Master’s dissertation on barriers to integration of mental health into primary care in Nigeria (Abdulmalik et al., Reference Abdulmalik, Kola and Gureje2016) and one project report on suicide prevention in prisons in Scotland, UK (Reid Howie Associates, 2006). Usually, the format of such studies allows for situation analysis and a more detailed description of methodology to be included. However, if made available online, such reports tend to be replaced or become inaccessible over years, may not be identifiable through traditional electronic database searches and are usually not peer-reviewed.

We used a broad definition to assess policy implementation focusing on three categories: progress, process and impact. Studies assessing the progress of implementation usually collected data through qualitative and quantitative questionnaires enquiring about progress or level of implementation against policy targets or goals. Findings of evaluations demonstrated that implementation of MHPPs in terms of target achievement or types of programmes adopted was most often partial. For example, in Northern Ireland, UK, only about a fifth of actions were fully progressing to plan, while the rest were in moderate or limited progress. Similarly, in Australia, a study showed that not even two-thirds of activities were measured, of which 42% were fully implemented while 20% were implemented partially. Studies in our review show that some activities are being implemented more effectively than others. In Greece, even though implementation of many activities of national mental health reform was successful, the rate of implementation substantially varied between rural and urban areas. In Japan, where authorities were left to choose the activities for suicide prevention on their own, most preferred to implement ‘public awareness campaign’ and ‘training of community service providers’ over ‘face-to-face counselling’ or introduction of ‘trauma-informed policies and practices’. Such results require a further deeper investigation into the reasons for and effects of certain patterns of implementation. For example, clearly defined one-off projects, activities with specified funding attached to them and having specific agencies responsible for their implementation increase the likelihood of full implementation, whereas activities that are less tangible and thus harder to define, and without a lead agency can be more difficult to implement (Sheehan et al., Reference Sheehan, Griffiths, Rickwood and Carron-Arthur2015).

Why certain activities were implemented over others can be understood through evaluation of implementation process in identifying challenges and facilitators as well as views of stakeholders on the process. We found that qualitative research methods, such as Theory of Change workshops, stakeholder meetings, qualitative interviews and focus group discussions, were frequently employed to understand barriers and facilitators of MHPPs’ implementation. Most cited barriers to implementation were poor dissemination of implementation guidelines, inadequate resources (e.g., financial, human or infrastructural) to support the reform process and resistance to changes. Some studies indicated low prioritisation of mental health and stigma as barriers; others reported weak management and poor intersectoral collaboration, difficult political context and the complex nature of interventions as factors hindering policy implementation. In LMICs, these challenges are more and greater than in HICs, especially in terms of funding, human resources and administration (Zhou et al., Reference Zhou, Yu, Yang, Chen and Xiao2018). In contrast, clear understanding of roles and responsibilities for implementation and ensuring coordination between different stakeholders were identified as facilitating factors.

Context is crucial for appropriate assessment and understanding of the implementation process. Studies largely adapted their evaluation questions to the features of the political, social or economic environment (Petersen et al., Reference Petersen, Ssebunnya, Bhana and Baillie2011; Ryan et al., Reference Ryan, Nwefoh, Aguocha, Ode, Okpoju, Ocheche, Woyengikuro, Abdulmalik and Eaton2020). In post-conflict areas like Bosnia and Herzegovina, foreign influence was identified as a central theme in implementation of mental health reform, which raised questions on sustainability. In resource-constrained contexts, prioritisation of mental healthcare can be challenging, especially when burden of physical health conditions is high, which hindered implementation of MHPPs (Doku et al., Reference Doku, Ofori-Atta, Akpalu, Read, Osei, Ae-Ngibise, Awenva, Lund, Flisher, Petersen, Bhana, Bird, Drew, Faydi, Funk, Green and Omar2008; Draper et al., Reference Draper, Lund, Kleintjes, Funk, Omar and Flisher2009; Ssebunnya et al., Reference Ssebunnya, Kigozi, Kizza and Ndyanabangi2010). In decentralised healthcare systems, such as in South Africa, translation of national policies into strategic plans appropriate to the provincial or district level contexts seems to be a key factor for ensuring their successful implementation (Draper et al., Reference Draper, Lund, Kleintjes, Funk, Omar and Flisher2009).

Assessment of the impact of implementation was largely performed via both quantitative methods, most often pre- and post-policy reform, and qualitative methods, most often by asking stakeholders about their perception on changes brought by MHPPs.

The relatively poor assessment of implementation of MHPPs is in contrast to the more advanced tools used to monitor and guide the implementation of policies at all levels in other public health areas such as in tobacco (WHO, 2013; Cox et al., Reference Cox, Lutz, Webb, Sahal-Estime, Small and Trivedi2014) and alcohol control (Rekve, Reference Rekve2011), breastfeeding promotion (WHO, 2003; The International Baby Food Action Network Asia, 2008; WHO, 2013) or family planning and reproductive health (Bhuyan et al., Reference Bhuyan, Jorgensen and Sharma2010). For instance, the Policy Implementation Assessment Tool was developed to guide an assessment of national family planning and reproductive health policy implementation. This tool includes instructions on policy assessment at various levels from stakeholder mapping to organising and analysing data (Bhuyan et al., Reference Bhuyan, Jorgensen and Sharma2010). It enables to gather information via multifaceted processes and in a systematic, user-friendly manner. The tool consists of an interview guide that is divided into eight sections that focus on assessing context, process of implementation and appropriateness of policy strategies in relation to its objectives. This tool could be potentially adjusted for mental health policies.

Limitations

We recognise that our search strategy was not able to capture all relevant studies, particularly those that focused on the impact of policy implementation. Potentially valuable information could have been missed when studies are published in project reports on certain areas of policy or published in academic journals without mentioning its relation to a specific policy.

There is a lack of information on the tools used in most studies included in our review, as such we were unable to assess the quality of evaluation methods. Instead, we provided information on whether studies described their tools sufficiently and whether they were interpretable.

Due to the broad scope of this review, we were unable to compare tools across contexts and applications. Further research is necessary to determine which tools are optimal for assessing the implementation of MHPPs and to develop recommendations and guidance on evaluation of MHPPs.

Conclusions and recommendations

Our review has highlighted substantial knowledge gaps in assessing the implementation of MHPPs. Our findings should contribute to policy dialogues on the development, implementation and assessment of implementation of national mental health strategies. Efforts should be made to consolidate available methods and tools into clear methodologies that would address various stages and objectives of implementation taking into consideration a variety of possible policy goals. Such a consolidated methodology might result in a checklist that would mirror each objective of MHPPs and that would allow for various contexts to be taken into account as well as for experiences and lessons from implementation and evaluation to be shared.

Based on our review, we recommend the following:

  1. 1. Strategy or plan of evaluation of implementation needs to be an integral part of MHPPs and it needs to contain responsibilities and funding.

  2. 2. Future evaluations of MHPPs implementation need to be more transparent in reporting details, especially on tools and methodologies used and, where possible, make them accessible to readers.

  3. 3. Since the resources are constrained in all settings, partnerships need to be built to ensure high-quality evaluations. Such partnerships might include universities, research institutes and other organisations, both nationally and internationally.

  4. 4. Evaluations can be fragmented into smaller studies focused on specific aspects of MHPPs; however, findings from these studies should be put together into complex evaluation reports of MHPPs implementation. Both smaller studies and complex evaluation reports shall be published in peer-reviewed journals to ensure their accessibility and impact.

  5. 5. More research needs to be done to understand the current implementation of MHPPs so the lessons made could be learned.

Open peer review

To view the open peer review materials for this article, please visit http://doi.org/10.1017/gmh.2023.3.

Supplementary material

To view supplementary material for this article, please visit https://doi.org/10.1017/gmh.2023.3.

Author contributions

A.A.A., P.W. and Y.Y. initiated, planed and designed the study. A.A.A. and H.T. conducted the literature review and prepared the first draft of the paper. A.K. and Z.G. participated in designing the study and contributed to writing and proofreading the draft. P.W., Y.Y. and L.L. supervised the whole study and critically revised the manuscript. M.R., C.R., A.M.T.I., and J.M. critically revised the manuscript and contributed to the final draft of the paper.

Financial support

This work was supported by the World Health Organisation Regional Office for Europe.

Competing interest

The authors declare none.

References

Abdulmalik, JO (2015) Barriers preventing the successful integration of mental health services into primary health care in Nigeria: A mixed methods approach.Google Scholar
Abdulmalik, J, Kola, L and Gureje, O (2016) Mental health system governance in Nigeria: Challenges, opportunities and strategies for improvement. Global Mental Health 3, e9.CrossRefGoogle ScholarPubMed
Australian Health Ministers Advisory Council (1997) Evaluation of the National Mental Health Strategy: Final report.Google Scholar
Aviram, U and Azary-Viesel, S (2018) Mental health reform in Israel: Challenge and opportunity. Israel Journal of Psychiatry 55, 5573.Google Scholar
Awenva, A, Read, U, Ofori-Attah, A, Doku, V, Akpalu, B, Osei, A and Flisher, A (2010) From mental health policy development in Ghana to implementation: What are the barriers? African Journal of Psychiatry 13, 184191.Google ScholarPubMed
Barbato, A (1998) Psychiatry in transition: Outcomes of mental health policy shift in Italy. Australian and New Zealand Journal of Psychiatry 32, 673679.CrossRefGoogle ScholarPubMed
Bhuyan, A, Jorgensen, A and Sharma, S (2010) Taking the pulse of policy: The policy implementation assessment tool. Health Policy Initiative 50. www.hrhresourcecenter.org/node/3930.htmlGoogle Scholar
Bikker, A, Lesmana, C and Tiliopoulos, N (2020) The Indonesian mental health act: Psychiatrists’ views on the act and its implementation. Health Policy and Planning 36, 196204.CrossRefGoogle Scholar
Bindman, J, Beck, A, Glover, G, Thornicroft, G, Knapp, M, Leese, M and Szmukler, G (1999) Evaluating mental health policy in England. British Journal of Psychiatry 175, 327330.CrossRefGoogle ScholarPubMed
Boeree, N, Zoller, C and Huijsman, R (2021) The implementation of national dementia plans: A multiple-case study on Denmark, Germany, and Italy. International Journal of Environmental Research and Public Health 18, 10220.CrossRefGoogle Scholar
Campion, J and Knapp, M (2018) The economic case for improved coverage of public mental health interventions. The Lancet Psychiatry 5, 103105.CrossRefGoogle ScholarPubMed
Cox, AB, Lutz, D, Webb, M, Sahal-Estime, M, Small, R and Trivedi, V (2014) Development Planning and Tobacco Control: Integrating the Who Framework Convention on Tobacco Control into UN and National Development Planning Instruments. New York: United Nations Development Programme.Google Scholar
de Girolamo, G and Cozza, M (2000) The Italian psychiatric reform: A 20-year perspective. International Journal of Law and Psychiatry 23, 197214.CrossRefGoogle ScholarPubMed
De Vries, AK and Klazinga, NS (2006) Mental health reform in post-conflict areas: a policy analysis based on experiences in Bosnia Herzegovina and Kosovo. European Journal of Public Health 16, 247252.CrossRefGoogle Scholar
Department of Health, Social Services and Public Safety (2012) Evaluation of the implementation of the Northern Ireland Protect Life Suicide Prevention Strategy and Action Plan 2006–2011.Google Scholar
Dlouhy, M (2014) Mental health policy in Eastern Europe: A comparative analysis of seven mental health systems. BMC Health Services Research 14, 42.CrossRefGoogle ScholarPubMed
Doku, V, Ofori-Atta, A, Akpalu, B, Read, U, Osei, A, Ae-Ngibise, K, Awenva, D, Lund, C, Flisher, AJ, Petersen, I, Bhana, A, Bird, P, Drew, N, Faydi, E, Funk, M, Green, A and Omar, M (2008) Phase 1. Country report: A situation analysis of mental health policy development and implementation in Ghana.Google Scholar
Draper, C, Lund, C, Kleintjes, S, Funk, M, Omar, M and Flisher, A (2009) Mental health policy in South Africa: Development process and content. Health Policy and Planning 24, 342356.CrossRefGoogle ScholarPubMed
Evans-Lacko, S, Rose, D, Little, K, Flach, C, Rhydderch, D, Henderson, C and Thornicroft, G (2011) Development and psychometric properties of the reported and intended behaviour scale (RIBS): A stigma-related behaviour measure. Epidemiology and Psychiatric Sciences 20, 263271.CrossRefGoogle ScholarPubMed
Fjellfeldt, M (2020) One new policy: A variety of applications—The implementation processes of a new mental health policy in Sweden. Social Policy & Administration 54, 733748.CrossRefGoogle Scholar
Grace, F, Meurk, C, Head, B, Hall, W, Carstensen, G, Harris, M and Whiteford, H (2015) An analysis of policy levers used to implement mental health reform in Australia 1992-2012. BMC Health Services Research 15, 479.CrossRefGoogle ScholarPubMed
Hickie, I and Groom, G (2004) Surveying perceptions of the progress of national mental health reform. Australasian Psychiatry 12, 123125.CrossRefGoogle ScholarPubMed
International Baby Food Action Network Asia (2008) World Breastfeeding Trends Initiative (WBTi). Delhi: The International Baby Food Action Network Asia.Google Scholar
Jones, J (2000) Mental health care reforms in Britain and Italy since 1950: A cross-national comparative study. Health & Place 6, 171187.CrossRefGoogle ScholarPubMed
Karastergiou, A, Mastrogianni, A, Georgiadou, E, Kotrotsios, S and Mauratziotou, K (2005) The reform of the Greek mental health services. Journal of Mental Health 14, 197203.CrossRefGoogle Scholar
Loukidou, E, Mastroyannakis, A, Power, T, Thornicroft, G, Craig, T and Bouras, N (2013) Evaluation of Greek psychiatric reforms: Methodological issues. International Journal of Mental Health Systems 7, 11.CrossRefGoogle ScholarPubMed
Lovell, A (1986) The paradoxes of reform: Reevaluating Italy’s mental health law of 1978. Psychiatric Services 37, 802808.CrossRefGoogle ScholarPubMed
Mackenzie, M, Blamey, A, Halliday, E, Maxwell, M, McCollam, A, McDaid, D, MacLean, J, Woodhouse, A and Platt, S (2007) Measuring the tail of the dog that doesn’t bark in the night: The case of the national evaluation of choose life (the national strategy and action plan to prevent suicide in Scotland). BMC Public Health 7, 146.CrossRefGoogle ScholarPubMed
Madianos, M (2002) Deinstitutionalization and the closure of public mental hospitals. International Journal of Mental Health 31, 6675.CrossRefGoogle Scholar
Madianos, M and Christodoulou, G (2007) Reform of the mental healthcare system in Greece, 1984–2006. International Psychiatry 4, 1619.CrossRefGoogle ScholarPubMed
Marais, DL and Petersen, I (2015) Health system governance to support integrated mental health care in South Africa: Challenges and opportunities. International Journal of Mental Health Systems 9, 121.CrossRefGoogle ScholarPubMed
Moher, D, Liberati, A, Tetzlaff, J and Altman, D (2010) Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. International Journal of Surgery 8, 336341.CrossRefGoogle ScholarPubMed
Munizza, C, Gonella, R, Pinciaroli, L, Rucci, P, Picci, R and Tibaldi, G (2011) CMHC adherence to national mental health plan standards in Italy: A survey 30 years after national reform law. Psychiatric Services 62, 10901093.CrossRefGoogle ScholarPubMed
Mwanza, J, Sikwese, A, Mwanza, B, Mayeya, J, Lund, C, Bird, P, Drew, N, Faydi, E, Funk, M and Green, A (2008) Phase 1 country report: Mental health policy development and implementation in Zambia: A situation analysis.Google Scholar
Nakanishi, M and Endo, K (2017) National suicide prevention, local mental health resources, and suicide rates in Japan. Crisis 38, 384392.CrossRefGoogle ScholarPubMed
Nakanishi, M, Yamauchi, T and Takeshima, T (2015) National strategy for suicide prevention in Japan: Impact of a national fund on progress of developing systems for suicide prevention and implementing initiatives among local authorities. Psychiatry and Clinical Neurosciences 69, 5564.CrossRefGoogle ScholarPubMed
Omar, M, Green, A, Bird, P, Mirzoev, T, Flisher, A, Kigozi, F, Lund, C, Mwanza, J and Ofori-Atta, A (2010) Mental health policy process: A comparative study of Ghana, South Africa, Uganda and Zambia. International Journal of Mental Health Systems 4, 24.CrossRefGoogle ScholarPubMed
Page, M, Moher, D, Bossuyt, P, Boutron, I, Hoffmann, T, Mulrow, C, Shamseer, L, Tetzlaff, J, Akl, E, Brennan, S, Chou, R, Glanville, J, Grimshaw, J, Hróbjartsson, A, Lalu, M, Li, T, Loder, E, Mayo-Wilson, E, McDonald, S, McGuinness, L, Stewart, L, Thomas, J, Tricco, A, Welch, V, Whiting, P and McKenzie, J (2021 ) PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. BMJ 372, n71.CrossRefGoogle ScholarPubMed
Palermo, GB (1991) The 1978 Italian mental health law–a personal evaluation: A review. Journal of the Royal Society of Medicine 84, 99102.CrossRefGoogle ScholarPubMed
Park, M, Lencucha, R, Mattingly, C, Zafran, H and Kirmayer, L (2015) A qualitative study on the ethics of transforming care: Examining the development and implementation of Canada’s first mental health strategy. Implementation Science 10, 19.CrossRefGoogle ScholarPubMed
Petersen, I, Baillie, K, Bhana, A and Mental Health and Poverty Research Programme Consortium (2012) Understanding the benefits and challenges of community engagement in the development of community mental health services for common mental disorders: Lessons from a case study in a rural south African subdistrict site. Transcultural Psychiatry 49, 418437.CrossRefGoogle Scholar
Petersen, I, Ssebunnya, J, Bhana, A and Baillie, K (2011) Lessons from case studies of integrating mental health into primary health care in South Africa and Uganda. International Journal of Mental Health Systems 5, 8.CrossRefGoogle ScholarPubMed
Phulkerd, S, Lawrence, M, Vandevijvere, S, Sacks, G, Worsley, A and Tangcharoensathien, V (2015) A review of methods and tools to assess the implementation of government policies to create healthy food environments for preventing obesity and diet-related non-communicable diseases. Implementation Science 11, 113.CrossRefGoogle Scholar
Reid Howie Associates (2006) SPS “CHOOSE LIFE” INITIATIVE REVIEW AND OVERALL EVALUATION.Google Scholar
Rekve, D (2011) Implementation of the Global Strategy to Reduce the Harmful use of Alcohol. In Committee on National Alcohol Policy and Action 8th Meeting.Google Scholar
Rey, J, Walter, G and Giuffrida, M (2004) Policy, structural change and quality of psychiatric Services in Australia: The views of psychiatrists. Australasian Psychiatry 12, 118122.CrossRefGoogle ScholarPubMed
Ryan, G, Nwefoh, E, Aguocha, C, Ode, P, Okpoju, S, Ocheche, P, Woyengikuro, A, Abdulmalik, J and Eaton, J (2020) Partnership for the implementation of mental health policy in Nigeria: A case study of the comprehensive community mental health programme in Benue state. International Journal of Mental Health Systems 14, 10.CrossRefGoogle ScholarPubMed
Saraceno, B, van Ommeren, M, Batniji, R, Cohen, A, Gureje, O, Mahoney, J, Sridhar, D and Underhill, C (2007) Barriers to improvement of mental health services in low-income and middle-income countries. The Lancet 370, 11641174.CrossRefGoogle ScholarPubMed
Sharkey, T (2017) Mental health strategy and impact evaluation in Qatar. BJPsych International 14, 1821.CrossRefGoogle ScholarPubMed
Sheehan, J, Griffiths, K, Rickwood, D and Carron-Arthur, B (2015) Evaluating the implementation of “managing the risk of suicide: A suicide prevention strategy for the ACT 2009–2014”. Crisis 36, 412.CrossRefGoogle Scholar
Ssebunnya, J, Kigozi, F, Kizza, D and Ndyanabangi, S (2010) Integration of mental health into primary health care in a rural district in Uganda. African Journal of Psychiatry 13, 128131.CrossRefGoogle Scholar
Stanley-Clarke, N, Sanders, J and Munford, R (2014) The role of government policy in service development in a New Zealand statutory mental health service: Implications for policy planning and development. Australasian Psychiatry 22, 557559.CrossRefGoogle Scholar
Substance Abuse and Mental Health Services Administration (2017) National Strategy for Suicide Prevention Implementation Assessment Report.Google Scholar
Taylor, SM and Dear, MJ (1981) Scaling community attitudes toward the mentally ill. Schizophrenia Bulletin 7, 225240.CrossRefGoogle ScholarPubMed
Toyoma, M, Castillo, H, Galea, JT, Brandt, LR, Mendoza, M, Herrera, V, Mitrani, M, Cutipé, Y, Cavero, V and Diez-Canseco, F (2017) Peruvian mental health reform: A framework for scaling-up mental health services. International Journal of Health Policy and Management 6, 501.CrossRefGoogle Scholar
Vázquez-Barquero, J, García, J and Torres-González, F (2001) Spanish psychiatric reform: What can be learned from two decades of experience? Acta Psychiatrica Scandinavica 104, 8995.CrossRefGoogle Scholar
Whiteford, H, Buckingham, B and Manderscheid, R (2002) Australia’s national mental health strategy. British Journal of Psychiatry 180, 210215.CrossRefGoogle ScholarPubMed
WHO (2001) The World Health Report 2001: Mental Health: New Understanding, New Hope. Geneva: World Health OrganizationGoogle Scholar
WHO (2003) Infant and Young Child Feeding: A Tool for Assessing National Practices, Policies and Programmes. Geneva: World Health Organization.Google Scholar
WHO (2013) Country Implementation of the International Code of Marketing of Breast-Milk Substitutes: Status Report 2011. Geneva: World Health Organization.Google Scholar
WHO (2013) WHO Report on the Global Tobacco Epidemic, 2013: Enforcing Bans on Tobacco Advertising, Promotion and Sponsorship. Geneva: World Health Organization.Google Scholar
WHO (2015) The European Mental Health Action Plan 2013–2020. Geneva: World Health Organization.Google Scholar
WHO (2018) Mental Health Atlas 2017. Geneva: World Health Organization.Google Scholar
Williams, P, Salvia, D and Tansella, M (1986) Suicide, psychiatric reform, and the provision of psychiatric services in Italy. Social Psychiatry 21, 8995.CrossRefGoogle ScholarPubMed
Winkler, P, Formánek, T, Mladá, K and Evans Lacko, S (2021) Development of public stigma toward people with mental health problems in Czechia 2013–2019. European Psychiatry 64, 17.CrossRefGoogle ScholarPubMed
Zhou, W, Yu, Y, Yang, M, Chen, L and Xiao, S (2018) Policy development and challenges of global mental health: A systematic review of published studies of national-level mental health policies. BMC Psychiatry 18, 19.CrossRefGoogle ScholarPubMed
Figure 0

Figure 1. Screening and selection of articles.

Figure 1

Table 1. Characteristics of included studies

Figure 2

Table 2. Summary of identified studies assessing the progress of implementation

Figure 3

Table 3. Summary of identified studies assessing the process of implementation

Figure 4

Table 4. Summary of identified studies assessing the impact of mental health policies and plans

Supplementary material: File

Aliev et al. supplementary material

Appendix 1

Download Aliev et al. supplementary material(File)
File 12.1 KB

Author comment: Methods and tools to assess implementation of mental health policies and plans: A systematic review — R0/PR1

Comments

No accompanying comment.

Review: Methods and tools to assess implementation of mental health policies and plans: A systematic review — R0/PR2

Conflict of interest statement

Reviewer declares none.

Comments

Comments to Author: A very good attempt in a difficult area.

The obvious obstacles in applying seemingly objective criteria to subjective issues is clear here.This is further complicated by both the nature of reporting and that of chronicling change. If we step further back, the way we think of the 'desirability' of the the change that we want, and a discussion around that, as also the relative importance to the three indices of progress, process and impact would help.

The inclusion of grey literature, and the fact that it broadens scope and changes the lenses of 'rigour' may need to be highlighted somewhat, as also the the nature of that literature.

So, i would think that if these questions were to be addressed, this piece would be benefited.

The essay actually raises more questions than it answers, which to my thinking is a characteristic of all good research.

Review: Methods and tools to assess implementation of mental health policies and plans: A systematic review — R0/PR3

Conflict of interest statement

Reviewer declares none.

Comments

Comments to Author: This paper sets out to examine mental health policy and programme implementation and impact as captured by published research to map and analyse tools and methodologies used to assess the extent and process of implementing national or regional MHPPs. The authors suggest that such “a mapping review will inform policy implementation and evaluation in the WHO European Region” (p. 3) of the World Health Organisation (WHO) Global and European Mental Health Action Plans in 2013 since many countries introduced new national MHPPs.

The authors shortlisted through a systematic process, 48 papers for the review of which the majority of the papers are from the European region. The paper provides information on various parameters used for assessing the papers. The themes cover policy, programme and impact aspects of mental health by organising the papers as per the areas covered, aspects measured and research methods and tools used. The authors highlight information gaps in the papers that they have studied and conclude the lack of tools available for monitoring and guiding the implementation of MHPPs.

While what the authors have arrived at is predictable, the paper would add much value if the authors could go beyond the surface and dig deep into these papers that represent major parts of the world to understand what the research is throwing light on.

The field of mental health covers a spectrum of subjects and different countries prioritize through their policies and programmes different aspects of mental health. Not having measurable indicators for the objectives set out in the policy could be a shortcoming in the country’s policy and not developing robust indicators or analytical frameworks to measure policy and programme outcomes could be a matter of poorly conceived research by the researcher. Both perhaps exist in this field. It is not clear through this paper, where the problem lies in most papers.

The paper also does not provide any understanding of the varying country and policy contexts as covered by the papers. The authors do not throw light on the policy and programme context of a particular country and what the published paper manages to cover or leaves out. In other words, the papers that were reviewed were coded on various parameters that the authors have developed without necessarily capturing the context of the country within the parameters.

The paper refers to WHO Action plans 2013 and the implementation of MHPPs across countries, but nowhere the authors mention the time frame used as inclusion and exclusion criteria for the literature search. The summaries provided at the end in Tables 2, 3 and 4 have reference to papers earlier to 2013 with one dated 1986.

The authors may refer to the action plan released by WHO in October 2021 with indicative indicators for measuring the policy and implementation issues. https://cdn.who.int/media/docs/default-source/campaigns-and-initiatives/world-mental-health-day/2021/mental_health_action_plan_flyer_member_states.pdf?sfvrsn=b420b6f1_7&download=true

The paper needs thorough reworking to go beyond compiling the information across 48 papers. The paper has to throw light not just on the deficiencies but what is the knowledge that published research is directing us to understand and how do these represent diverse economies and contexts.

Recommendation: Methods and tools to assess implementation of mental health policies and plans: A systematic review — R0/PR4

Comments

Comments to Author: Needless to say, it is imperative that evidence based MHPPs are discussed in scholarly literature. More importantly, discussions around the efficacy of their implementation and the adequacy of appropriate tools to assess the same, assume tremendous significance. This article therefore sets for itself a scope and framework, and to that end, delivers (using a systematic review), some results that need greater reflection. The article argues that such tools and instruments that assess the impact of policies on implementation are inadequate, which to me is a significant result and the authors perhaps need to stay with it a bit longer. While one reviewer has suggested minor revisions, the second has rejected the article. While I feel, the points that the the second reviewer has raised need due attention, some modifications to this article and a somewhat , ' deeper dive' ( as the reviewer says 'beneath the surface') may be useful. Most policies set the tone for what one may aspire to change ( let's say in suicide prevention or substance use or service usage, as the authors have indicated) ; but the socio- political contexts and much else need to come together for effective translation of a policy into something that impacts lives in a positive way on the ground. I recommend that some of these challenges are better highlighted and possible brief commentaries on the same discussed using an analytical lens. In parallel, themes that jump out in the papers reviewed may also be discussed in a more engaging manner in the results section. With these modifications, and perhaps some editorial support and structuring , the article may be used to highlight the lacunae and gaps and as a consequence present the complexity and messiness associated with policy making more emphatically.

Decision: Methods and tools to assess implementation of mental health policies and plans: A systematic review — R0/PR5

Comments

No accompanying comment.

Decision: Methods and tools to assess implementation of mental health policies and plans: A systematic review — R0/PR6

Comments

No accompanying comment.

Author comment: Methods and tools to assess implementation of mental health policies and plans: A systematic review — R1/PR7

Comments

No accompanying comment.

Review: Methods and tools to assess implementation of mental health policies and plans: A systematic review — R1/PR8

Conflict of interest statement

Reviewer declares none.

Comments

Comments to Author: Thank you for the changes.

The paper reads much more coherently now.

It is an important paper because it helps us understand both the limitations and the need for reviewing publication.

It can also help in highlighting markers for the way forward.

An exploration of what makes 'grey' literature grey, and the significance of that may enhance this, or perhaps future work.

Recommendation: Methods and tools to assess implementation of mental health policies and plans: A systematic review — R1/PR9

Comments

Comments to Author: I'm happy with the flow and the efforts taken to incorporate all suggestions.

Decision: Methods and tools to assess implementation of mental health policies and plans: A systematic review — R1/PR10

Comments

No accompanying comment.

Decision: Methods and tools to assess implementation of mental health policies and plans: A systematic review — R1/PR11

Comments

No accompanying comment.