Skip to main content Accessibility help


  • Access
  • Cited by 4


      • Send article to Kindle

        To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Evaluation of benefit to patients of training mental health professionals in suicide guidelines: Cluster randomised trial
        Available formats

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Evaluation of benefit to patients of training mental health professionals in suicide guidelines: Cluster randomised trial
        Available formats

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Evaluation of benefit to patients of training mental health professionals in suicide guidelines: Cluster randomised trial
        Available formats
Export citation



Randomised studies examining the effect on patients of training professionals in adherence to suicide guidelines are scarce.


To assess whether patients benefited from the training of professionals in adherence to suicide guidelines.


In total 45 psychiatric departments were randomised (Dutch trial register: NTR3092). In the intervention condition, all staff in the departments were trained with an e-learning supported train-the-trainer programme. After the intervention, patients were assessed at admission and at 3-month follow-up. Primary outcome was change in suicide ideation, assessed with the Beck Scale for Suicide Ideation.


For the total group of 566 patients with a positive score on the Beck Scale for Suicide Ideation at baseline, intention-to-treat analysis showed no effects of the intervention on patient outcomes at 3-month follow-up. Patients who were suicidal with a DSM-IV diagnosis of depression (n = 154) showed a significant decrease in suicide ideation when treated in the intervention group. Patients in the intervention group more often reported that suicidality was discussed during treatment.


Overall, no effect of our intervention on patients was found. However, we did find a beneficial effect of the training of professionals on patients with depression.


Declaration of interest


Evidence-based practice guidelines should improve patient care. 1,2 To strengthen suicide prevention in Dutch mental healthcare, an evidence-based multidisciplinary practice guideline for the assessment and treatment of suicidal behaviour (PGSB) 3 was issued in May 2012. The PGSB is based on various international guidelines on the assessment and treatment of suicidal behaviour 47 and two empirical reviews of the Scottish government. 8,9 The PGSB combines the stress–diathesis 10 and entrapment 11 model to explain the onset and persistence of suicidal behaviour and offers concrete recommendations to clinicians. Internationally, a large number of psychiatric guidelines has been made available over the past 20 years. 12 However, adherence to these guidelines in mental healthcare has been incomplete 1315 with the result that patients do not always receive appropriate care. 12 It is argued that the training of professionals in guideline recommendations may improve adherence to guidelines and thereby improve patient care. 1,16,17 A systematic review 13 described all studies addressing the implementation of psychiatric guidelines, and concluded that randomised controlled studies on the effect of educational strategies on patients' well-being and recovery are needed to adequately implement new evidence and improve patient care. Suicide prevention training has been shown to improve the knowledge, skills and attitudes towards suicidal behaviour of gatekeepers (teachers, general practitioners) 1826 and mental health professionals. 27,28 Also, professional and gatekeeper training in diagnosis and treatment of depressive disorder, likely results in a reduction in suicide rates. 24,2932 This might be explained by the strong association between depressive disorders and suicide. 33 However, the actual effects of a specific suicide practice guideline training on suicidality of patients has not yet been examined in a randomised controlled trial (RCT).

Considering the relevance of the PGSB for Dutch mental heathcare, 34 and the need for evidence on the effectiveness of training professionals in guideline adherence, 35 the Dutch health funding agency ZONMW funded a cluster RCT on the structured training of professionals in recommendations of the PGSB called PITSTOP suicide (Professionals in Training to STOP suicide, Dutch trial register: NTR3092). The intervention consisted of an e-learning supported train-the-trainer programme (TtT-e), the content of which reflects the PGSB, and that was delivered to multidisciplinary teams of mental healthcare departments. We previously found that the TtT-e resulted in an approximately 10% improvement in confidence, knowledge and guideline adherence of mental healthcare professionals 36 as assessed with self-report questionnaires. As practice guidelines reflect everyday practice, professionals already show high levels of guideline adherence without being trained. In a systematic review of 235 studies on guideline dissemination, 10% change has been found to be the maximum expected change when training experienced professionals. 37 In the current study, we hypothesised that individuals who were suicidal and who were treated by professionals who were trained using the TtT-e (intervention) would recover more quickly from suicidal ideation compared with patients treated by professionals who were not trained using the TtT-e (control) but received information on the release of the guideline via the usual methods (internet, conferences, workshops, etc.). Secondary outcomes were self-reported non-fatal suicide attempts and treatment satisfaction. As fostering a working relationship with the patient is the most important part of the TtT-e, we hypothesised that all patients, whether they present suicide ideation at baseline or not, would show more treatment satisfaction in the intervention condition than patients in the control condition.


Design and sample recruitment

The PITSTOP suicide trial has been described in detail elsewhere. 38 In summary, the PITSTOP suicide trial was a multicentre cluster RCT. Clusters were care departments of mental healthcare institutions throughout The Netherlands. Departments were considered eligible for participation if they treated patients ⩾18 years of age and if professionals felt a need for training in suicide prevention skills. Patients of included departments were eligible if they were willing to provide written informed consent. Whether a patient was able to enter the study was left to the discretion of the staff. Patients who were deemed emotionally and/or cognitively unable to complete questionnaires were excluded.

Matching and randomisation

Eligible departments were matched in pairs on the basis of the main diagnostic DSM-IV category 39 of patients treated in the department, and on comparable average length of treatment. 40 Members of matched pairs were randomly allocated to either implementation as usual (control condition) or the TtT-e plus implementation as usual (intervention condition). Binary randomisation was performed by an independent researcher of the Dutch Institute for Health and Care Research (EMGO) who was not involved in the study. Patients did not know whether they were in the intervention or the control condition.


In the intervention condition, complete multidisciplinary teams of mental health professionals were trained in the application of the guideline via a 1-day small interactive group programme supported by e-learning modules (TtT-e). 40 Personalised feedback was an important element of the training. The training was provided by peers, who were trained by experts/masters in the field of suicidology according to the PITSTOP training protocol. The train-the-trainer model is based on adult learning theory, 41 which states that adult learners learn more when the education is relevant to their context and draws on their previous experience, and on the diffusion of innovation theory, 42 which states that people adopt new information better through their trusted social networks. The TtT-e combines 1-day face-to-face training with an additional e-learning module; this form of blended learning is used extensively in medical education and has been found to be more effective compared with traditional instructor-based training. 43,44 The PGSB recommendations served as the starting point to set the content of the TtT-e. The PGSB recommends systematic investigation of the suicidal condition of patients by using the Chronological Assessment of Suicidal Events (CASE) interview. 45 Based on its outcome, the risk and protection factors for suicide of individual patients are weighted. Subsequently, structured diagnosis, treatment strategy and a safety protocol are determined. In the TtT-e, the CASE interview was the overall framework for each of four role plays in which one trainee acted as a patient who was suicidal and the other trainee interviewed the ‘patient’ via the CASE interview.

Data-collection procedure

If possible, patient data were collected via routine outcome monitoring (ROM), an online assessment by which data on the effectiveness of treatment in everyday clinical practice are systematically collected. 46 In mental healthcare institutions not using ROM, data were collected by graduate students and/or research assistants using paper-and-pencil questionnaires. In the intervention condition, as soon as all staff were trained, newly admitted patients were assessed at admission (T 0) and subsequently 3 months after admission (T 1). If a patient was discharged within 3 months, T 1 was arranged just before discharge. In the control departments, T 0 started at the time that the department was informed of the allocation outcome. At both T 0 and T 1, suicide ideation was measured with the first 19 items of the Beck Scale for Suicidal Ideation (BSS). 47 Total score ranged from 0 to 38, a higher score reflects stronger suicide ideation. In the user manual of the BSS it is stated that any positive response to any BSS item may reflect the presence of suicidal intention. 48 Frequency of self-reported non-fatal suicide attempts were assessed at admission (‘Did you ever make one or more suicide attempts?’) and at T 1 (‘Did you make one or more suicide attempts between now and the first questionnaire?’). At T 1, we also assessed treatment satisfaction with four items that had been established to measure the quality of therapeutic alliance. The first two items were: ‘How satisfied are you overall with your treatment?’ and ‘How would you evaluate your relationship with your therapist?’ These two items were rated from zero to ten. The other two items were: ‘Was there any attention for your suicidal thoughts during treatment?’ and ‘How did your therapist deal with your suicidal thoughts?’ These two items were scored on a four-point scale ranging from 1, yes/well to 4, no/poor. Finally, the main DSM-IV diagnosis of each patient was collected from their electronic health record.

Written informed consent was obtained for all participants after the procedures had been fully explained. The study was approved by the medical ethical commission of the VU Medical Center (2011/151).

Patient recruitment, follow-up and sample size

The first patients were assessed on 20 January 2012. Final 3-month follow-up assessments were completed by 26 September 2013. For the primary outcome (change in suicide ideation) the number of patients that needed to be included was set at 423. This number is sufficient to find a small effect size (Cohen's d) of 0.3, assuming an alpha of 0.05 and the statistical power of 1 – β = 0.80. A correction of 20% for clustering of effects within departments was applied.

Statistical analyses

Differences in baseline characteristics of patients who dropped out and those who did not were analysed by using logistic regression.

Suicide ideation and non-fatal suicide attempts

In our study, we were interested in any change on the BSS, even in participants with the lowest score on the BSS. Therefore, for our primary analysis, we selected patients that scored ⩾1 on the BSS. Within this group of patients that scored ⩾1 on the BSS, baseline differences between the intervention and control group were described. Effects of the intervention on the BSS were analysed on an intention-to-treat basis by using multiple imputation for missing values. 49 We fitted a multilevel model with a random intercept at the department level. Score on the BSS at baseline was added as covariate and the randomisation condition was the between-participants factor. A multilevel logistic regression model was fitted to establish the effect of the intervention on self-reported non-fatal suicide attempt at T 1. A subgroup analysis was done by repeating the same analysis for patients diagnosed with a depressive disorder and scored ⩾1 on the BSS at baseline. Results were presented using regression coefficients (b) or odds ratios (ORs), 95% confidence intervals and P-values. Cohen's ds represented the effect size of TtT-e.

Treatment satisfaction

The first two items on treatment satisfaction were analysed by fitting linear multilevel models for all patients and, separately, for patients that scored ⩾1 on the BSS. Data were analysed on an intention-to-treat basis. Means, standard deviations and effect sizes are presented. Answers to the final two treatment satisfaction items were dichotomised by adding response option 1 and 2 into one response category (yes/well), and response 3 and 4 into another (no/poor). All analyses were done with SPSS 21.


Figure 1 shows the flow of participants and departments through the trial, showing that 881 patients in 29 departments were included. Patients who were lost to follow-up were more likely allocated to the intervention group (OR = 1.96, 95% CI 1.45–2.68, P<0.01).

Fig. 1 CONSORT 2010 flow diagram of the trial.

Intervention condition, implementation as usual plus the e-learning supported train-the-trainer programme; control condition, implementation as usual.

All 881 patients completed baseline assessments; 566 (64%) had a score on the BSS ⩾1 and 250 patients (28%) reported a history of at least one suicide attempt.

Within the sample of 566 patients with a score on the BSS ⩾1 at baseline, relatively more women were included in the intervention group compared with the control condition (Table 1). Also, the intervention condition contained more patients with a diagnosis of a personality disorder or an eating disorder, whereas the control group contained more patients with a depressive or a substance dependence disorder. Distribution of suicidal ideation and percentage of people who had made previous attempts within the suicidal sample were comparable between both groups.

Table 1 Baseline characteristics of patients with a score on the Beck Scale for Suicidal Ideation ⩾1, split per condition


(n = 566)
Intervention group

(n = 312)
Control group

(n = 254)
n a
Demographic characteristics
Female, n (%) 295 (55) 160 (56) 134 (53) 540
Age, years: mean (s.d.) 42 (15) 42 (15) 41 (14) 566
Education, n (%) 339
    Lower 37 (11) 21 (10) 16 (12)
    Intermediate 233 (69) 145 (71) 88 (65)
    Higher 69 (20) 37 (18) 32 (24)
Living with partner, n (%) 133 (38) 84 (40) 49 (35) 353
Born in The Netherlands, n (%) 253 (94) 162 (95) 91 (92) 268
Paid employment, n (%) 47 (15) 20 (10) 27 (11) 341
Data collected with ROM, n (%) 192 (34) 98 (31) 94 (37) 566
Clinical characteristics
Suicidal thoughts, mean (s.d.) 12 (9) 13 (9) 11 (8) 566
Attempted suicide, n (%) 537
    Never 303 (57) 166 (55) 137 (59)
    Once 120 (22) 66 (22) 54 (23)
    More than once 114 (21) 72 (23) 42 (18)
Diagnosis, n (%) 358
    Depression 154 (41) 75 (36) 79 (50)
    Anxiety 25 (7) 12 (6) 13 (8)
    Psychosis 30 (8) 18 (9) 12 (8)
    Personality disorder 62 (17) 47 (23) 15 (9)
    Substance dependence 39 (11) 9 (4) 30 (19)
    Post-traumatic stress disorder 21 (6) 12 (6) 9 (6)
    Eating disorder 27 (7) 26 (12) 1 (6)

ROM, routine outcome monitoring.

a. Data are not available for all patients for all variables.

Suicide ideation results and non-fatal suicide attempts

Multilevel analysis showed no effect of the intervention on change in suicide ideation (b = −0.68, 95% CI −0.66 to 1.21, P>0.05) or on frequency of self-reported attempted suicide between baseline and follow-up (OR = 1.18, 95% CI 0.62 to 2.57, P>0.05). To our knowledge, no suicides took place during the follow-up period. However, in a subgroup of patients with a diagnosis of depression that also scored ⩾1 on the BSS at baseline (n = 154; intervention, 75, control, 79), a significant effect on change in suicide ideation between conditions was found: scores on the BSS decreased 8.4 points between baseline and follow-up in the intervention group, compared with a decrease of 4.8 in the control group (b = 3.41, 95% CI 0.38–5.93, P = 0.008, effect size 0.4 (Table 2)). No effect of the intervention on self-reported attempted suicide was found within this subgroup (OR = 1.18, 95% CI 0.63–2.52). For any other subgroup (anxiety, personality, psychosis), numbers were too small or too unbalanced for any significant testing (Table 2).

Table 2 Mean decrease in Beck Scale for Suicidal Ideation (BSS) scores from baseline to follow-up and effect sizes

Intervention group Control group Effect

Mean (s.d.) n Mean (s.d.) n b (95% CI) P
All patients with a score on the BSS ⩾1 4.2 (13.4) 312 4.9 (10.5) 254 −0.68 (−0.66 to 1.21) ns
Patients with diagnoses of depression and a score

on the BSS ⩾ 1
8.4 (7.7) 75 4.8 (7.9) 79 3.41 (0.38 to 5.93) 0.008 0.4

ns, not significant.

Treatment satisfaction

When analysing the total group of patients, no significant effect on overall satisfaction with treatment was found (Table 3). Satisfaction with the relationship with the therapist was higher in the control group when compared with the intervention condition (intervention group mean 6.9 (s.d. = 4.1), control condition mean 7.4 (s.d. = 3.9), b = 0.61, 95% CI 0.04–1.22, P = 0.03, effect size 0.4).

Table 3 Mean treatment satisfaction at follow-up for all patients

Intervention group

Mean (s.d.) (n = 484)
Control group

Mean (s.d.) (n = 397)
b (95% CI) P Effect

Overall satisfaction with therapy, range (0–10) 6.8 (4.4) 6.8 (4.3) −0.01 (−0.58 to 0.56) ns
Satisfaction with relationship with therapist (range 0–10) 6.9 (4.1) 7.4 (3.9) 0.61 (0.04 to 1.22) 0.03 0.4

ns, not significant.

Since >70% of the responses to the two items ‘Were suicidal thoughts addressed during treatment’ and ‘How did your therapist deal with your suicidal thoughts?’ were missing, we performed a complete case analysis only 50 (Table 4). Suicidal thoughts were more likely to be discussed in the intervention group compared with the control group (b = 1.78 CI 95% 1.12–2.93, P = 0.02, risk ratio 1.3). No differences were found regarding the appraisal of the way the therapist dealt with suicidal ideations. When applying the same analysis for patients that scored ⩾1 on the BSS at baseline, no significant effects of the intervention on any of the four items was found.

Table 4 n (%) yes/well at follow-up for completers only


group, n/N (%)
Control group

n/N (%)
Effect size
OR (95% CI) P Risk ratio Risk difference
Were suicidal thoughts addressed during

treatment? Yes
85/154 (55) 48/118 (41) 1.78 (1.1–2.93) 0.018 1.3 0.14
How did your therapist deal with your suicidal

thoughts? Well
102/133 (77) 63/89 (71) 1.33 (0.74–2.49) ns

ns, not significant.

Sensitivity analysis

When comparing the analysis of the imputed data and the complete case data, all results were comparable, expect for ‘Satisfaction with relationship with the therapist’. When using complete case analysis, no significant difference between intervention and control condition was found on that item (intervention group mean 7.1 (s.d. = 3.2), control group 7.5 (s.d. = 3.6), b = 0.41, 95% CI −0.32 to 1.16, P = 0.2).


This study examined the additional effects on patients of an e-learning supported train-the-trainer programme aimed at training professionals in adherence to the Dutch guideline on the assessment and treatment of suicidal behaviour. The TtT-e was found to result in a 10% improvement in confidence, knowledge and guideline adherence among professionals. 36 We tested whether patients benefited from this 10% improvement of professionals. For the total group of patients who were suicidal, no effect of the intervention was found on change in suicidal ideation or frequency of self-reported non-fatal suicide attempts at 3-month follow-up. As patients in the control group also recovered significantly from suicidal ideation, any additional effect of the TtT-e for the total group of patients who were suicidal was probably smaller than our sample size allowed us to detect. This is a common observation when implementing guidelines in psychiatry. 13 We did find a significant effect of our intervention for the group of 154 patients diagnosed with depression who scored positive on the BSS at baseline. This is in line with a multicentre trial that demonstrated the effectiveness of depression guideline training of general practitioners on decline in suicide ideation among elderly people with depression. 51 The effect of the intervention on only the subgroup of patients with depression might be explained by the focus of the TtT-e on making contact and discussing suicidality, which might be more appropriate for patients who are suicidal who have a depressive disorder and less so for those with, for example, a borderline or psychotic disorder.

No effect of our intervention was found on overall satisfaction with the treatment. Patients in both groups appeared to be quite satisfied with their treatment, making it difficult to achieve a significant increase in overall treatment satisfaction by training professionals in suicide guideline adherence. When analysing the imputed data, patients in the control group appear to be more satisfied with their therapist. As no significant effect was found when analysing completers-only data, and a sensitivity analysis found no other discrepancies between imputed data and completers only, we argue that this effect might be explained by the imputation of the data for a single item on subjective experience. 50 An alternative explanation is that the additional focus on suicidal behaviour (which is the core element of the PGSB and therefore the TtT-e) during treatment may be perceived as uncomfortable or too confronting by patients, thereby resulting in less satisfaction with the therapist. It might also be that recently trained staff who were more confident, knowledgeable and adherent to the guideline (as a result of the TtT-e) were not so good at engaging patients in treatment – perhaps overconfident or more focused on the new technical aspects at the expense of being less sensitive to the person and the therapeutic relationship.

We have shown that patients reported more often that suicidality was discussed during treatment, which indicates that we were able to change the behaviour of professionals in individual treatment sessions. This is in line with another randomised study that found that general practitioners assessed more patients for suicide risk after a tailored depression guideline implementation. 16

Limitations and strengths

Patients who were lost to follow-up were more likely to be allocated to the intervention group. This might be explained by the fact that departments in the intervention condition seemed to be more motivated and successful in starting data collection, resulting in the inclusion of more patients at baseline. Some of these patients might have had more severe levels in pathology, and were therefore more likely to drop out at follow-up. Although mental healthcare institutional boards agreed on collecting patient data using ROM, only 34% of the data were collected via ROM. For various reasons, mostly technical and organisational, most institutions were not able to add our questionnaires to their existing ROM. As data collection via ROM is more systematic and on a larger scale when compared with data collection via paper-and-pencil questionnaires, we included fewer patients and had more missing values than we originally anticipated.

It was especially difficult for our research assistants to get access to the DSM-IV diagnosis of patients. This resulted in a large amount of missing patient diagnoses (37%) and a difference in proportion of patient diagnoses between the intervention and the control conditions. As the success of collecting diagnoses depended on factors unrelated to patient or professionals' characteristics, but rather on factors related to the managerial authorisation to access data within a department, we do not expect that the difference between conditions influenced our results. It did, however, mean that we were unable to test the effect of our intervention on subgroups other than patients with a diagnosis of depression. Therefore, the conclusion that our intervention has effects on patients with depression and not on other patients should be considered with these limitations in mind.

Budget cuts in Dutch mental healthcare were introduced just after our randomisation was completed. Various mental healthcare departments were shut down or did not have the resources anymore to fulfil the study requirements, resulting in a loss of 11 departments after randomisation, leading to less power. Also, during our data-collection period, government retrenchment for mental healthcare made mental healthcare less accessible for patients. Patients suddenly were forced to pay a considerable sum of money for treatment in specialised mental healthcare, which resulted in a decrease in the numbers of new patients being admitted to the psychiatric departments, resulting in less power.

An important focus of our intervention was making contact with patients who were suicidal, and paying more attention to suicidal ideation. Since most of the data (66%) were collected via paper-and-pencil instead of via the ROM, patients in both conditions might have experienced more attention being paid to their suicidal thoughts because of the assessment of suicidal ideation, making any effect of our intervention more difficult to detect. Finally, as part of our safety plan, when a patient showed increased suicidal ideation at baseline in either the control or the intervention group, we reported this to their therapist. This monitoring and supervision led to more attention being given to patients who were suicidal in both conditions, making it more difficult to find an effect of our intervention.

A strength of this study is its randomised controlled design, which is scarce in this field of research. 13 Also, the departments that were included are a good representation of the psychiatric departments in The Netherlands, 40 making the results generalisable to other institutions. Finally, given the difficulty in collecting data from patients who are suicidal and admitted to mental healthcare, difficulties with ROM and the challenges of the ongoing budget cuts and reorganisations, the patient data collection in our trial can be regarded as quite successful. The large number of patients included in our study makes our findings more reliable and generalisable.

Implications and further study

This is the first randomised trial to examine the effect on patients of training professionals in adherence to the PGSB. We found that the structural training of professionals resulted in more attention being given to suicidality during treatment. This indicates that training in suicide guideline adherence can have an impact on actual clinical practice. Overall, we found no effect of our intervention on patients. We did find a significant effect on decrease in suicide ideation within a subgroup of patients with a diagnosis of depression. It appears that compared with implementation as usual, training professionals in adherence to suicide guidelines might have beneficial effects for patients who are depressed and suicidal. Future studies should investigate whether a more tailored programme, with a focus on the specific patient group in a department, would result in the same effect on, for example, patients with a personality disorder who are suicidal, as that currently found for patients who are depressed and suicidal.

Also, the relative effectiveness of the different elements of the TtT-e (the train the-trainer element, the face-to-face training, the e-learning module, the multidisciplinary element) has not been examined separately. Future studies may disentangle the effects of the different elements, so that more targeted programmes can be developed. 52 Finally, longitudinal studies should investigate the long-term effect of suicide guideline implementation on patients.

Considering the methodological and diagnostic issues discussed earlier, our study needs to be replicated. Ideally, data on suicidal ideation should be collected in a more systematic and less obtrusive manner via computerised outcome monitoring. However, implementation studies inevitably need to be done in a naturalistic sample, which has its limitations. In the current study, we managed to collect a large amount of data from multiple psychiatric departments among a heterogeneous sample of patients within a randomised design. Therefore, our findings offer the first evidence of effectiveness for patients who are depressed and suicidal of suicide guideline training of professionals.


This study is funded by The Dutch organisation for health research and development (ZonMW).


1 Grol, R, Grimshaw, J. From best evidence to best practice: effective implementation of change in patients' care. Lancet 2003; 362: 1225–30.
2 Grol, R, Grimshaw, J. Evidence-based implementation of evidence-based medicine. Jt Comm J Qual Improv 1999; 25: 503–13.
3 van Hemert, AM, Kerkhof, AFJM, de Keijser, J, Verwey, B. Multidisciplinaire Richtlijn Voor Diagnostiek en Behandeling van Suïcidaal Gedrag [Multidisciplinary Guidelines on the Assessment and Treatment of Suicidal Behaviour]. De Tijdstroom Uitgevery, 2012.
4 American Psychiatric Association. Practice Guideline for the Assessment and Treatment of Patients with Suicidal Behaviors. APA, 2003.
5 National Institute for Health and Clinical Excellence. Self-Harm: The Short-Term Physical and Psychological Management and Secondary Prevention of Self-Harm in Primary and Secondary Care. NICE, 2004.
6 CGP team for deliberate self-harm. Australian and New Zealand clinical practice guidelines for the management of adult deliberate self-harm. Aust NZ J Psychiatry 2004; 38: 868–84.
7 New Zealand Guideline Group. The Assessment and Management of People at Risk for Suicide. New Zealand Ministry of Health, 2003.
8 McLean, J, Maxwell, M, Platt, S, Harris, F, Jepson, R. Risk and Protective Factors for Suicide and Suicidal Behaviour: A Literature Review. Scottish Government Social Research, 2008.
9 Leitner, M, Barr, W, Hobby, L. Effectiveness of Interventions to Prevent Suicide and Suicidal Behaviour: A Systematic Review. Scottish Government Social Research, 2008.
10 Goldney, RD. Contributing factors to suicide. In Suicide Prevention: 1925. Oxford University Press, 2008.
11 Williams, JMG, Pollock, L. Psychological aspects of the suicidal process. In Understanding Suicidal Behaviour (ed. van Heeringen, C): 7693. John Wiley, 2001.
12 Franx, G, Niesink, P, Swinkels, J, Burgers, J, Wensing, M, Grol, R. Ten years of multidisciplinary mental health guidelines in the Netherlands. Int Rev Psychiatry 2011; 23: 371–8.
13 Weinmann, S, Koesters, M, Becker, T. Effects of implementation of psychiatric guidelines on provider performance and patient outcome: systematic review. Acta Psychiatr Scand 2007; 115: 420–33.
14 Leucht, S. Psychiatric treatment guidelines: doctors' non-compliance or insufficient evidence? Acta Psychiatr Scand 2007; 115: 417–9.
15 van Splunteren, P, Hermens, M. Van kennis naar actie. Toepassing van richtlijnen in de ggz [From knowledge to action. Applying guidelines in mental health care]. Gedragstherapie 2011; 44: 285.
16 Baker, R, Reddish, S, Robertson, N, Hearnshawd:, H, Jones, B. Randomised controlled trial of tailored strategies to implement guidelines for the management of patients with depression in general practice. Br J Gen Pract 2001; 51: 737.
17 Worrall, G, Angel, J, Chaulk, P, Clarke, C, Robbins, M. Effectiveness of an educational strategy to improve family physicians' detection and management of depression a randomised controlled trial. CMAJ 1999; 161: 3740.
18 King, KA, Smith, J. Project SOAR: a training program to increase school counselors' knowledge and confidence regarding suicide prevention and intervention. J Sch Health 2000; 70: 402–7.
19 Isaac, M, Elias, B, Katz, LY, Belik, S-L, Deane, FP, Enns, MW, et al. Gatekeeper training as a preventive intervention for suicide: a systematic review. Can J Psychiatry 2009; 54; 206–68.
20 Wyman, PA, Brown, CH, Inman, J, Cross, W, Schmeelk-Cone, K, Guo, J, et al. Randomised trial of a gatekeeper program for suicide prevention: 1-year impact on secondary school staff. J Consult Clin Psychol 2008; 76: 104.
21 Stuart, C, Waalen, JK, Haelstromm, E. Many helping hearts: an evaluation of peer gatekeeper training in suicide risk assessment. Death Stud 2003; 27: 321–33.
22 Joffe, P. An empirically supported program to prevent suicide in a college student population. Suicide Life Threat Behav 2008; 38: 87103.
23 Capp, K, Deane, FP, Lambert, G. Suicide prevention in Aboriginal communities: Application of community gatekeeper training. Aust NZ J Public Health 2001; 25: 315–21.
24 Matthieu, MM, Cross, W, Batres, AR, Flora, CM, Knox, KL. Evaluation of gatekeeper training for suicide prevention in veterans. Arch Suicide Res 2008; 12: 148–54.
25 Gullestrup, J, Lequertier, B, Martin, G. MATES in construction: impact of a multimodal, community-based program for suicide prevention in the construction industry. Int J Environm Res Public Health 2011; 8: 4180–96.
26 Chagnon, F, Houle, J, Marcoux, I, Renaud, J. Control-group study of an intervention training program for youth suicide prevention. Suicide Life Threat Behav 2007; 37: 135–44.
27 Oordt, MS, Jobes, DA, Fonseca, VP, Schmidt, SM. Training mental health professionals to assess and manage suicidal behavior: can provider confidence and practice behaviors be altered? Suicide Life Threat Behav 2009; 39: 2132.
28 Appleby, L, Morriss, R, Gask, L, Roland, M, Lewis, B, Perry, A, et al. An educational intervention for frontline health professionals in the assessment and management of suicidal patients (The STORM Project). Psychol Med 2000; 30: 805–12.
29 Rutz, W, Knorring, VL, Wålinder, J. Frequency of suicide on Gotland after systematic postgraduate education of general practitioners. Acta Psychiatr Scand 1989; 80: 151–4.
30 Hegerl, U, Mergl, R, Havers, I, Schmidtke, A, Lehfeld, H, Niklewski, G, et al. Sustainable effects on suicidality were found for the Nuremberg alliance against depression. Eur Arch Psychiatry Clin Neurosci 2010; 260: 401–6.
31 Szanto, K, Kalmar, S, Hendin, H, Rihmer, Z, Mann, JJ. A suicide prevention program in a region with a very high suicide rate. Arch Gen Psychiatry 2007; 64: 914.
32 Knox, KL, Litts, DA, Talcott, GW, Feig, JC, Caine, ED. Risk of suicide and related adverse outcomes after exposure to a suicide prevention programme in the US Air Force: cohort study. BMJ 2003; 327: 1376.
33 Hawton, K, van Heeringen, K. Suicide. Lancet 2009; 373: 1372–81.
34 Bool, M, Doeven, M. Verminderen van Suïcidaliteit: Actualisering van het Advies inzake Suïcide, Gezondheidsraad 1986 [Reduction of Suicidality: Updating the Advice on Suicide, Health Council 1986]. Trimbos-Institute, 2007.
35 Sinnema, H, Franx, G., Op 't Land, H. Multidisciplinaire Richtlijnen in de GGZ. Themarapportage [Multidisciplinary Guidelines in Mental Health Care. Special Issue]. Trimbos Institute, 2009.
36 de Beurs, DP, de Groot, MH, de Keijser, J, Van Duijn, E, de Winter, R, Kerkhof, AJFM. The effect of an e-learning supported Train-the-Trainer program to implement suicide guidelines in mental heathcare. J Affect Disord 2015; 175: 446–53.
37 Grimshaw, J, Thomas, R, MacLennan, G, Fraser, C, Ramsay, C, Vale, L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Int J Technol Assess Health Care: 2004; 21: 149.
38 de Beurs, DP, de Groot, MH, Bosmans, JE, de Keijser, J, Mokkenstorm, J, Verwey, B, et al. Reducing patients' suicide ideation through training mental health teams in the application of the Dutch multidisciplinary practice guideline on assessment and treatment of suicidal behavior: study protocol of a randomized controlled trial. Trials 2013; 14: 372.
39 American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (4th edn) (DSM-IV). APA, 1994.
40 de Beurs, DP, de Groot, MH, de Keijser, J, Verwey, B, Mokkenstorm, J, Twisk, JW, et al. Improving the application of a practice guideline for the assessment and treatment of suicidal behavior by training the full staff of psychiatric departments via an e-learning supported Train-the-Trainer program: study protocol for a randomized controlled trial. Trials 2013; 14: 9.
41 Knowles, MMS. The Modern Practice of Adult Education. Association Press, 1970.
42 Rogers, EM. Diffusion of Innovations. Simon and Schuster, 2010.
43 Means, B, Toyama, Y, Murphy, RF, Baki, M. The effectiveness of online and blended learning: a meta-analysis of the empirical literature. Teach Coll Rec 2013; 115: 147.
44 Pearce, J, Mann, MK, Jones, C, van Buschbach, S, Olff, M, Bisson, JI. The most effective way of delivering a train-the-trainers program: a systematic review. J Contin Educ Health Prof 2012; 32: 215–26.
45 Shea, SC. The chronological assessment of suicide events: a practical interviewing strategy for the elicitation of suicidal ideation. J Clin Psychiatry 1998; 59: 5872.
46 De Beurs, E, den Hollander-Gijsman, M, Van Rood, Y, Van der Wee, N, Giltay, E, Van Noorden, M, et al. Routine outcome monitoring in the Netherlands: practical experiences with a web-based strategy for the assessment of treatment outcome in clinical practice. Clin Psychol Psychother 2011; 18: 112.
47 Beck, AT, Kovacs, M, Weissman, A. Assessment of suicidal intention: the Scale for Suicide Ideation. J Consult Clin Psychol 1979; 47: 343–52.
48 Beck, AT, Steer, RA. Manual for the Beck Scale for Suicide Ideation. Psychological Corporation, 1991.
49 Van Buuren, S. Multiple imputation of discrete and continuous data by fully conditional specification. Stat Methods Med Res 2007; 16: 219–42.
50 Sterne, JA, White, IR, Carlin, JB, Spratt, M, Royston, P, Kenward, MG, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ 2009; 338: b2393.
51 Alexopoulos, G, Reynolds, C, Bruce, M, Katz, I, Raue, P, Mulsant, B, et al. Reducing suicidal ideation and depression in older primary care patients: 24-month outcomes of the PROSPECT study. Am J Psychiatry 2009; 166: 882–90.
52 Cherry, MG, Brown, JM, Bethell, GS, Neal, T, Shaw, NJ. Features of educational interventions that lead to compliance with hand hygiene in healthcare professionals within a hospital care setting. A BEME systematic review: BEME Guide No. 22. Med Teach 2012; 34: e40620.