Literature search

In total, 2479 papers were identified from the three databases, of which 1481 were non-duplicates (Fig. 1); 39 review papers were identified, with the 3 most recent reviews in each condition covering patient-level models published up to December 2015.^{Reference Jin, Tappenden, Robinson, Achilla, Aceituno and Byford25,Reference Kolovos, Bosmans, Riper, Chevreul, Coupé and van Tulder28,Reference Mavranezouli and Lokkerbol50} Inclusion criteria for these reviews closely match this study and are detailed in full in supplementary Appendix 5. To update previous systematic reviews, 572 papers published between 1 January 2016 and 26 August 2020 were identified from the 1481 non-duplicate papers. An additional 5 patient-level economic models not covered by previous systematic reviews were identified from the 572 papers. Hence, we identified 15 unique patient-level economic models from a total of 28 studies.^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51–65} Full detail of records assessed is provided in supplementary Appendix 6.

Fig. 1 Literature search – PRISMA diagram.

Description of health economic models

An overview of the 15 models is provided in Table 1. Seven patient-level models (47%) targeted major depressive disorder,^{Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59–65} six (40%) targeted schizophrenia^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51–Reference Jin, Tappenden, MacCabe, Robinson and Byford56} and two models (13%) targeted bipolar I or bipolar II disorder.^{Reference Klok, Al Hadithy, van Schayk, Antonisse, Caro and Brouwers57,Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58} Five models (33%) examined a UK setting,^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58,Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63} four models (27%) were set in North America,^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Furiak, Ascher-Svanum, Klein, Smolen, Lawson and Conley53,55,65} three models (20%) had a European setting^{Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54,Reference Klok, Al Hadithy, van Schayk, Antonisse, Caro and Brouwers57,Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59} and the remaining three models were set in Asia/Oceania.^{Reference Prukkanone, Vos, Bertram and Lim60,Reference Saylan, Treur, Postema, Dilbaz, Savas and Heeg61,Reference Nguyen and Gordon64}

Table 1 Description of the 15 included studies

CBT, cognitive–behavioural therapy; CCA, cost–consequence analysis; CEA, Cost-effectiveness analysis; CUA, cost–utility analysis; DES, discrete-event simulation; PLDT, patient-level decision tree; PLMM, patient-level Markov model; QALY, quality-adjusted life-years; rTMS, repetitive transcranial magnetic stimulation.

a. Incidence of relapse of symptoms and side-effects, therapy discontinuation, expected costs of antipsychotic therapy, psychiatric and non-psychiatric services.

b. Cost per relapse avoided, total time in psychosis, total costs of psychosis.

c. Time to response/remission, side-effects, non-adherence, length of stay, total costs.

The majority of models (n = 11, 73%) evaluated pharmacological interventions,^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51–Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54,Reference Jin, Tappenden, MacCabe, Robinson and Byford56–Reference Saylan, Treur, Postema, Dilbaz, Savas and Heeg61,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63} five models (33%) evaluated non-pharmacological individual interventions^{55,Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Prukkanone, Vos, Bertram and Lim60,Reference Nguyen and Gordon64,65} and one model (7%) examined the effect of system-level reorganisation.^{Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62} Models were generally structured as discrete-event simulations (n = 9, 60%)^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54,Reference Jin, Tappenden, MacCabe, Robinson and Byford56–Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58,Reference Prukkanone, Vos, Bertram and Lim60–Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63} or individual-level Markov models (n = 5, 33%),^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51,Reference Furiak, Ascher-Svanum, Klein, Smolen, Lawson and Conley53,55,Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59,65} although there was also an individual-level decision tree (n = 1, 7%).^{Reference Nguyen and Gordon64} The majority of models (n = 12, 80%) took a health system or payer perspective^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51–Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58,Reference Saylan, Treur, Postema, Dilbaz, Savas and Heeg61,Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62,Reference Nguyen and Gordon64,65} and three models (20%) considered a societal perspective.^{Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59,Reference Prukkanone, Vos, Bertram and Lim60,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63}

Five models (33%) considered a lifetime perspective for their analysis,^{55,Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Saylan, Treur, Postema, Dilbaz, Savas and Heeg61,Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62,65} with the most common time horizon considered being 5 years (n = 7, 47%).^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54,Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58–Reference Prukkanone, Vos, Bertram and Lim60,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63,Reference Nguyen and Gordon64} Two models (13%) considered a 1-year time horizon^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51,Reference Furiak, Ascher-Svanum, Klein, Smolen, Lawson and Conley53} and one model (7%) used a time horizon of only 100 days.^{Reference Klok, Al Hadithy, van Schayk, Antonisse, Caro and Brouwers57}

Twelve models (80%) informed cost–utility analysis (CUA) and estimated health outcomes in terms of quality-adjusted (QALY) or disability-adjusted life-years (DALY).^{Reference Furiak, Ascher-Svanum, Klein, Smolen, Lawson and Conley53–Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58–65} The remaining three models (20%) estimated health outcomes such as relapse,^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51,Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Klok, Al Hadithy, van Schayk, Antonisse, Caro and Brouwers57} side-effects,^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51,Reference Klok, Al Hadithy, van Schayk, Antonisse, Caro and Brouwers57} treatment discontinuation^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51,Reference Klok, Al Hadithy, van Schayk, Antonisse, Caro and Brouwers57} and time in a psychotic state.^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Klok, Al Hadithy, van Schayk, Antonisse, Caro and Brouwers57} These three models estimated the total disease costs and presented disaggregated results in the form of cost–consequence analyses.^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51,Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Klok, Al Hadithy, van Schayk, Antonisse, Caro and Brouwers57}

Structural complexity

Table 2 summarises the features of the individual models. Simplified model structures for each study are presented in supplementary Appendix 7. All studies modelled an episodic/relapse state and a separate health state representing time not in episode. All models allowed participants to experience multiple relapses, except for the single model (7%) structured as a decision tree, which allowed only one relapse.^{Reference Nguyen and Gordon64}

Table 2 Complexity of the 15 models

Two models (13%) explicitly modelled incident disease in otherwise healthy populations,^{Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62} whereas the remaining models considered populations with established psychosis or major depressive disorder. All but one model (7%)^{Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62} incorporated an excess mortality risk to reflect the increased risk of death by suicide in populations with psychosis or major depressive disorder, compared with the general population. Eleven models (73%) incorporated parameter uncertainty in their modelled results, conducting a probabilistic sensitivity analysis informed by non-arbitrary parameter distributions.^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54–Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58–Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62,Reference Nguyen and Gordon64,65}

In six models (40%) relapse severity was modelled explicitly.^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Furiak, Ascher-Svanum, Klein, Smolen, Lawson and Conley53,Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58,Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59,Reference Nguyen and Gordon64,65} Of these, three models (20%) categorised two types of severity: a more severe relapse requiring hospital admission and a less severe relapse with the patient remaining in the community.^{Reference Furiak, Ascher-Svanum, Klein, Smolen, Lawson and Conley53,Reference Nguyen and Gordon64,65} The remaining three models allowed a more continuous distribution of episode severity based on patient characteristics^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59} or simply modelled relapse severity based on random chance.^{Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58} The latter three models varied time to remission from relapse, so specific patients could experience a shorter or longer relapse than the cohort average. One study^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52} modelled time-updated disease severity scores for individual patients, which jointly determined the frequency, length and symptom severity of each episode.

Nine models (60%) distinguished between those experiencing temporary relief of symptoms (remission between episodes) and those experiencing longer-term relief of symptoms (recovery), with these states being characterised by different costs and/or health outcomes.^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54,Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59,Reference Saylan, Treur, Postema, Dilbaz, Savas and Heeg61–65} In six models (40%), recovery was simulated by transition to a discrete ‘recovery state’.^{Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59,Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62–65} However, two models (13%) used a gradient of recovery (via a continuous variable or categorical variable with several levels) based on the time since last relapse and category of disease severity as determined by baseline patient characteristics.^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54}

Thirteen models (87%) explicitly simulated changes in treatment status by incorporating treatment switches and/or periods of non-adherence.^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51–Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58,Reference Prukkanone, Vos, Bertram and Lim60,Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62–65} Seven models (47%) considered both treatment switching and treatment discontinuation,^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51–Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54,Reference Jin, Tappenden, MacCabe, Robinson and Byford56–Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58} for example low treatment adherence driving treatment switching, whereas six models (40%) incorporated only treatment switching.^{55,Reference Saylan, Treur, Postema, Dilbaz, Savas and Heeg61–65}

Nine models (60%) predicted the impact of one or more side-effects resulting from antipsychotic treatments on patient health outcomes and costs.^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51–Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63,Reference Nguyen and Gordon64} However, the type of side-effects incorporated varied considerably between models. Eight models (53%) incorporated the impact of extrapyramidal symptoms^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51–Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58} and seven models (47%) incorporated weight gain.^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51–Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54,Reference Jin, Tappenden, MacCabe, Robinson and Byford56–Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58} Prolactin-related disorders,^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51,Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,55} neutropenia^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,55,Reference Jin, Tappenden, MacCabe, Robinson and Byford56} and drowsiness^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63} were each considered in three models (20%), and tardive dyskinesia^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54} and sexual dysfunction^{Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63} were each considered in two models (13%). Four models (27%) considered a disparate range of other side-effects,^{Reference Furiak, Ascher-Svanum, Klein, Smolen, Lawson and Conley53,Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63,Reference Nguyen and Gordon64} including insomnia, diarrhoea and post-injection syndrome.

Three models (20%) extended their modelling of medication-related side-effects,^{Reference Furiak, Ascher-Svanum, Klein, Smolen, Lawson and Conley53,55,Reference Jin, Tappenden, MacCabe, Robinson and Byford56} explicitly simulating a pathway from short-term transient side-effects into long-term comorbidities: diabetes^{Reference Furiak, Ascher-Svanum, Klein, Smolen, Lawson and Conley53,55,Reference Jin, Tappenden, MacCabe, Robinson and Byford56} and cardiovascular disease.^{Reference Furiak, Ascher-Svanum, Klein, Smolen, Lawson and Conley53,55} The risk of developing long-term comorbidities was conditional on side-effects such as weight gain, via mediating pathways such as hyperlipidaemia and impaired glucose tolerance. In contrast, one model accounted for the long-term impact of comorbidities implicitly,^{Reference Prukkanone, Vos, Bertram and Lim60} with its authors applying a utility adjustment to the whole patient cohort, taking into account the incidence rate of medication-related comorbidities and their average health effect across all patients.

The medication-related side-effects incorporated by each model, and further detail of the precise modelling approach for long-term comorbidities, are presented in supplementary Appendix 8.

Incorporation of individual heterogeneity and patient history

Six patient-level models (40%) allowed the risk of a relapse to vary conditional on patient characteristics^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59,Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63,65} – mainly disease severity,^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59,Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63,65} age^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59,Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63} and gender.^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52,Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63} Full details of all patient characteristics considered in each model are provided in supplementary Appendix 8. The remaining nine models (60%) assumed the risk of first relapse to be equal across all individuals.^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51,Reference Furiak, Ascher-Svanum, Klein, Smolen, Lawson and Conley53–55,Reference Klok, Al Hadithy, van Schayk, Antonisse, Caro and Brouwers57,Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58,Reference Prukkanone, Vos, Bertram and Lim60,Reference Saylan, Treur, Postema, Dilbaz, Savas and Heeg61,Reference Nguyen and Gordon64} In eight models (53%), including five models that varied relapse risk on the basis of patient characteristics, the risk of subsequent relapse was conditional on the number of previous relapses modelled.^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52–55,Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59,Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63,65} The risk of subsequent relapse varied in complexity. The simplest approach applied a single hazard ratio adjustment if a patient had any previous relapse^{Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59} and the most complex approach modelled future relapse risk as a continuous function driven by the duration of and time between previous relapses.^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52} In the remaining seven models (47%), the relapse risk was independent of the number of previous relapses modelled.^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51,Reference Jin, Tappenden, MacCabe, Robinson and Byford56–Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58,Reference Prukkanone, Vos, Bertram and Lim60,Reference Saylan, Treur, Postema, Dilbaz, Savas and Heeg61,Reference Nguyen and Gordon64}

Hierarchy of evidence informing the models

The hierarchy of evidence used in the models is summarised in Fig. 2, ranging from high-quality evidence (ranked 1) to the lowest rank of evidence (rank 6). Full ranking criteria for evidence sources across all categories are presented in supplementary Appendix 4.

Fig. 2 Hierarchy of evidence quality. NA, not applicable.

Quality of evidence informing the models was mixed, and few models used high-quality evidence across all model elements. Evidence was particularly poor for treatment effect extrapolation, with seven models (47%) relying on expert opinion to inform treatment effect extrapolation beyond observed data to model efficacy over the whole simulated time.^{Reference Heeg, Buskens, Knapp, van Aalst, Dries and de Haan52–Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54,Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58,Reference Prukkanone, Vos, Bertram and Lim60,Reference Nguyen and Gordon64} Often, there was no report of external expert consultation, so the opinion on the durability of the treatment effect over the time horizon was tacitly assumed by the model authors.^{Reference Dilla, Möller, O'Donohoe, Álvarez, Sacristán and Happich54,Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58,Reference Prukkanone, Vos, Bertram and Lim60,Reference Nguyen and Gordon64} Two models (13%) did not attempt to extrapolate beyond their observed evidence sources^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51,Reference Klok, Al Hadithy, van Schayk, Antonisse, Caro and Brouwers57} and two models (13%) evaluated hypothetical treatments,^{Reference Tosh, Kearns, Brennan, Parry, Ricketts and Saxon62,Reference Vataire, Aballéa, Antonanzas, Roijen, Lam and McCrone63} meaning that this evidence category was not applicable to these four studies. Side-effect data were also often poorly evidenced, with studies often reporting several sources of input data with poor reporting of how each evidence source was used or how studies were combined to inform model parameters.

Model quality and validity

Models generally performed reasonably against ISPOR 2014 checklist criteria, which provide a broad overview of model quality and relevance (Fig. 3). Model design and analysis were generally adequate, with model credibility weakest in terms of data and reporting. Although the ISPOR checklist provides a high-level perspective over many diverse model attributes, only one checklist item directly scrutinises the model structure. Individual checklist items encompass multiple diverse topic areas. For example, short model time horizons are only assessed within a broad checklist item assessing the applicability of the model ‘context’. In terms of model credibility, the models performed poorly against ISPOR validation criteria, as corroborated by poor performance on the AdVISHE validation checklist (Fig. 3).

Fig. 3 Validation and quality checklists. ISPOR, International Society for Pharmacoeconomics and Outcomes; AdViSHE, Assessment of the Validation Status of Health-Economic Decision Models; NA, not applicable.

Of the 15 studies, 13 (87%) performed at least 1 of the 12 validation checks listed on the AdVISHE checklist.^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51–Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58,Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59,Reference Saylan, Treur, Postema, Dilbaz, Savas and Heeg61–65} However, model validation was generally restricted to face validity checks of model structure and the suitability of model inputs, and checks of model structure and inputs against published literature. Importantly, data and output validation was reported by only a handful of models. Two models (13%) reported validation of regression model fit,^{Reference Sobocki, Ekman, Ågren, Jönsson and Rehnberg59,Reference Nguyen and Gordon64} one model (7%) reported testing with alternative input data^{Reference Furiak, Ascher-Svanum, Klein, Smolen, Lawson and Conley53} and three models (20%) reported validation checks against empirical data.^{Reference Vera-Llonch, Delea, Richardson, Rupnow, Grogg and Oster51,Reference Jin, Tappenden, MacCabe, Robinson and Byford56,Reference Ekman, Lindgren, Miltenburger, Meier, Locklear and Chatterton58} Validation of the model as implemented in software (i.e. the computerised model) was particularly infrequent. Model authors rarely sought external expert model appraisal and rarely reported that basic model checks had been undertaken, such as extreme value testing or patient tracking through the computerised model. Jin et al (2020)^{Reference Jin, Tappenden, MacCabe, Robinson and Byford56} was an exception – providing full details of their validation procedures in supplementary material, documenting 10 of the 12 AdVISHE validation checks.

Full data for both quality checklists are presented in supplementary Appendix 9.