Sample

The target sample comprised 35–40 consecutively admitted patients at each site. Exclusion criteria were opting out of the study, being admitted for non-crisis purposes (e.g. planned respite care) and being transferred from another acute ward for non-clinical reasons.

Measures

The Health of the Nation Outcome Scales (HoNOS) is a 12-item staff-rated measure of social disability. ^{Reference Wing, Beevor, Curtis, Park, Hadden and Burns7} Each item is scored from 0 (no problem) to 4 (severe to very severe problem), and the HoNOS score is the sum of the item scores (range 0–48, low score better). The Global Assessment of Functioning (GAF) is a two-item staff-rated measure of symptoms and social functioning; each item is scored from 1 (most severe) to 90 (no problems). ^{Reference Jones, Thornicroft, Coffey and Dunn8} The Threshold Assessment Grid (TAG) is a seven-item staff-rated measure of severity of mental health problems, across domains of need, safety and risk. ^{Reference Salvi, Leese and Slade9} Four items are scored from 0 (none) to 3 (severe), and the remaining three items can also be scored as 4 (very severe). The TAG score is the sum of the item scores (range 0–24, low score better).

Procedures

A consecutive series of participants were recruited to the study on admission to each service. All participants received information through posters displayed on the unit and information sheets received on admission and at discharge, and were given the opportunity to opt out. Clinical staff received researcher-led training to rate the measures, and completed the assessments as soon as possible after the participant's admission. Participants were rated again at discharge, by the same member of staff where possible. The manager of each alternative service provided budget data for the service to calculate cost per bed-day. The cost of each standard ward was taken from nationally available estimates. ¹⁰

Statistical analysis

Baseline values and changes for each of the outcomes were calculated and presented for each service, and t-tests and Wilcoxon tests used to compare lengths of stay. Mean differences between the standard and alternative admissions at follow-up, adjusted for area and baseline level only and also for possible confounding factors, were estimated from linear regression models, one for each outcome of interest. Choice of possible confounders was informed by results from a previous paper in this series, ^{Reference Johnson, Lloyd-Evans, Morant, Gilburt, Shepherd and Slade6} which identified predictors of being admitted to an alternative rather than a standard service. Service was included as a clustering variable and robust standard errors computed. Area was included as a fixed effect, and analyses were repeated excluding the single non-paired site as a sensitivity analysis. A logistic regression was also performed with the binary dependent variable a clinically significant change of at least 7 points in HoNOS score, ^{Reference Parabiaghi, Barbato, D'Avanzo, Erlicher and Lora11} and proportions improving in the two types of service were estimated from this model (adjusting for area and baseline level). Multiple imputation was used for missing values (these were mainly for the predictor variables, about 13% of the total).

The total cost of the stay in the initial alternative or standard service was calculated and differences in the cost of standard and alternative admissions (adjusted for baseline predictors) compared using standard parametric tests with bootstrapped analyses carried out to address skewed cost distributions. Costs were then linked to outcomes using an incremental cost-effectiveness analysis to explore the relationship between cost and HoNOS score. The probability that alternative facilities are more cost-effective than standard services was then explored through the calculation of cost-effectiveness acceptability curves, which show the probability that admission to an alternative service is more cost-effective than admission to a standard ward for different values that a decision-maker might be willing to pay for improvements in the HoNOS outcome. ^{Reference Fenwick and Byford12}

Implications for patients

In relation to patient care, outcomes improved in both classes of service. The improvement was larger in the standard services, and equivalent to reductions from admission (mean score 14.3) to discharge (mean score 7.2) in the largest HoNOS outcomes database (n = 101 820), generated through routine outcome collection in Australia. ¹⁴ There are at least three possible reasons for the greater improvement in HoNOS ratings for standard admissions. First, standard services may be more effective than alternative services in addressing the domains of health and social functioning specifically assessed with HoNOS. However, the difference after adjustment was of only marginal clinical importance: 2.0 points on HoNOS and 1.4 points on TAG. Second, the admission HoNOS score for alternative services was either equal to (non-clinical alternative 1 only) or lower than the comparison standard service. A similar pattern was evident for TAG, although less so for GAF. A proportionally similar improvement in outcome across both classes of service would therefore lead to a greater absolute improvement in standard services, and preliminary emerging evidence of proportional rather than absolute change in different subpopulations suggests this possibility might warrant further consideration. ^{Reference Trauer, Eagar and Mellsop15}

Fig. 1 Cost-effectiveness acceptability curve showing the probability that admission to standard services is cost-effective for different values that a decision-maker is willing to pay for a one-point improvement in Health of the Nation Outcome Scales (HoNOS) score.

Finally, the length of stay was longer at standard services (mean 38.2 days) than at alternative services (mean 17.6 days). There was a complex relationship between length of stay and outcome improvement across the four outcomes and 12 services, but no consistent relationship between length of stay and outcome at the patient level, despite this finding at the aggregate level. Therefore length of stay was not included as a confounder. This leaves open the possibility that the greater improvement in HoNOS score was partly or wholly attributable for some patients to the longer duration of stay associated with admission to a standard service. Our data cannot directly address this possibility, but the pattern for some patients in alternative services of very short stay and relatively limited improvement raises questions about whether they are discharged prematurely (and longer stays would lead to improved outcome) or whether they are rapidly assessed as unsuitable for or not requiring the service and transferred to a standard ward (or discharged).

Service-level implications

The shorter length of stay in alternative services means that (assuming 100% bed occupancy) they can offer 2.2 times the admissions per bed that is possible in standard services. Although it is tempting to link throughput data with the number of HoNOS improvers to arrive at an overall cost-effectiveness measure for alternative compared with standard services, this is not appropriate for three reasons. First, the study was an observational design, and the choice of where to admit an individual was not random. People admitted to alternatives were more likely to be known to services, ^{Reference Johnson, Lloyd-Evans, Morant, Gilburt, Shepherd and Slade6} and so the two classes of service are working with different, albeit overlapping, groups. If people admitted to alternatives service are on average less acutely ill (i.e. have more chronic symptoms and social disability with occasional exacerbation to a point requiring admission) then this raises the question of whether measures such as HoNOS are the best outcome for evaluating the effectiveness of admission. Second, staff ratings were made by a mixed group of workers in the alternative services, not all of whom had a formal mental health training. It is possible that, despite the provision of training for all staff raters, clinicians in standard services were better trained to detect changes in the domains assessed in HoNOS. Finally, it is not possible to know whether standard services would achieve the same gains in outcome using shorter admissions, and hence more efficient use of beds, or indeed whether alternative services would achieve greater gains using longer admissions. One approach evaluated in Scotland was to close a ward and use the resulting resources to fund a marginal increase in beds available on other wards and increased staff in community services. ^{Reference Melvin, Hall and Bienek16} No difference was found in post-closure rates of admission to other wards, bed occupancy, number of incidents, number of days the ward doors were locked, observation levels, sickness levels or number of temporary staff used.

Limitations and future research

Alternative services now comprise 10% of the overall in-patient provision, ^{Reference Salvi, Leese and Slade9} but this growth has been unevaluated and uncoordinated. Our observational study demonstrates that in relation to short-term clinical outcomes, and consistent with other studies, ^{Reference Howard, Flach, Leese, Byford, Killaspy and Cole17} clinical improvement occurs in both classes of service. Without more experimental designs it is not possible to make definitive statements about relative and absolute cost-effectiveness. For example, randomisation of the subgroup of patients who are served by both classes of service to either an alternative or a standard admission would allow comparative effectiveness to be established. ^{Reference Johnson, Lloyd-Evans, Morant, Gilburt, Shepherd and Slade6} Barriers to a randomised controlled trial include lack of fidelity scales for the identified types of alternative service and for what constitutes a ‘standard’ in-patient service, lack of consensus in relation to evaluation strategies, and the likely requirement for substantial resourcing to both run and evaluate several services, which in turn raises questions about generalisability. ^{Reference Howard, Leese, Byford, Killaspy, Cole and Lawlor18}

A more feasible short-term research strategy would be to identify and amplify the active and positive ingredients of in-patient services. There is no consensus on the ideal in-patient service model, and there is great variation in even the most basic service planning decisions. For example, the most recent survey of in-patient care in England identified that the numbers of beds per ward ranged from 5 to 32. ¹⁹ A complex relationship between service type and content of care provided was also found in this study, ^{Reference Lloyd-Evans, Johnson, Morant, Gilburt, Osborn and Jagielska13} a finding replicated in relation to service type and satisfaction and the experience of services. ^{Reference Osborn, Lloyd-Evans, Johnson, Gilburt, Byford and Leese20,Reference Gilburt, Slade, Rose, Lloyd-Evans, Johnson and Osborn21} Experimental intervention to systematically vary potential active ingredients of care will allow the development of testable models, which can then be investigated using established methods for evaluating complex interventions. ^{Reference Campbell, Murray, Darbyshire, Emery, Farmer and Griffiths22} It is plausible that the concept of an alternative service may prove to be a proxy measure for a constellation of features, such as more patient choice and control, less coercion, more motivated staff, less staff–patient social distance, community-based care and the provision of specific types of intervention. A more detailed understanding of these components and their effectiveness may over time lead to the abandonment of the binary alternative v. standard service distinction, in favour of a more sophisticated understanding of the impact of each feature for individual patients. ^{Reference Pawson and Tilley23}

A specific research question relates to the impact of different lengths of stay. As part of the wider study, we undertook stake-holder interviews (primarily reported elsewhere). ^{Reference Johnson, Lloyd-Evans, Morant, Gilburt, Shepherd and Slade6} These revealed a number of influences on length of stay: alternatives often had strong links with local crisis resolution teams – for which there is an emerging evidence base ^{Reference McCrone, Johnson, Nolan, Pilling, Sandor and Hoult24} – or had their own aftercare services, and these more robust community aftercare arrangements facilitated earlier discharge; standard services had more standardised discharge procedures, some of which (such as waiting until the next weekly ward round) delayed discharge; patients admitted to standard services presented on average more risk issues, leading to greater caution in discharge decision-making. Keeping lengths of stay equal in the two classes of service would allow a direct comparison of effectiveness, but for some services (e.g. the short-stay ward) this would not be possible, and for others it would be problematic for the reasons shown in the stakeholder interviews. ^{Reference Johnson, Lloyd-Evans, Morant, Gilburt, Shepherd and Slade6} A repeated measures design, for example using routinely collected outcome data, ^{Reference Plovnick25} would allow investigation of the extent to which outcomes improve continually during an admission, or whether there is a plateau after a certain length of time. This would inform guidelines about the ideal length of admission.

A final research strategy involves more systematic exploration of trade-offs between different desirable dimensions of evaluation. Service provision is not all about outcome. For example, it is known that rates of detention are higher for people from Black and minority ethnic (BME) communities: 38% compared with 19% among people from non-BME communities. ²⁶ The rationale for the BME-focused service investigated in our study was not to generate improved outcome but to offer a more culturally sensitive experience of admission. The relative weighting placed on accessibility, satisfaction, effectiveness, cost-effectiveness and efficiency is a sociopolitical not a clinical decision. The contribution research can make is to inform the debate, and identify the implications of different weightings.

Avoiding the need for hospitalisation is desirable, and crisis resolution teams which provide intensive support leading to reduced admission rates are now widespread in England. ^{Reference Johnson, Nolan, Pilling, Sandor, Hoult and McKenzie27,Reference Glover, Arts and Babu28} Where admission is necessary, we found that people are likely to improve in relation to short-term clinical outcomes whether admitted to an alternative or a standard service, with higher improvement (but of marginal clinical importance) in standard services. Our costing approach used a narrow perspective, only including the costs of the admission. A key question is what happens following discharge in terms of longer-term outcomes and costs. In a separate paper we provide preliminary evidence regarding longer-term costs following admission to the two types of service. ^{Reference Byford, Sharac, Lloyd-Evans, Gilburt, Osborn and Leese29} There was little difference in the use and cost of subsequent admissions or other hospital and community mental health services between people initially admitted to traditional or to alternative services. Hence the shorter lengths of stay in alternative services were not associated with a greater need for subsequent admissions or support from other mental health services, and led to significantly lower 12-month total costs for service use.

It is known that the experience of admission is central. Satisfaction associated with an involuntary admission predicts 1-year involuntary readmission rates, ^{Reference Priebe, Katsakou, Amos, Leese, Morriss and Rose30} yet in 2008 the Mental Health Act Commission found that standard wards ‘appear to be tougher and scarier places than we saw a decade ago’. ³¹ Recent calls from professionals and consumers, ^{32,Reference Slade33} and emerging policy guidance, ³⁴ have all emphasised the need for more focus within in-patient services on ‘putting a greater focus on the individual and care that is personalised’. ³⁵ Our empirical finding was that there is no substantial difference in short-term staff-rated outcomes – the largest change was in HoNOS scores (an adjusted difference of 2 on a scale ranging from 0 to 48), with less clear-cut advantages for the other three outcome measures. This provides a further argument for ensuring that service users' preferences and experiences should strongly inform clinical decision-making.