Skip to main content Accessibility help

Combining routine outcomes measurement and ‘Payment by Results’: will it work and is it worth it?

  • Alastair J. D. Macdonald (a1) and Martin Elphick (a2)


The Department of Health in England has long encouraged the routine measurement of clinical outcomes in mental health services but has now decided to use outcome measures as part of a new payments system – Payment by Results. We examine how these two policies should or might interact.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Combining routine outcomes measurement and ‘Payment by Results’: will it work and is it worth it?
      Available formats

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Combining routine outcomes measurement and ‘Payment by Results’: will it work and is it worth it?
      Available formats

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Combining routine outcomes measurement and ‘Payment by Results’: will it work and is it worth it?
      Available formats


Corresponding author

Alastair J. D. Macdonald, PO 26, HSPR Department, David Goldberg Building, Institute of Psychiatry, De Crespigny Park, London SE5 8AP, UK. Email:


Hide All

Declaration of interest

A.J.D.M. receives payment for training in HoNOS65+.



Hide All
1 Elphick, M. Information-based management of mental health services: a two-stage model for improving funding mechanisms and clinical governance. Psychiatr Bull 2007; 31: 44–8.
2 Department of Health. Equity and Excellence: Liberating the NHS. TSO (The Stationery Office), 2010.
3 Macdonald, AJ, Trauer, T. Objections to routine clinical outcomes measurement in mental health services: any evidence so far? J Ment Health 2010; 19: 517–22.
4 Macdonald, A. Routine clinical outcomes measurement in old age psychiatry. Int Psychogeriatr 2009; 21: 990–5.
5 Hill, AB. The environment and disease: association or causation? President's address. Proc R Soc Med 1965; 58: 295300.
6 Wing, JK, Beevor, AS, Curtis, RH, Park, SB, Hadden, S, Burns, A. Health of the Nation Outcome Scales (HoNOS). Research and development. Br J Psychiatry 1998; 172: 11–8.
7 National Institute for Mental Health in England, Barts and the London School of Medicine, Department of Health. Outcomes Compendium: Helping You Select the Right Tools for Best Mental Health Care Practice in Your Field. TSO (The Stationery Office), 2009.
8 Department of Health. Developing Payment by Results for mental health. The National Archives, 2010 (
9 Fetter, RB, Shin, Y, Freeman, JL, Averill, RF, Thompson, JP. Case mix definitions by diagnosis-related groups. Medical Care 1980; 18 (suppl 2): 153.
10 Elphick, M, Anthony, P. Casemix groupings for psychiatry: strengths and weaknesses of Version 2 Healthcare Resource Groups (HRGs). J Ment Health 1996; 5: 441–8.


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed

Combining routine outcomes measurement and ‘Payment by Results’: will it work and is it worth it?

  • Alastair J. D. Macdonald (a1) and Martin Elphick (a2)
Submit a response


Re:Care Clusters and Mental Health Payment by Results

Alastair Macdonald, Professor
29 November 2011

We are delighted that these responses to our editorial expand upon issues that we could not explore more fully. Tulloch gives a cogent account of the typical methodology- not now being followed in England- forderiving casemix groupings and finds the present plan wanting. He suggests a slow, careful change to commissioning based on activity and casemix. Kingdon et al make the case for a system in which both diagnosisand care pathways are central in costing and thus purchasing, only en passant asking the crucial question of how (not whether) cost can be firmly linked to the quality of services delivered.

Both letters focus on which type of data should be chosen. The intended benefit of casemix systems is to improve the direction of resources towards the greatest local need. If that were the only eventualuse of the data items under discussion then MH units should collect whichever (activity counts, clusters, diagnoses, pathways, etc) best satisfy criteria such as Fetter's, as Tulloch implies. But data, once collected, have many other uses and misuses.

Kingdon et al argue on theoretical grounds that diagnostic categoriesshould be better indicators than clusters of the type and quantity of carethat is required by patients. Yet as Tulloch points out, findings from international analysis of variance studies of actual resource consumption within diagnostic groupings have tended to lead to their abandonment. We can add that similar methodology was used in Mental Health Services in England from the early 1990's by the NHS Information Authority, testing both diagnostic and multi-domain descriptors of patient's problems, in national and multi-site trial datasets1. Diagnostically-defined HRGs wereabandoned by the DH, not only because of the modest reduction in variance achieved but also because of resistance by non-psychiatrists to the collection of diagnostic data. There was also resistance to informatics in general by a substantial proportion of clinicians, including senior Royal College leaders at the time, although that is no longer the case2. Clusters were seen by policy makers as more likely to be acceptable. The fact that they become mandatory on 31st December 2011 with only this discussion in the Journal suggests that this approach is working.

As Kingdon et al point out, diagnostic categories enable us to use therapeutic research findings to decide which type of drug or psycho-social approach is chosen, but that does not much affect overall costs, and people often retain the same diagnosis throughout many life changes. By contrast, multi-domain scores include more factors that indicate whether someone currently needs admission, or frequent contact with paid professionals, which are the main financial determinants. And since the MHCT includes symptomatology ratings, and separates clusters into broad diagnostic groups anyway, the statistical benefits of diagnosis have not been entirely lost. Until there is more empirical evidence from costing studies, the relative merits of diagnostic versus multi-domain data will remain debatable. Of course their value in outcomes and other quality monitoring, and predicting prognosis, must also be considered in developing MH informatics generally. Prognosis is important because thereis more 'value' in resolving a situation that would otherwise become chronic.

We do not support the automatic assignment of patients to any form oftreatment, pathway, or package of care on the basis of MHCT scores alone. The data may raise retrospective questions about clinical judgments, but should not replace them.

So what should we be doing about commissioning? Tulloch suggests in effect returning to the 1993 position and starting again. We do not thinkthis is possible; while we looked away, boats were burnt. Kingdon et al propose the combination of diagnoses with pathway data for costing purposes, but do not say quite how. The strong argument against using intervention counts, pathway data, or other activity measures on their ownfor remuneration is that there is no safeguard against unnecessary, ineffective or inefficient interventions or pathways. Tariff 'matrices' in which prices are applied to cells containing both broad diagnoses and clinical management data have been proposed in the past1 but as we said above, they were abandoned. The large number of resulting categories should theoretically reduce costing variance, but it may be that commissioners would not in practice be able to use them effectively.

Yeomans concentrates on Routine Clinical Outcomes Measurement (RCOM),arguing strongly for its development and enhancement, whilst wisely refraining from almost suggesting "Payment by Outcomes" which would violate Goodhart's law, succinctly put by Strathern: "When a measure becomes a target, it ceases to be a good measure"3. We agree with nearly all his points, especially on the importance of feedback, which are, notwithstanding the dated survey he quotes, already coming to pass in someparts of England, as are developments in Patient Reported Outcomes Measures (PROMs). Efforts to usefully involve HoNOS in clinical work itself are being reported, though from the other end of the earth4. As hesays, HoNOS are a start but not the last word in outcomes measures, and wewould caution against using them for thresholds for referral or discharge.Validity in groups is no guarantee of validity in individual cases.

With exceptions, we have been slow to grasp the twin nettles of outcomes and costing of services, and if we are to regain the initiative we have to think widely and deeply about what systems we think will work best for service users, even whilst change in these very systems is accelerating. A start would be made when Trusts have clinical, outcomes, intervention, costing, HR and finance data on the same spreadsheets for themselves.

Alastair MacdonaldMartin Elphick

Reference List

1.Carthew R, Elphick M, Page A. Report on the Development of Mental Health Groupings. http://www icservices nhs uk/casemix/pages/downloads/sub10/PHSMI_Mental_Health_Findings pdf 2003 [accessed 2011 Nov 19];

2.Royal College of Psychiatrists. Information-based Funding, Qualityand Outcomes: Statement of Principles. http://www rcpsych ac uk/pdf/IFQO%20Principles%20May%202009 pdf 2009 [accessed 2011 Nov 19];

3.Strathern M. 'Improving ratings': audit in the British University system. European Review 1997;5(3):305-21.

4.Stewart M. Making the HoNOS(CA) clinically useful: a strategy for making HoNOS, HoNOSCA, and HoNOS65+ useful to the clinical team. Australian & New Zealand Journal of Psychiatry 2008;42:A5.

... More

Conflict of interest: AM receives payment for training in HoNOS65+ ME declares no conflict of interest.

Write a reply

Broad diagnoses are central to maintaining quality with PbR

David G Kingdon, Professor
14 September 2011

MacDonald & Elphick 1 have lucidly described the proposed introduction of Payment-by-Results into mental health. They mention however that 'concerns include the validity and reliability of the MHCT' -the mental health clustering tool - and there is also the major issue of how cost can be firmly linked to the quality of services delivered.

The Department of Health approach to reliability has been to rely on local initiatives and to commission the development of an algorithm based on the MHCT to ensure that clusters are reliably allocated. Exactly why DH believes this is possible by either route is not clear. Local initiatives is the route to mayhem and none of the attempts at devising algorithms so far have been successful. The instrument on which the MHCT is based, HoNOS, was not devised for this purpose. Additional items have been added but this was for clinical reasons. HoNOS was devised as a clinical outcome measure, not for needs assessment and certainly not as a classification tool. Internationally recognised tools (SCAN, SCID) have been devised to classify conditions but these use a range of information, e.g. symptoms, beliefs and timescales with specified criteria which have been and continue to be subject to international expert scrutiny.

A unit of costing must be directly related to quality and outcome measures or the UK will have the same problems as the US have encountered in its payment systems. It is difficult to understand how clusters can besuch units of cost unless there is a very substantial body of research investigating evidence for efficacy of interventions, e.g. cognitive therapy and medication, for individual clusters, and for the development and reliability testing of outcome measures - which would take years. Attempting to match cluster to pathway/intervention has to be done by using diagnosis as an intermediate step because that is where the evidencecurrently exists. The problem then is that each cluster relates to a number of guidelines and monitoring quality becomes complicated - as Trusts, and previously DH, are finding in attempting to devise cluster pathways. GP commissioners won't have the time, resource or inclination toundertake such complex monitoring - so cost will rule.

Broad diagnoses, as used by NICE, have proved satisfactory for clinical purposes and have readily available, reliable and applicable outcome measures2 and, whilst diagnosis alone is not sufficient for costing, in combination with clinical pathways3 they can be costed and used for tariffs with much better chance of reliability and homogeneity. The very limited data that has been produced so far is promising (available on request) but there needs to be more extensive examination ofdata. DH needs to take a lead because Trusts are not going to reanalyse their data using diagnosis and allocation to pathways unless DH asks them to do so.

As MacDonald and Elphick1 describe, outcome measurement is needed in any system and clustering has been very effective at promoting use of HoNOS. However combining diagnosis and pathways could provide a simpler, practical approach to gathering data for costing and tracking change than can use of clusters. It would also lead to an improved quality of care by linking cost directly to the use of evidence-based guidelines and care pathways by empowered patients, carers, providers and commissioners.

1.MacDonald AJD, Elphick M. Combining routine outcomes measurement and Payment by Results: will it work and is it worth it? Br J Psychiatry 2011; 199:178-179.

2.NHS South Central. Emotional wellbeing website. 5th September 2011)

3.NHS South Central. Emotional wellbeing website. 5th September 2011)

... More

Conflict of interest: None declared

Write a reply

Care Clusters and Mental Health Payment by Results

Alex D. Tulloch, BRC Preparatory Clinician Scientist Fellow
14 September 2011

In their piece on mental health Payment by Results (1), Macdonald andElphick note "a lack of reassurance that costs per case within a cluster will be similar enough to support a viable tariff calculation." This may underestimate the difficulties with the proposed new payment mechanism, which may have effects wider than disruption of nascent routine outcome measurement systems.

The United Kingdom has come relatively late to the process of paymentreform for mental health services, but despite this it has followed an approach unlike that in other countries. The fundamental principle behind the care cluster approach seems to be the presumption that individuals with similar needs for care, as notionally defined by clusters of scores on the HoNOS, will receive similar care and therefore incur similar costs.Importantly, costs themselves did not enter into the original process of defining care clusters (2).

A more typical approach is to combine consultation with clinicians and statistical analysis of clinical, administrative and cost data using variance reduction so that case-mix groupings are both expected to producesimilar "clinical responses" (3) and also do in fact demonstrate acceptable homogeneity of costs. This approach was followed in the original United States' attempt to design a per case prospective payment systems for Medicare in the early 1980s (4), and also by Australia and NewZealand (5-7), when they attempted to develop payment systems based on theHealth of the Nation Outcome Scales (HoNOS; 5). Achieving homogeneous costs within groups is crucial because it minimises the random risk to providers (the risk that appropriately incurred costs and therefore revenue differ randomly from those reimbursed). A typical cut-off for acceptable cost homogeneity is for each case-mix group to have a coefficient of variation of one or less (mean divided by standard deviation). It is also essential to make sure that factors relevant in resource use which may be more or less prevalent among different providersare also represented; otherwise there may be an element of systematic risk, with certain providers being systematically underpaid and others systematically overpaid. Even when this more standard approach is followed, it may not be successful, especially in mental health, where cost variation is high. Infamously, the original Medicare prospective payment system was never implemented in specialist mental health units in the face of evidence that resource homogeneity was too great and that the system would systematically disadvantage those units (9), and has now beenabandoned in favour of an across the board per diem payment system for psychiatric inpatients (10,11). Neither the Australian nor New Zealand systems were ever implemented.

In the light of this background, it is perhaps not surprising that the Department of Health's own pilot studies from 2006 demonstrate both that resource homogeneity is unacceptably low for care clusters, with onlyone of thirteen clusters having a coefficient of variation of less than one, and also that far better homogeneity could have been achieved, especially for inpatients, had the standard variance-reduction approach been followed (12). At present, it seems to me that the lowest risk approach to reforming payment for mental health services is to adopt a basic system of activity-based funding, and use the data collected in thisway, along with clinical and administrative data, as part of a slow and careful process of reform. Certainly, payment for mental health services in the UK is ripe for reform, as variations in resource use between providers are far wider than could be accounted for by any difference in case-mix (13). But this may not be the best way of approaching it.

1. Macdonald AJD, Elphick M. Combining routine outcomes measurement and "Payment by Results": will it work and is it worth it? The British Journal of Psychiatry 2011; 199(3): 178 -179.

2. Self R, Rigby A, Leggett C, Paxton R. Clinical Decision Support Tool: a rational needs-based approach to making clinical decisions. J MentHealth 2008; 17(1): 33-48.

3. Fetter RB. The new ICD-9-CM diagnosis related groups classification scheme. [Internet]. Baltimore, MD: Health Care Financing Administration; 1983 [cited 2011 Feb 15]. Available from:

4. Fetter RB, Shin Y, Freeman JL, Averill RF, Thompson JD. Case mix definition by diagnosis-related groups. Med Care 1980; 18(2 Suppl): iii, 1-53.

5. Buckingham B, Burgess P, Solomon S, Pirkis JE, Eagar K. Developing a casemix classification for mental health services. Volume 1: main report. Canberra: Commonwealth Department of Health and Family Services; 1998.

6. Eagar K, Gaines P, Burgess P, Green J, Bower A, Buckingham B, et al. Developing a New Zealand casemix classification for mental health services. World Psychiatry 2004; 3(3): 172-177.

7. Gaines P, Bower A, Buckingham B, Eagar K, Burgess P, Green J. NewZealand Mental Health Classification and Outcomes Study: Final report. Auckland: Health Research Council of New Zealand; 2003.

8. Wing JK, Marriott S, Palmer C, Thomas V. Management of imminent violence: clinical practice guidelines to support mental health services. London: Royal College of Psychiatrists; 1998.

9. English JT, McCarrick RG. DRGs: an overview of the issues. Gen Hosp Psychiatry 1986; 8(5): 359-64.

10. Lave JR. Developing a Medicare prospective payment system for inpatient psychiatric care. Health Aff 2003; 22(5): 97-109.

11. Cotterill PG, Thomas FG. Prospective payment for Medicare inpatient psychiatric care: Assessing the alternatives. Health Care FinancRev 2004; 26(1): 85-101.

12. Health and Social Care Information Centre Casemix Service. Mental Health Casemix Classification Development: End Stage Report. Leeds:Health and Social Care Information Centre; 2006.

13. Audit Commission. Maximising resources in adult mental health [Internet]. London: Audit Commission; 2010. Available from: http://www.audit-

... More

Conflict of interest: None declared

Write a reply

Doing outcomes well

David Yeomans, Psychiatrist
14 September 2011

The key to doing routine mental health outcomes well (MacDonald & Elphick 2011) is to make them relevant, meaningful and available to practitioners, service users and managers. HoNOS (the Health of the Nation Outcome Scales) is now a front runner for a general outcome measure since it is required for Payment by Results (PbR), a new contracting system for mental health care in the UK. Only one HoNOS rating is currently required in order to allocate patients to PbR care clusters, so managers have little incentive to take the extra step and mandate more than one HoNOS rating to assess the effectiveness of interventions. The simplest way to introduce outcome measurement with HoNOS would be to mandate at least two ratings, one at the outset of an intervention and one at the close. A benchmark for this approach has been set by the Priory Group for HoNOS outcomes of in-patient stays (Priory Group 2011). Psychological therapists are ahead of the curve, since many already use a commercial outcomes tool (CORE (2011)) in their work to monitor treatment progress, to support reflection and to aid supervision. They also involve patients, who make their own ratings on a standard questionnaire. They have made outcomes relevant and meaningful. Could their experience help develop HoNOS as an outcome tool?

The HoNOS could be put to work supporting practitioners. For example, HoNOS could inform referral and assessment systems, by helping select between primary and secondary care services. If no individual HoNOS item rating is greater than 2 (mild), then secondary services may not be indicated. Individual scale scores could also indicate priorities for interventions: A high score on "hallucinations & delusions" and a low score on "living conditions" could suggest a focus on treatment over accommodation (and vice versa).

The HoNOS total and individual scale scores would also indicate progress and could be used in supervision. HoNOS scores that fall below predetermined thresholds may indicate readiness for discharge. These could even be agreed as goals with patients. Co-producing HoNOS with service users and carers could balance the perspective of HoNOS as a clinician-rated measure.

Getting all practitioners on board will need vision and effort. Gilbody et al (2002) found that psychiatrists were not very interested in recording standardised outcomes. Feedback is crucial to engagement. Trusts should invest time to design their information systems so that they report person-centred outcomes directly to practitioners & teams in a meaningful format. Simply reporting outcome returns centrally would miss a huge opportunity to engage clinicians with outcomes, but still burden them with data collection.

Outcomes information will create new challenges, for example the apparent ability to compare the effectiveness of teams and individual practitioners. For some, this could be intensely motivating or intimidating. The introduction of standard outcome measures should be done thoughtfully with ongoing input from service users, practitioners, managers and academics; or as Macdonald and Elphick put it: Well.


Macdonald, AJD and Elphick M. Combining routine outcomes measurement and 'Payment by Results': will it work and is it worth it? British Journal of Psychiatry (2011) 199: 178-179

CORE Clinical Outcomes in Routine Evaluation website accessed on 04/09/2011.

Priory Group website accessed on 04/09/2011.

Gilbody, SM, House, AO, Sheldon TA. Psychiatrists in the UK do not use outcomes measures. National survey. British Journal of Psychiatry (2002) 180: 101-103
... More

Conflict of interest: Outcomes data affect my appraisal. I have a clinical information lead role in my Trust

Write a reply


Reply to: Submit a response

Your details

Conflicting interests

Do you have any conflicting interests? *