Hostname: page-component-7c8c6479df-94d59 Total loading time: 0 Render date: 2024-03-29T12:05:23.557Z Has data issue: false hasContentIssue false

Avoiding errors about ‘margins of error’

Published online by Cambridge University Press:  02 January 2018

D. Mossman
Affiliation:
Department of Psychiatry, Wright State University Boonshoft School of Medicine, 627 S. Edwin C. Moses Boulevard, Dayton, Ohio 45408-1461, USA. Email: douglas.mossman@wright.edu
T. Sellke
Affiliation:
Department of Statistics, Purdue University, West Lafayette, Indiana, USA
Rights & Permissions [Opens in a new window]

Abstract

Type
Columns
Copyright
Copyright © Royal College of Psychiatrists, 2007 

When discussing actuarial risk assessment instruments (ARAIs), Hart et al (Reference Hart, Michie and Cooke2007) acknowledge that ‘prediction’ may refer to probabilistic statements (e.g. a ‘prediction’ that an individual ‘falls in a category for which the estimated risk of violence was 52%’: p. s60). For unclear reasons, however, the authors seem to value only predictions with right or wrong outcomes. They therefore regard statements about future behaviour of large groups (where one can be almost certain that the fraction of persons who act a certain way will fall within a narrow range of proportions) as potentially ‘credible’, but predictions for individuals as meaningless.

If the purpose of risk assessment is to make choices, then well-grounded probabilistic predictions about single events help us. Suppose we conclude that it is legally and ethically acceptable to impose preventive confinement upon individuals in ARAI categories with estimated recidivism rates above a specified threshold. This policy entails making ‘false-negative’ and ‘false-positive’ decision errors. We recognise, however, that unless we are omniscient perfection is not an option and ARAIs simply help us make better decisions than we otherwise could.

How do ‘margins of error’ in estimated recidivism rates affect our decision process? Hart et al believe their ‘group risk’ and ‘individual risk’ 95% confidence intervals speak to this problem. Their group intervals are standard confidence intervals for estimated population proportions based on random samples. If the threshold lies outside the group risk confidence interval for a category, then we can be reasonably certain that a decision we make concerning someone in that category is the same decision we would make if we knew the true recidivism rate for that category. If the threshold falls within a category's group risk confidence interval, then our estimate quite possibly might lead to the ‘wrong’ decision. Statistical decision theory (Reference BergerBerger, 1985) shows, however, that it is still a sensible strategy to choose whether to confine a member of a category based on which side of the threshold our estimated risk falls.

Hart et al talk about ‘individual risk’ as though it is something different from category (or ‘group’) risk. Yet if all one knows about an individual is his membership of a risk group, what can ‘individual risk’ mean? The authors do not say. If ‘individual risk’ refers to believed-to-exist-but-unspecified differences between individuals within a category, such differences should not affect choices by a rational decision-maker. The 95% CIs for ‘individual risk’ pile nonsense on top of meaninglessness. Hart et al describe the replacement of ‘n’ by ‘1’ in the Wilson (Reference Wilson1927) formulae as ‘ad hoc’, but this substitution makes no sense when the basis for the estimated proportion is an n-member sample. With ‘1’ in place of ‘n’, the formulae just don't mean anything.

Using ARAIs raises serious moral problems as well as the valid scientific questions that Hart et al mention. But in faulting the capacity of ARAIs to address an unspecified quantity called ‘individual risk’, and in dressing up this notion with misapplied formulae for confidence intervals, Hart et al ultimately create a muddle.

References

Berger, J. O. (1985) Statistical Decision Theory and Bayesian Analysis (2nd edn). Springer-Verlag.CrossRefGoogle Scholar
Hart, S. D., Michie, C. & Cooke, D. J. (2007) Precision of actuarial risk assessment instruments. Evaluating the ‘margins of error’ of group v. individual predictions of violence. British Journal of Psychiatry, 190 (suppl. 49), s60s65.Google Scholar
Wilson, E. B. (1927) Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22, 209212.Google Scholar
Submit a response

eLetters

No eLetters have been published for this article.