Most of the literature on cost–effectiveness, cost–benefit, and program evaluation has focused either on what should in principle be included in costs, benefits, and effects of programs or on the empirical implementation for specific treatments and programs. The rest of this book discusses many of those issues. In Chapter 2, Allan Detsky examined issues related to the evidence of effectiveness for a treatment based on the nature of a study's design. Among other issues, he discussed the relative merits of randomized controlled trials (RCTs) and various observational study designs. In Chapter 4, David Dranove examined issues of measuring economic or opportunity costs in cost–effectiveness studies. In this chapter, we will concentrate on a number of statistical issues that arise in comparing alternative treatments. Part of our discussion provides the rationale for a preference for randomized over observational studies. The majority of our discussion, however, applies in differing degrees to RCTs and observational studies.

Our basic concern is that the statistical modeling or analysis must mirror an understanding about how the behavior came about and what process generated the data, whether the data are observational or experimental. Observational behavior and data, especially in clinical or claims databases, did not come about at random. One has to understand the process that generated the treatment choices by physicians, the care seeking and compliance of patients, and the data reported to adequately understand whether the observed associations among the treatment, costs, and outcomes are really causal or merely an artifact of the correlation between the treatment and some other latent or unobservable causal factor.