Abstract. Proportional-hazards models are frequently used to analyze data from randomized controlled trials. This is a mistake. Randomization does not justify the models, which are rarely informative. Simpler methods work better. This discussion is salient because the misuse of survival analysis has introduced a new hazard in epidemiology: It can lead to serious mistakes in medical treatment. Life tables, Kaplan-Meier curves, and proportional-hazards models, aka “Cox models,” all require strong assumptions, such as stationarity of mortality and independence of competing risks. Where the assumptions fail, the methods also tend to fail. Justifying those assumptions is fraught with difficulty. This is illustrated with examples: the impact of religious feelings on survival and the efficacy of hormone replacement therapy. What are the implications for statistical practice? With observational studies, the models could help disentangle causal relations if the assumptions behind the models can be justified.

In this chapter, I will discuss life tables and Kaplan-Meier estimators, which are similar to life tables. Then I turn to proportional-hazards models, aka “Cox models.” Along the way, I will look at the efficacy of screening for lung cancer, the impact of negative religious feelings on survival, and the efficacy of hormone replacement therapy.

What are the conclusions about statistical practice? Proportional-hazards models are frequently used to analyze data from randomized controlled trials.

Abstract. Regression models have been used in the social sciences at least since 1899, when Yule published a paper on the causes of pauperism. Regression models are now used to make causal arguments in a wide variety of applications, and it is perhaps time to evaluate the results. No definitive answers can be given, but this chapter takes a rather negative view. Snow's work on cholera is presented as a success story for scientific reasoning based on nonexperimental data. Failure stories are also discussed, and comparisons may provide some insight. In particular, this chapter suggests that statistical technique can seldom be an adequate substitute for good design, relevant data, and testing predictions against reality in a variety of settings.

Introduction

Regression models have been used in social sciences at least since 1899, when Yule published his paper on changes in “out-relief” as a cause of pauperism: He argued that providing income support outside the poorhouse increased the number of people on relief. At present, regression models are used to make causal arguments in a wide variety of social science applications, and it is perhaps time to evaluate the results.

A crude four-point scale may be useful:

1. Regression usually works, although it is (like anything else) imperfect and may sometimes go wrong.

2. Regression sometimes works in the hands of skillful practitioners, but it isn't suitable for routine use.

3. Regression might work, but it hasn't yet.

4. Regression can't work.

Abstract. Making sense of earthquake forecasts is surprisingly difficult. In part, this is because the forecasts are based on a complicated mixture of geological maps, rules of thumb, expert opinion, physical models, stochastic models, and numerical simulations, as well as geodetic, seismic, and paleoseismic data. Even the concept of probability is hard to define in this context. For instance, the U.S. Geological Survey developed a probability model according to which the chance of an earthquake of magnitude 6.7 or greater before the year 2030 in the San Francisco Bay Area is 0.7 ± 0.1. How is that to be understood? Standard interpretations of probability cannot be applied. Despite their careful work, the USGS probability estimate is shaky, as is the uncertainty estimate.

Introduction

What is the chance that an earthquake of magnitude 6.7 or greater will occur before the year 2030 in the San Francisco Bay Area? The U.S. Geological Survey estimated the chance to be 0.7 ± 0.1 (USGS, 1999). In this chapter, we try to interpret such probabilities.

Making sense of earthquake forecasts is surprisingly difficult. In part, this is because the forecasts are based on a complicated mixture of geological maps, rules of thumb, expert opinion, physical models, stochastic models, numerical simulations, as well as geodetic, seismic, and paleoseismic data.

Abstract. The “Huber Sandwich Estimator” can be used to estimate the variance of the MLE when the underlying model is incorrect. If the model is nearly correct, so are the usual standard errors, and robustification is unlikely to help much. On the other hand, if the model is seriously in error, the sandwich may help on the variance side, but the parameters being estimated by the MLE are likely to be meaningless–except perhaps as descriptive statistics.

Introduction

This chapter gives an informal account of the so-called “Huber Sandwich Estimator,” for which Peter Huber is not to be blamed. We discuss the algorithm and mention some of the ways in which it is applied. Although the chapter is mainly expository, the theoretical framework outlined here may have some elements of novelty. In brief, under rather stringent conditions the algorithm can be used to estimate the variance of the MLE when the underlying model is incorrect. However, the algorithm ignores bias, which may be appreciable. Thus, results are liable to be misleading.

To begin the mathematical exposition, let i index observations whose values are yi. Let θ ∈ Rp be a p × 1 parameter vector. Let y → fi (y|θ) be a positive density. If yi takes only the values 0 or 1, which is the chief case of interest here, then fi (0|θ) > 0, fi (1|θ) > 0, and fi (0|θ) + fi (0|θ) + fi (1|θ) = 1. Some examples involve real- or vector-valued yi, and the notation is set up in terms of integrals rather than sums.

Abstract. Epidemiologic methods were developed to prove general causation: identifying exposures that increase the risk of particular diseases. Courts often are more interested in specific causation: On balance of probabilities, was the plaintiff's disease caused by exposure to the agent in question? Some authorities have suggested that a relative risk greater than 2.0 meets the standard of proof for specific causation. Such a definite criterion is appealing, but there are difficulties. Bias and confounding are familiar problems; and individual differences must also be considered. The issues are explored in the context of the swine flu vaccine and Guillain-Barré syndrome. The conclusion: There is a considerable gap between relative risks and proof of specific causation.

Introduction

This article discusses the role of epidemiologic evidence in toxic tort cases, especially, relative risk: Does a relative risk above 2.0 show specific causation? Relative risk compares groups in an epidemiologic study: One group is exposed to some hazard–like a toxic substance; the other “control” group is not exposed. For present purposes, relative risk is the ratio

RR = Observed/Expected.

The numerator in this fraction is the number of injuries observed in the exposed group. The expected number in the denominator is computed on the theory that exposure has no effect, so that injury rates in the exposed group should be the same as injury rates in the control group.

Abstract. After sketching the conflict between objectivists and subjectivists on the foundations of statistics, this chapter discusses an issue facing statisticians of both schools, namely, model validation. Statistical models originate in the study of games of chance and have been successfully applied in the physical and life sciences. However, there are basic problems in applying the models to social phenomena; some of the difficulties will be pointed out. Hooke's law will be contrasted with regression models for salary discrimination, the latter being a fairly typical application in the social sciences.

What is probability?

For a contemporary mathematician, probability is easy to define, as a countably additive set function on a σ-field, with a total mass of one. This definition, perhaps cryptic for non-mathematicians, was introduced by A. N. Kolmogorov around 1930, and has been extremely convenient for mathematical work; theorems can be stated with clarity, and proved with rigor.

Abstract. Regression adjustments are often made to experimental data to address confounders that may not be balanced by randomization. Since randomization does not justify the models, bias is likely; nor are the usual variance calculations to be trusted. Here, we evaluate regression adjustments using Neyman's non-parametric model. Previous results are generalized, and more intuitive proofs are given. A bias term is isolated, and conditions are given for unbiased estimation in finite samples.

Introduction

Data from randomized controlled experiments (including clinical trials) are often analyzed using regression models and the like. The behavior of the estimates can be calibrated using the non-parametric model in Neyman (1923), where each subject has potential responses to several possible treatments. Only one response can be observed, according to the subject's assignment; the other potential responses must then remain unobserved. Covariates are measured for each subject and may be entered into the regression, perhaps with the hope of improving precision by adjusting the data to compensate for minor imbalances in the assignment groups.

As discussed in Freedman (2006b [Chapter 17], 2008a), randomization does not justify the regression model, so that bias can be expected, and the usual formulas do not give the right variances. Moreover, regression need not improve precision. Here, we extend some of those results, with proofs that are more intuitive.

Drawing sound causal inferences from observational data is a central goal in social science. How to do so is controversial. Technical approaches based on statistical models—graphical models, non-parametric structural equation models, instrumental variable estimators, hierarchical Bayesian models, etc.—are proliferating. But David Freedman has long argued that these methods are not reliable. He demonstrated repeatedly that it can be better to rely on subject-matter expertise and to exploit natural variation to mitigate confounding and rule out competing explanations.

When Freedman first enunciated this position decades ago, many were skeptical. They found it hard to believe that a probabilist and mathematical statistician of his stature would favor “low-tech” approaches. But the tide is turning. An increasing number of social scientists now agree that statistical technique cannot substitute for good research design and subject-matter knowledge. This view is particularly common among those who understand the mathematics and have on-the-ground experience.

Historically, “shoe-leather epidemiology” is epitomized by intensive, door-to-door canvassing that wears out investigators' shoes. In contrast, advocates of statistical modeling sometimes claim that their methods can salvage poor research design or low-quality data. Some suggest that their algorithms are general-purpose inference engines: Put in data, turn the crank, out come quantitative causal relationships, no knowledge of the subject required.

This is tantamount to pulling a rabbit from a hat. Freedman's conservation of rabbits principle says “to pull a rabbit from a hat, a rabbit must first be placed in the hat.” In statistical modeling, assumptions put the rabbit in the hat.

Abstract. The logit model is often used to analyze experimental data. However, randomization does not justify the model, so the usual estimators can be inconsistent. A consistent estimator is proposed. Neyman's non-parametric setup is used as a benchmark. In this setup, each subject has two potential responses, one if treated and the other if untreated; only one of the two responses can be observed. Beside the mathematics, there are simulation results, a brief review of the literature, and some recommendations for practice.

Introduction

The logit model is often fitted to experimental data. As explained below, randomization does not justify the assumptions behind the model. Thus, the conventional estimator of log odds is difficult to interpret; an alternative will be suggested. Neyman's setup is used to define parameters and prove results. (Grammatical niceties apart, the terms “logit model” and “logistic regression” are used interchangeably.)

After explaining the models and estimators, we present simulations to illustrate the findings. A brief review of the literature describes the history and current usage. Some practical recommendations are derived from the theory. Analytic proofs are sketched at the end of the chapter.

David A. Freedman presents in this book the foundations of statistical models and their limitations for causal inference. Examples, drawn from political science, public policy, law, and epidemiology, are real and important.

A statistical model is a set of equations that relate observable data to underlying parameters. The parameters are supposed to characterize the real world. Formulating a statistical model requires assumptions. Rarely are those assumptions tested. Indeed, some are untestable in principle, as Freedman shows in this volume. Assumptions are involved in choosing which parameters to include, the functional relationship between the data and the parameters, and how chance enters the model. It is common to assume that the data are a simple function of one or more parameters, plus random error. Linear regression is often used to estimate those parameters. More complicated models are increasingly common, but all models are limited by the validity of the assumptions on which they ride.

Freedman's observation that statistical models are fragile pervades this volume. Modeling assumptions—rarely examined or even enunciated—fail in ways that undermine model-based causal inference. Because of their unrealistic assumptions, many new techniques constitute not progress but regress. Freedman advocates instead “shoe leather” methods, which identify and exploit natural variation to mitigate confounding and which require intimate subject-matter knowledge to develop appropriate research designs and eliminate rival explanations.