We are frequently faced with the problem of determining whether a result is significant. People in a region may seem to have a high rate of cancer, but is it out of the ordinary? Are children much more likely to develop autism after being vaccinated? Which genes are differentially expressed in tumor and normal cells? In most cases, we end up with a numerical result, and must determine a threshold at which to call it significant. As we will see shortly, this becomes more complicated when we test many hypotheses at once, as it is then likelier that we will observe something that seems significant by chance. This chapter reviews the basic concepts in and approaches to evaluating statistical tests. We begin with the simplest case of a single result before discussing the modifications needed when doing many tests at once.

Preliminaries

Say we want to determine whether or not a coin is fair, so we flip it 10 times. Our assumption is that the coin is fair, meaning that the probability of heads (H) (or tails (T)) on any given flip is 1/2. However, the sequence we observe is 9 heads and 1 tail. We then want to determine how likely it is that this would occur given our initial hypothesis (called the null hypothesis) that the coin is fair. The basic concept is that we attempt to determine whether the results conform to the null hypothesis, usually denoted H0, that there is no difference, or whether the observations do deviate significantly, and might be more plausibly explained by an alternative hypothesis, usually denoted H1.