Henceforth it will be assumed as proven that an adequate theory of testing must consider not only the statistical hypothesis under test, but also rivals to it. This may be common-sense: ‘don't reject something unless you've something better’. It involves a conception now becoming general in the philosophy of science, and which is currently striving to oust the former idea that an hypothesis could be rejected independently of what other theories are available.

Even if this kind of attitude be agreeable, it is far from evident how rival hypotheses are to be weighed. There may be, as has already been warned in the last chapter, radically different kinds of test. The sequel will show that controversy between classical theories of testing stems in part from their trying to answer different questions. The most famous theory, due to Neyman and Pearson, will be shown to have very limited application. For the present I shall continue to regard the central problem as that of tests which can imply, on the basis of experimental results, whether or not an individual hypothesis should be rejected. Tests thus divide possible experimental results into two classes, those which suffice to reject the hypothesis, and those which do not. The former class of results will be called the rejection class.

Likelihood tests

Our theory of support leads directly to the theory of testing suggested in the last chapter. An hypothesis should be rejected if and only if there is some rival hypothesis much better supported than it is. Support has already been analysed in terms of ratios of likelihoods. But what shall serve as ‘much better supported’? For the present I leave this in abeyance, and speak merely of tests of different stringency. With each test will be associated a critical ratio. The greater the critical ratio, the more stringent the test. Roughly speaking hypothesis h will be rejected in favour of rival i at critical level ∝, just if the likelihood ratio of i to h exceeds ∝.

Hence suppose we have data stating that the true distribution on trials of some kind is a member of the class ∆. By a simple statistical hypothesis we mean one of the form, ‘The true distribution is D’, where D will be in ∆.