Data gathering

If, as an evaluator, you choose to make use of a positivistic, quantitative design, one of the most important concerns you will have will be identifying an appropriate test or other instrument for measuring the effect of the program. Finding the appropriate test is further complicated when the evaluation design includes a control, or comparison, group. In that case, the test needs to be appropriate for the curricula of both the program and comparison group. As we saw in Chapter 2, a test of oral/aural skills will tend to show an audio-lingual program to be superior to a grammar-translation program, whereas a test of grammar and reading skills will tend to show the opposite. Tests that do not favor one program type over another may be insensitive to the types of effects that occur in either, and result in a finding of no differences where important differences actually exist.

As introduced in Chapter 4, there are two basic types of tests that can be drawn upon for the purposes of program evaluation: norm-referenced and criterion-referenced tests. Norm-referenced tests (NRTs) are what are usually referred to as standardized tests (cf. Beretta 1986c). They are usually developed using large samples of examinees, resulting in published statistical properties that are used for comparison across groups of examinees and administrations of the test. The key element to NRTs, however, is the fact that they are developed by selecting items that are successful at maximizing variability in the total test score distribution.