Introduction

In this chapter, we will cover the basic statistics testers use for describing and revising criterion-referenced tests. Although this is a book about CRT, the chapter will include a fairly detailed discussion of NRT statistics as well. This is done in order to provide a foundation against which to compare the differing orientations utilized by each. We feel this is important in order for the reader to put the kinds of issues each approach attempts to accommodate into perspective. The chapter first discusses assumptions that each approach makes about the distributions of test scores, and presents basic concepts of descriptive statistics. It then turns to details of item analysis. This examination should make clear that NRT item analyses are designed to help achieve a test which distributes examinees across a scale whereas the CRT item analyses are more concerned with finding items that distribute examinees into known or predicted categories according to their knowledge of the domain criteria.

Since tests are made up of units called items, the chapter will examine the types of item-related analyses that are used for the two basic families of tests. For NRTs, the techniques described here for developing, analyzing, selecting, and improving items will include item format analysis, item facility, and item discrimination indices, as well as distractor efficiency analysis. For CRTs, some of the same analyses will often be used plus others: a focus on item quality analysis, an index which compares item performance of masters and non-masters (called the difference index), and three statistics that are based on whether students passed or failed the test, called the B-index, agreement statistic, and item phi (ϕ).