A recent debate on implicit measures of racial attitudes has focused on the relative roles of the person, the situation, and their interaction in determining the measurement outcomes. The chapter describes process models for assessing the roles of the situation and the person-situation interaction on the one hand and stable person-related components on the other hand in implicit measures. Latent state-trait models allow one to assess to what extent the measure is a reliable measure of the person and/or the situation and the person-situation interaction (Steyer, Geiser, & Fiege, 2012). Moreover, trait factor scores as well as situation-specific residual factor scores can be computed and related to third variables, thereby allowing one to assess to what extent the implicit measure is a valid measure of the person and/or the situation and the person-situation interaction. These methods are particularly helpful when combined with a process decomposition of implicit-measure data such as a diffusion-model analysis of the IAT (Klauer, Voss, Schmitz, & Teige-Mocigemba, 2007).

Adequate measurement of psychological phenomena is a fundamental aspect of theory construction and validation. Forming composite scales from individual items has a long and honored tradition, although, for predictive purposes, the power of using individual items should be considered. We outline several fundamental steps in the scale construction process, including (1) choosing between prediction and explanation; (2) specifying the construct(s) to measure; (3) choosing items thought to measure these constructs; (4) administering the items; (5) examining the structure and properties of composites of items (scales); (6) forming, scoring, and examining the scales; and (7) validating the resulting scales.

Without doubt the validity of scientific theories and their usability for solving societal, economic, ecological, and health-related problems are contingent on the existence of robust and replicable empirical findings. However, a review of the recently published replication literature portrays a rather pessimistic picture of the replicability of even very prominent empirical results, as is evident both from large-scale meta-analytic replication projects and from distinct attempts to replicate selected examples of well-established key findings of personality and social psychology. The present chapter offers a twofold explanation for this undesirable state of affairs. On one hand, the widespread evidence on replication failures reflects to a considerable extent the neglect of logical and methodological standards in replication studies, which sorely ignore such essential issues as manipulation checks, reliability control, regressive shrinkage, and intricacies of multi-causality. On the other hand, however, the community of behavioral scientists must blame themselves for an intrinsic weakness of their corporate identity and their incentive and publication system, namely the tendency to mistake sexy and unexpected findings for original insights and neglect of the assets of cumulative science and theoretical constraints.

This chapter is written for conversation analysts and is methodological. It discusses, in a step-by-step fashion, how to code practices of action (e.g., particles, gaze orientation) and/or social actions (e.g., inviting, information seeking) for purposes of their statistical association in ways that respect conversation-analytic (CA) principles (e.g., the prioritization of social action, the importance of sequential position, order at all points, the relevance of codes to participants). As such, this chapter focuses on coding as part of engaging in basic CA and advancing its findings, for example as a tool of both discovery and proof (e.g., regarding action formation and sequential implicature). While not its main focus, this chapter should also be useful to analysts seeking to associate interactional variables with demographic, social-psychological, and/or institutional-outcome variables. The chapter’s advice is grounded in case studies of published CA research utilizing coding and statistics (e.g., those of Gail Jefferson, Charles Goodwin, and the present author). These case studies are elaborated by discussions of cautions when creating code categories, inter-rater reliability, the maintenance of a codebook, and the validity of statistical association itself. Both misperceptions and limitations of coding are addressed.

In Chapter 12, the author discusses approaches to judging the effectiveness of both criterion-referenced and norm-reference performance assessments. The chapter includes guidelines for helping create or evaluate performance assessments for particular contexts. It also describes how to use statistical techniques for the same purpose. The chapter presents actual ratings from a classroom-based group oral discussion test and shows how teachers used statistics to determine both score dependability and reliability. The author discusses how to calculate coefficient agreement to help determine the dependability of the assessment and Cronbach’s Alpha to help determine its reliability. A major point of the chapter is that when certain conditions exist, test users can exploit an assessment to determine test takers’ mastery of language criteria (criterion-referenced purpose) and to compare their abilities (norm-referenced purpose). The author provides an appendix that shows readers how to use Excel software to calculate the statistics in the chapter.

In Chapter 8, the author introduces readers to item-level statistics and Cronbach’s Alpha as an estimate of internal reliability, all of which are useful for analyzing and revising dichotomously scored language assessments. The chapter focuses on two item-level statistics: item facility, or how easy an item is for a group of test takers, and point-biserial, which provides an indication of how well an item measures the targeted language ability. The author shows readers how to calculate these values and interpret their meaning for particular language assessment contexts. The chapter concludes with a discussion of how to calculate and interpret reliability based on Cronbach’s Alpha. The author also provides guidelines for interpreting reliability estimates for various language assessment contexts. An appendix provides guidance for using Excel software to calculate these statistics.

In Chapter 5, the author lays out the principles of uniformity – the consistency of the setting, content, and scoring procedures – and reliability, which is the consistency and stability of assessment scores. The chapter details the aspects of setting, content, and scoring procedures in various assessment contexts and what measures test users should take to maintain their consistency across test takers and test administrations. The chapter includes a discussion of the role of test accommodations, and how researchers can balance the needs of test takers with the principles of uniformity and reliability. An important feature of the chapter is the introduction of assistive technologies, such as generative artificial intelligence for helping ensure uniformity in an assessment context. The chapter concludes with a brief introduction to the concepts of test-retest, parallel forms, and internal reliability.

The focus of the chapter is on turning concepts into measurable variables, also known as operationalization.Best practices in created variable based on the concept are covered.The differences between nominal, ordinal, and interval level variables are discussed along with their relevance for statistical tests later covered in the book.The importance of measurement validity and reliability and their implications for research provide another focus of the chapter.

To wit, we have three specific goals here. First, we want to review the activities of the three-hatted pollster. We do this to provide greater context for each type of pollster. Some of us are all three; others are some combination of these. Any pollster worth their salt must at least be a data scientist, or they risk losing credibility.

Second, we explore the role of the pollster in society. Ultimately, what is the purpose of the pollster? In our view, pollsters are critically important in any democracy. We believe this is often overlooked due to the ranking frenzy after every electoral cycle. Here, we put the profession into proper perspective.

And third, we discuss the use of non-survey, or alternative data, inputs as proxy measures for public opinion. We provide a framework for pollsters to think through them in a critical manner. Validation is a key concept which we introduce here – one more tool for the data scientist.