The validity of conclusions drawn from specific research studies must be evaluated in light of the purposes for which the research was undertaken. We distinguish four general types of research: description and point estimation, correlation and prediction, causal inference, and explanation. For causal and explanatory research, internal validity is critical – the extent to which a causal relationship can be inferred from the results of variation in the independent and dependent variables of an experiment. Random assignment is discussed as the key to avoiding threats to internal validity. Internal validity is distinguished from construct validity (the relationship between a theoretical construct and the methods used to operationalize that concept) and external validity (the extent to which the results of a research study can be generalized to other contexts). Construct validity is discussed in terms of multiple operations and discriminant and convergent validity assessment. External validity is discussed in terms of replicability, robustness, and relevance of specific research findings.

This chapter examines the generalizability of the book’s main argument. It synthesizes the conclusions of other studies on the consequences of three similar episodes of forced migration in the twentieth century: the Greek-Turkish population exchange, the Partition of India, and the repatriation of Pied-Noirs to France. It then considers ways in which the argument can be extended to other cases of forced and voluntary migration.

Field experiments have a long tradition in some areas of the social and behavioral sciences and have become increasingly popular in sociology. Field experiments are staged in "natural" research settings where individuals usually interact in everyday life and regularly complete the task under investigation. The implementation in the field is the core feature distinguishing the approach from laboratory experiments. It is also one of the major reasons why researchers use field experiments; they allow incorporating social context, investigating subjects under "natural" conditions, and collecting unobtrusive measures of behavior. However, these advantages of field experiments come at the price of reduced control. In contrast to the controlled setting of the laboratory, many factors can influence the outcome but are not under the experimenter’s control and are often hard to measure in the field. Using field experiments on the broken windows theory, the strengths and potential pitfalls of experimenting in the field are illustrated. The chapter also covers the nascent area of digital field experiments, which share key features with other types of experiments but offer exciting new ways to study social behavior by enabling the collection large-scale data with fine-grained and unobtrusive behavioral measures at relatively low variable costs.

Peer review supports decisions related to publications, grant proposals, awards, or personnel selection. Independent of the specific occasion, we propose validity as a chief evaluation criterion for reviews. While applicable to all occasions, the principles of validity-oriented quality control are particularly suited for journal reviews. Beyond evaluating validity and the scientific potential of a given piece of research, we address how peer reviewing serves important functions and is accountable for the growth of science at a more superordinate level. We also provide guidelines and concrete recommendations for how a good peer review may serve these functions. Good peer review, thereby, fosters both the advancement of scientific research and the quality, precision, and sincerity of the scientific literature. The end of the chapter is devoted to a core set of good reviewer practices, conceived as an essential feature of academic culture.

We discuss the single embedded case study design in this chapter. We deliberate how this design is different from multiple and single holistic designs in terms of the levels of analysis and the nature of replication. The selection rationale and sampling are discussed next. Afterwards, we move on to the longitudinal and/or cross-sectional single embedded designs. The strengths and the weaknesses of the design in terms of internal validity, external validity, and the number of variables are discussed subsequently. This chapter also discusses the (mis)conception regarding longitudinal designs and temporal embedded units.

We introduce and define the single holistic case study design in this chapter. The strengths of the design are discussed in detail, with examples. In particular, we discuss the potential of single holistic design in providing a detailed explanation of processes. Single holistic case studies also explore the theorizing potential of unique cases which hold the potential to reveal new dimensions of a phenomenon or falsify/refute an existing theory. Relatively high data access, construct validity, potential to include an unlimited number of variables, etc., are some other strengths that we discuss. The weaknesses of the design (i.e. low internal and external validity) are discussed afterwards. The chapter also addresses some common (mis)conceptions regarding single holistic designs and their external validity.

The final chapter in this book discusses some methodological considerations and debates surrounding case study research and its quality. In particular, we revisit the topic of research paradigms (i.e. positivism and interpretivism). Relatedly, we discuss different quality criteria as proposed by prior researchers from both paradigmatic camps. In particular, we focus on the rigor versus trustworthiness discussion and the internal versus external validity debate. Afterwards, we briefly discuss the iterative cycles of data collection and analysis one would encounter during a qualitative case study research process. We end the chapter (and subsequently the book) with a guiding framework which will help researchers in sequencing case study designs by acknowledging the weaknesses of individual designs and leveraging their strengths. The framework can be adopted and adapted to suit the specific research objectives of the study in hand.

We discuss multiple case studies in this chapter. We start off with a discussion of theoretical sampling and replication logic. We specifically discuss literal and theoretical replication (LR and TR) in connection with multiple case studies. The strengths and limitations of LR and TR are discussed thereafter. In particular, we deliberate upon the potential of TR to enhance the internal and external validity of a case study. Henceforth, we address some common (mis)conceptions regarding replication logic, internal validity, external validity (generalizability), and reliability. We also discuss how multiple case studies might need to sacrifice the depth of observation for breadth. Other potential weaknesses, such as the smaller number of independent variables and the difficulty in controlling context, are also discussed thereafter.

This chapter provides a comprehensive overview of the verbal-guise technique (henceforth VGT), a variant of the speaker evaluation paradigm in which guises representing different language varieties are produced by different speakers, each speaking in their habitual language variety. First, key features of the VGT are discussed. Second, a brief historical sketch of the technique’s introduction, development, and proliferation in language attitudes research is offered. Third, key advantages (e.g. speaker authenticity, ease of implementation) and disadvantages (e.g. lack of full experimental control) of the technique are reviewed. Fourth, various practical considerations surrounding research planning and design are described and several recommendations are offered, including: (a) matching speakers on various demographic factors (e.g. sex, age), (b) matching speakers on extraneous vocal characteristics not of interest in the study (e.g. pitch), and (c) using multiple speakers to represent each variety of interest. Fifth, the main concerns surrounding the analysis and interpretation of data obtained using the VGT are discussed. Finally, a brief sketch is provided of the methodological considerations that were involved in designing a recent study utilising the VGT, which examined Americans’ attitudes toward standard American English and nine foreign accents: Arabic, Farsi, French, German, Hindi, Hispanic, Mandarin, Russian, and Vietnamese.

Woolcock focuses on the utility of qualitative case studies for addressing the decision-maker’s perennial external validity concern: What works there may not work here. He asks how to generate the facts that are important in determining whether an intervention can be scaled and replicated in a given setting. He focuses our attention on three categories: 1) causal density, 2) implementation capability, and 3) reasoned expectations about what can be achieved by when. Experiments are helpful for sorting out causally simple outcomes like the impact of deworming, but they are less insightful when there are many causal pathways, feedback loops, and exogenous influences. Nor do they help sort out the effect of mandate, management capacity, and supply chains, or the way results will materialize – whether some will materialize before others or increase or dissipate over time. Analytic case studies, Woolcock argues, are the main method available for assessing the generalizability of any given intervention.

Achen aims to correct what he perceives as an imbalance in favor of randomized controlled trials – experiments – within contemporary social science. “The argument for experiments depends critically on emphasizing the central challenge of observational work – accounting for unobserved confounders – while ignoring entirely the central challenge of experimentation – achieving external validity,” he writes. Using the mathematics behind randomized controlled trials to make his point, he shows that once this imbalance is corrected, we are closer to Cartwright’s view (Chapter 2) than to the current belief that RCTs constitute the gold standard for good policy research. Achen concludes: “Causal inference of any kind is just plain hard. If the evidence is observational, patient consideration of plausible counterarguments, followed by the assembling of relevant evidence, can be, and often is, a painstaking process.” Well-structured qualitative case studies are one important tool; experiments, another.

Chapter 3 focuses on how to think when evaluating experiments. This includes a discussion of realism, particularly why mundane realism or resemblance to the “real world” receives far too much attention, as well as an overview of how to design experimental treatments. The chapter then turns to validity issues, offering a new way to think about external validity in assessing experiments. This includes a detailed discussion of sampling and why the onus should be more on critics of an experimental sample than on the experimentalist him/herself (i.e., to justify a sample).