Experiments often include multiple treatments, with the primary goal to compare the causal effects of those treatments. This study focuses on comparing the causal anatomies of multiple treatments through the use of causal mediation analysis. It proposes a novel set of comparative causal mediation (CCM) estimands that compare the mediation effects of different treatments via a common mediator. Further, it derives the properties of a set of estimators for the CCM estimands and shows these estimators to be consistent (or conservative) under assumptions that do not require the absence of unobserved confounding of the mediator–outcome relationship, which is a strong and nonrefutable assumption that must typically be made for consistent estimation of individual causal mediation effects. To illustrate the method, the study presents an original application investigating whether and how the international legal status of a foreign policy commitment can increase the domestic political “audience costs” that democratic governments suffer for violating such a commitment. The results provide novel evidence that international legalization can enhance audience costs via multiple causal channels, including by amplifying the perceived immorality of violating the commitment.

A common causal identification strategy in political science is selection on observables. This strategy assumes one observes a set of covariates that is, after statistical adjustment, sufficient to make treatment status as-if random. Under adjustment methods such as matching or inverse probability weighting, coefficients for control variables are treated as nuisance parameters and are not directly estimated. This is in direct contrast to regression approaches where estimated parameters are obtained for all covariates. Analysts often find it tempting to give a causal interpretation to all the parameters in such regression models—indeed, such interpretations are often central to the proposed research design. In this paper, we ask when we can justify interpreting two or more coefficients in a regression model as causal parameters. We demonstrate that analysts must appeal to causal identification assumptions to give estimates causal interpretations. Under selection on observables, this task is complicated by the fact that more than one causal effect might be identified. We show how causal graphs provide a framework for clearly delineating which effects are presumed to be identified and thus merit a causal interpretation, and which are not. We conclude with a set of recommendations for how researchers should interpret estimates from regression models when causal inference is the goal.

An increasing number of studies exploit the occurrence of unexpected events during the fieldwork of public opinion surveys to estimate causal effects. In this paper, we discuss the use of this identification strategy based on unforeseen and salient events that split the sample of respondents into treatment and control groups: the Unexpected Event during Survey Design. In particular, we focus on the assumptions under which unexpected events can be exploited to estimate causal effects and we discuss potential threats to identification, paying especial attention to the observable and testable implications of these assumptions. We propose a series of best practices in the form of various estimation strategies and robustness checks that can be used to lend credibility to the causal estimates. Drawing on data from the European Social Survey, we illustrate the discussion of this method with an original study of the impact of the Charlie Hebdo terrorist attacks (Paris, 01/07/2015) on French citizens’ satisfaction with their national government.

Difference-in-differences is a widely used evaluation strategy that draws causal inference from observational panel data. Its causal identification relies on the assumption of parallel trends, which is scale-dependent and may be questionable in some applications. A common alternative is a regression model that adjusts for the lagged dependent variable, which rests on the assumption of ignorability conditional on past outcomes. In the context of linear models, Angrist and Pischke (2009) show that the difference-in-differences and lagged-dependent-variable regression estimates have a bracketing relationship. Namely, for a true positive effect, if ignorability is correct, then mistakenly assuming parallel trends will overestimate the effect; in contrast, if the parallel trends assumption is correct, then mistakenly assuming ignorability will underestimate the effect. We show that the same bracketing relationship holds in general nonparametric (model-free) settings. We also extend the result to semiparametric estimation based on inverse probability weighting. We provide three examples to illustrate the theoretical results with replication files in Ding and Li (2019).

A growing literature has argued that electoral turnout decreases the more government policy constrained by economic and institutional factors. This paper investigates whether a certain type of policy constraint, fiscal rules, lowers turnout. Since fiscal rules set limits for government fiscal policy, they should lower the incentive for citizens to participate electorally. However, using parliamentary turnout data in a large panel of democratic countries, little robust evidence is found in favor of fiscal rules having a depressing effect on electoral turnout. Analysis of European individual-level data also suggests that national fiscal rules do not affect inequality in electoral turnout between income groups either. Difference-in-discontinuity evidence from Italian municipalities further suggests that the results are causally identified.

Dropping subjects based on the results of a manipulation check following treatment assignment is common practice across the social sciences, presumably to restrict estimates to a subpopulation of subjects who understand the experimental prompt. We show that this practice can lead to serious bias and argue for a focus on what is revealed without discarding subjects. Generalizing results developed in Zhang and Rubin (2003) and Lee (2009) to the case of multiple treatments, we provide sharp bounds for potential outcomes among those who would pass a manipulation check regardless of treatment assignment. These bounds may have large or infinite width, implying that this inferential target is often out of reach. As an application, we replicate Press, Sagan, and Valentino (2013) with a design that does not drop subjects that failed the manipulation check and show that the findings are likely stronger than originally reported. We conclude with suggestions for practice, namely alterations to the experimental design.

Does exposure to the refugee crisis fuel support for extreme-right parties? Despite heated debates about the political repercussions of the refugee crisis in Europe, there exists very little—and sometimes conflicting—evidence with which to assess the impact of a large influx of refugees on natives’ political attitudes and behavior. We provide causal evidence from a natural experiment in Greece, where some Aegean islands close to the Turkish border experienced sudden and drastic increases in the number of Syrian refugees while other islands slightly farther away—but with otherwise similar institutional and socioeconomic characteristics—did not. Placebo tests suggest that precrisis trends in vote shares for exposed and nonexposed islands were virtually identical. This allows us to obtain unbiased estimates of the electoral consequences of the refugee crisis. Our study shows that among islands that faced a massive but transient inflow of refugees passing through just before the September 2015 election, vote shares for Golden Dawn, the most extreme-right party in Europe, moderately increased by 2 percentage points (a 44 percent increase at the average). The finding that mere exposure to the refugee crisis is sufficient to fuel support for extreme-right parties has important implications for our theoretical understanding of the drivers of antirefugee backlash.

Determining (1) the direction of causation and (2) the size of causal effects between two constructs is a central challenge of the scientific study of humans. In the early 1990s, researchers in behavioral genetics invented what was termed the direction of causation (DoC) model to address exactly these two concerns. The model claims that for any two traits whose mode of inheritance is sufficiently different, the direction of causation can be ascertained using a sufficiently large genetically informative sample. Using a series of simulation studies, we demonstrate a major challenge to the DoC model, namely that it is extremely sensitive to even tiny amounts of non-shared confounding. Even under ideal conditions for the DoC model (a large sample, N = 10,000), a large causal relationship (e.g., a causal correlation of .50) with very different modes of inheritance between the two traits (e.g., a pure AE model for one trait and a pure CE model for another trait) and a modest degree (correlation of .10) of non-shared confounding between the two traits results in the choice of the wrong causal models and estimating the wrong causal effects.

The impact that providing care to ageing parents has on adult children's lives may depend on the long-term care (LTC) context. A common approach to test this is to compare whether the impact of care-giving varies between countries with different LTC coverage. However, this approach leaves considerable room for omitted variable bias. We use individual fixed-effects analyses to reduce bias in the estimates of the effects of informal care-giving on quality of life, and combine this with a difference-in-difference approach to reduce bias in the estimated moderating impact of LTC coverage on these effects. We draw on longitudinal data for Sweden and Denmark from the Survey of Health, Ageing and Retirement in Europe (SHARE) collected between 2004 and 2015. Both countries traditionally had generous LTC coverage, but cutbacks were implemented at the end of the 20th century in Sweden and more recently in Denmark. We use this country difference in the timing of the cutbacks to shed light on effects of LTC coverage on the impact care-giving has on quality of life. Our analyses show that care-giving was more detrimental for quality of life in Sweden than in Denmark, and this difference weakened significantly when LTC coverage was reduced in Denmark, but not in Sweden. This suggests that LTC coverage shapes the impact of care-giving on quality of life.

Political scientists are increasingly interested in causal mediation, and to this end, recent studies focus on estimating a quantity called the controlled direct effect (CDE). The CDE measures the strength of the causal relationship between a treatment and outcome when a mediator is fixed at a given value. To estimate the CDE, Joffe and Greene (2009) and Vansteelandt (2009) developed the method of sequential g-estimation, which was introduced to political science by Acharya, Blackwell, and Sen (2016). In this letter, we propose an alternative method called “regression-with-residuals” (RWR) for estimating the CDE. In special cases, we show that these two methods are algebraically equivalent. Yet, unlike sequential g-estimation, RWR can easily accommodate several types of effect moderation, including cases in which the effect of the mediator on the outcome is moderated by a posttreatment confounder. Although common in the social sciences, this type of effect moderation is typically assumed away in applications of sequential g-estimation, which may lead to bias if effect moderation is in fact present. We illustrate RWR by estimating the CDE of negative media framing on public support for immigration, controlling for respondent anxiety.

Researchers who generate data often optimize efficiency and robustness by choosing stratified over simple random sampling designs. Yet, all theories of inference proposed to justify matching methods are based on simple random sampling. This is all the more troubling because, although these theories require exact matching, most matching applications resort to some form of ex post stratification (on a propensity score, distance metric, or the covariates) to find approximate matches, thus nullifying the statistical properties these theories are designed to ensure. Fortunately, the type of sampling used in a theory of inference is an axiom, rather than an assumption vulnerable to being proven wrong, and so we can replace simple with stratified sampling, so long as we can show, as we do here, that the implications of the theory are coherent and remain true. Properties of estimators based on this theory are much easier to understand and can be satisfied without the unattractive properties of existing theories, such as assumptions hidden in data analyses rather than stated up front, asymptotics, unfamiliar estimators, and complex variance calculations. Our theory of inference makes it possible for researchers to treat matching as a simple form of preprocessing to reduce model dependence, after which all the familiar inferential techniques and uncertainty calculations can be applied. This theory also allows binary, multicategory, and continuous treatment variables from the outset and straightforward extensions for imperfect treatment assignment and different versions of treatments.

Researchers investigating causal mechanisms in survey experiments often rely on nonrandomized quantities to isolate the indirect effect of treatment through these variables. Such an approach, however, requires a “selection-on-observables” assumption, which undermines the advantages of a randomized experiment. In this paper, we show what can be learned about casual mechanisms through experimental design alone. We propose a factorial design that provides or withholds information on mediating variables and allows for the identification of the overall average treatment effect and the controlled direct effect of treatment fixing a potential mediator. While this design cannot identify indirect effects on its own, it avoids making the selection-on-observable assumption of the standard mediation approach while providing evidence for a broader understanding of causal mechanisms that encompasses both indirect effects and interactions. We illustrate these approaches via two examples: one on evaluations of US Supreme Court nominees and the other on perceptions of the democratic peace.

Survey experiments often manipulate the description of attributes in a hypothetical scenario, with the goal of learning about those attributes’ real-world effects. Such inferences rely on an underappreciated assumption: experimental conditions must be information equivalent (IE) with respect to background features of the scenario. IE is often violated because subjects, when presented with information about one attribute, update their beliefs about others too. Labeling a country “a democracy,” for example, affects subjects’ beliefs about the country’s geographic location. When IE is violated, the effect of the manipulation need not correspond to the quantity of interest (the effect of beliefs about the focal attribute). We formally define the IE assumption, relating it to the exclusion restriction in instrumental-variable analysis. We show how to predict IE violations ex ante and diagnose them ex post with placebo tests. We evaluate three strategies for achieving IE. Abstract encouragement is ineffective. Specifying background details reduces imbalance on the specified details and highly correlated details, but not others. Embedding a natural experiment in the scenario can reduce imbalance on all background beliefs, but raises other issues. We illustrate with four survey experiments, focusing on an extension of a prominent study of the democratic peace.

The popularity of online surveys has increased the prominence of using sampling weights to enhance claims of representativeness. Yet, much uncertainty remains regarding how these weights should be employed in survey experiment analysis: should they be used? If so, which estimators are preferred? We offer practical advice, rooted in the Neyman–Rubin model, for researchers working with survey experimental data. We examine simple, efficient estimators, and give formulas for their biases and variances. We provide simulations that examine these estimators as well as real examples from experiments administered online through YouGov. We find that for examining the existence of population treatment effects using high-quality, broadly representative samples recruited by top online survey firms, sample quantities, which do not rely on weights, are often sufficient. We found that sample average treatment effect (SATE) estimates did not appear to differ substantially from their weighted counterparts, and they avoided the substantial loss of statistical power that accompanies weighting. When precise estimates of population average treatment effects (PATE) are essential, we analytically show poststratifying on survey weights and/or covariates highly correlated with outcomes to be a conservative choice. While we show substantial gains in simulations, we find limited evidence of them in practice.

Survey experiments have become a central methodology across the social sciences. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, due to the expense of population-based samples, much research relies on convenience samples (e.g. students, online opt-in samples). The emergence of affordable, but non-representative online samples has reinvigorated debates about the external validity of experiments. We conduct two studies of how experimental treatment effects obtained from convenience samples compare to effects produced by population samples. In Study 1, we compare effect estimates from four different types of convenience samples and a population-based sample. In Study 2, we analyze treatment effects obtained from 20 experiments implemented on a population-based sample and Amazon's Mechanical Turk (MTurk). The results reveal considerable similarity between many treatment effects obtained from convenience and nationally representative population-based samples. While the results thus bolster confidence in the utility of convenience samples, we conclude with guidance for the use of a multitude of samples for advancing scientific knowledge.