Hostname: page-component-8448b6f56d-cfpbc Total loading time: 0 Render date: 2024-04-24T14:26:05.064Z Has data issue: false hasContentIssue false
  • English
  • Français

When Natural Experiments Are Neither Natural nor Experiments

Published online by Cambridge University Press:  09 February 2012

JASJEET S. SEKHON  and
ROCÍO TITIUNIK
JASJEET S. SEKHON*
Affiliation:
University of California, Berkeley
ROCÍO TITIUNIK*
Affiliation:
University of Michigan
*
Jasjeet S. Sekhon is Associate Professor, Travers Department of Political Science and Department of Statistics, University of California at Berkeley, 210 Barrows Hall, #1950, Berkeley, CA 94720 (sekhon@berkeley.edu).
Rocío Titiunik is Assistant Professor, Department of Political Science and Faculty Associate, Center for Political Studies, University of Michigan, 426 Thompson Street, Ann Arbor, MI 48104 (titiunik@umich.edu).
Get access Rights & Permissions [Opens in a new window]

Abstract

Natural experiments help to overcome some of the obstacles researchers face when making causal inferences in the social sciences. However, even when natural interventions are randomly assigned, some of the treatment–control comparisons made available by natural experiments may not be valid. We offer a framework for clarifying the issues involved, which are subtle and often overlooked. We illustrate our framework by examining four different natural experiments used in the literature. In each case, random assignment of the intervention is not sufficient to provide an unbiased estimate of the causal effect. Additional assumptions are required that are problematic. For some examples, we propose alternative research designs that avoid these conceptual difficulties.

Type
Research Article
Information
American Political Science Review , Volume 106 , Issue 1 , February 2012 , pp. 35 - 57
Copyright
Copyright © American Political Science Association 2012

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Abrajano, Marisa A., Nagler, Jonathan, and Alvarez, R. Michael. 2005. “A Natural Experiment of Race-Based and Issue Voting: The 2001 City of Los Angeles Elections.” Political Research Quarterly 58 (2): 203–18.CrossRefGoogle Scholar
Ansolabehere, Stephen, Snyder, James M., and Stewart, Charles. 2000. “Old Voters, New Voters, and the Personal Vote: Using Redistricting to Measure the Incumbency Advantage.” American Journal of Political Science 44 (1): 17–34.CrossRefGoogle Scholar
Bhavnani, Rikhil R. 2009. “Do Electoral Quotas Work after They Are Withdrawn? Evidence from a Natural Experiment in India.” American Political Science Review 103 (1): 23–35.CrossRefGoogle Scholar
Butler, Daniel Mark. 2009. “A Regression Discontinuity Design Analysis of the Incumbency Advantage and Tenure in the U.S. House.” Electoral Studies 28 (2): 123–28.CrossRefGoogle Scholar
Carman, Christopher, Mitchell, James, and Johns, Robert. 2008. “The Unfortunate Natural Experiment in Ballot Design: The Scottish Parliamentary Elections of 2007.” Electoral Studies 27 (3): 442–59.CrossRefGoogle Scholar
Carson, Jamie L., Engstrom, Erik J., and Roberts, Jason M.. 2007. “Candidate Quality, the Personal Vote, and the Incumbency Advantage in Congress.” American Political Science Review 101 (2): 289–301.CrossRefGoogle Scholar
Caughey, Devin, and Sekhon, Jasjeet S.. 2011. “Elections and the Regression-Discontinuity Design: Lessons from Close U.S. House Races, 1942–2008.” Political Analysis 19 (4): 385–408.CrossRefGoogle Scholar
Cox, Gary W., and Katz, Jonathan N.. 2002. Elbridge Gerry's Salamander: The Electoral Consequences of the Reapportionment Revolution. New York: Cambridge University Press.CrossRefGoogle Scholar
Desposato, Scott W., and Petrocik, John R.. 2003. “The Variable Incumbency Advantage: New Voters, Redistricting, and the Personal Vote.” American Journal of Political Science 47 (1): 18–32.CrossRefGoogle Scholar
Diamond, Alexis, and Sekhon, Jasjeet S.. 2005. “Genetic Matching for Estimating Causal Effects: A General Multivariate Matching Method for Achieving Balance in Observational Studies.” Working Paper. http://sekhon.berkeley.edu/papers/GenMatch.pdf (accessed November 16, 2011).Google Scholar
Diamond, Jared, and Robinson, James A., eds. 2010. Natural Experiments of History. Cambridge, MA: Belknap Press.Google Scholar
Dunning, Thad. 2008. “Improving Causal Inference: Strengths and Limitations of Natural Experiments.” Political Science Quarterly 61 (2): 282–93.Google Scholar
Erikson, Robert S. 1971. “The Advantage of Incumbency in Congressional Elections.” Polity 3 (3): 395–405.CrossRefGoogle Scholar
Gelman, Andrew, and King, Gary. 1990. “Estimating Incumbency Advantage without Bias.” American Journal of Political Science 34 (4): 1142–64.CrossRefGoogle Scholar
Gordon, Sandy, and Huber, Greg. 2007. “The Effect of Electoral Competitiveness on Incumbent Behavior.” Quarterly Journal of Political Science 2 (2): 107–38.CrossRefGoogle Scholar
Grofman, Bernard, Griffin, Robert, and Berry, Gregory. 1995. “House Members Who Become Senators: Learning from a ‘Natural Experiment’ in Representation.” Legislative Studies Quarterly 20 (4): 513–29.CrossRefGoogle Scholar
Hahn, Jinyong, Todd, Petra, and van der Klaauw, Wilbert. 2001. “Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design.” Econometrica 69: 201–09.CrossRefGoogle Scholar
Heckman, James J., Ichimura, Hidehiko, Smith, Jeffrey, and Todd, Petra. 1998. “Characterizing Selection Bias Using Experimental Data.” Econometrica 66 (5): 1017–98.CrossRefGoogle Scholar
Heckman, James J., Ichimura, Hidehiko, and Todd, Petra. 1998. “Matching as an Econometric Evaluation Estimator.” Review of Economic Studies 65 (2): 261–94.CrossRefGoogle Scholar
Holland, Paul W. 1986. “Statistics and Causal Inference.” Journal of the American Statistical Association 81 (396): 945–60.CrossRefGoogle Scholar
Kaushik, Susheela. 1992. “Women and Political Participation.” In Women in Politics: Forms and Processes, ed. Sankaran, Kamala. New Delhi: Friedrich Ebert Stiftung.Google Scholar
Kishwar, Madhu. 1996. “Women and Politics: Beyond Quotas.” Economic and Political Weekly 31 (43): 2867–74.Google Scholar
Krasno, Jonathan S., and Green, Donald P.. 2008. “Do Televised Presidential Ads Increase Voter Turnout? Evidence from a Natural Experiment.” Journal of Politics 70 (1): 245–61.CrossRefGoogle Scholar
Lassen, David D. 2005. “The Effect of Information on Voter Turnout: Evidence from a Natural Experiment.” American Journal of Political Science 49 (1): 103–18.CrossRefGoogle Scholar
Lee, David S. 2008. “Randomized Experiments from Non-random Selection in U.S. House Elections.” Journal of Econometrics 142 (2): 675–97.CrossRefGoogle Scholar
Neyman, Jerzy. [1923] 1990. “On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.” Statistical Science 5 (4): 465–72. Trans. Dorota M. Dabrowska and Terence P. Speed.Google Scholar
Posner, Daniel N. 2004. “The Political Salience of Cultural Difference: Why Chewas and Tumbukas Are Allies in Zambia and Adversaries in Malawi.” American Political Science Review 98 (4): 529–45.CrossRefGoogle Scholar
Rosenbaum, Paul R. 2004. “Design Sensitivity in Observational Studies.” Biometrika 91 (1): 153–64.CrossRefGoogle Scholar
Rosenbaum, Paul R. 2010. Design of Observational Studies. New York: Springer-Verlag.CrossRefGoogle ScholarPubMed
Rubin, Donald B. 1974. “Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies.” Journal of Educational Psychology 66 (6): 688–701.CrossRefGoogle Scholar
Sekhon, Jasjeet S. 2008. “The Neyman-Rubin Model of Causal Inference and Estimation via Matching Methods.” In The Oxford Handbook of Political Methodology, eds. Box-Steffensmeier, Janet M., Brady, Henry E., and Collier, David. New York: Oxford University Press, 271–99.Google Scholar
Sekhon, Jasjeet S. 2010. “Opiates for the Matches: Matching Methods for Causal Inference.” Annual Review of Political Science 12: 487–508.CrossRefGoogle Scholar
Sekhon, Jasjeet S. 2011. “Matching: Multivariate and Propensity Score Matching with Automated Balance Search.” Journal of Statistical Software 42 (7): 1–52. http://sekhon.berkeley.edu/matching/ (accessed November 16, 2011).CrossRefGoogle Scholar
Sekhon, Jasjeet S., and Grieve, Richard. N.d. “A Non-parametric Matching Method for Bias Adjustment with Applications to Economic Evaluations.” Health Economics. Forthcoming.Google Scholar
Singh, Gopal Simrita, Lele, Medha Kotwal, Sathe, Nirmala, Sonalkar, Wandana, and Maydeo, Anjali. 1992. “Participation of Women in Electoral Politics in Maharashtra.” In Women in Politics: Forms and Processes, ed. Sankaran, Kamala. New Delhi: Friedrich Ebert Stiftung, 63–108.Google Scholar
van der Brug, Wouter. 2001. “Perceptions, Opinions and Party Preferences in the Face of a Real World Event: Chernobyl as a Natural Experiment in Political Psychology.” Journal of Theoretical Politics 13 (1): 53–80.CrossRefGoogle Scholar
Whitford, Andrew B. 2002. “Decentralization and Political Control of the Bureaucracy.” Journal of Theoretical Politics 14 (2): 167–93.CrossRefGoogle Scholar
Supplementary material: PDF

Sekhon supplementary material

Appendix.pdf

Download Sekhon supplementary material(PDF)
PDF 228.1 KB