Published online by Cambridge University Press: 14 November 2016
This article uses a replication experiment of ninety-four specifications from sixteen different studies to show the severity of the problem of selection on unobservables. Using a variety of approaches, it shows that membership in the General Agreement on Tariffs and Trade/World Trade Organization has a significant effect on a surprisingly high number of dependent variables (34 per cent) that have little or no theoretical relationship to the WTO. To make the exercise even more conservative, the study demonstrates that membership in a low-impact environmental treaty, the Convention on Trade in Endangered Species, yields similarly high false positive rates. The authors advocate theoretically informed sensitivity analysis, showing how prior theoretical knowledge conditions the crucial choice of covariates for sensitivity tests. While the current study focuses on international institutions, the arguments also apply to other subfields and applications.
Department of Political Science, University of Illinois at Urbana-Champaign (email: firstname.lastname@example.org); Department of Political Science, University of Pittsburgh (email: email@example.com); Department of Politics, Princeton University (email: firstname.lastname@example.org). We appreciate helpful advice from Marc Busch, Jake Bowers, William Clark, Mark Fredrickson, Kristian Skrede Gleditsch, Emilie Hafner-Burton, Sarah Hummel, In Song Kim, Moritz Marbach, Michael Miller, Dan Nielson, Dominik Schraff, Christopher Stanton and Dustin Tingley. We also appreciate comments from audiences at the International Political Economy Society, American Political Science Association, Political Economy of International Organizations and International Studies Association conferences, as well as the UCSD Workshop on International Law and Regulation and the Harvard Government Department 3005 Seminar participants. We appreciate the constructive comments from our editor at BJPS, Sona Golder, as well as from the anonymous reviewers. Data replication sets are available at http://dataverse.harvard.edu/dataverse/BJPolS, and online appendices are available at http://dx.doi.org/doi:10.1017/S000712341600034X.