Hostname: page-component-8448b6f56d-m8qmq Total loading time: 0 Render date: 2024-04-24T03:14:08.225Z Has data issue: false hasContentIssue false

A Quantitative Method for Substantive Robustness Assessment

Published online by Cambridge University Press:  23 September 2014

Abstract

Empirical political science is not simply about reporting evidence; it is also about coming to conclusions on the basis of that evidence and acting on those conclusions. But whether a result is substantively significant––strong and certain enough to justify acting upon the belief that the null hypothesis is false––is difficult to objectively pin down, in part because different researchers have different standards for interpreting evidence. Instead, this article advocates judging results according to their “substantive robustness,” the degree to which a community with heterogeneous standards for interpreting evidence would agree that the result is substantively significant. This study illustrates how this can be done using Bayesian statistical decision techniques. Judging results in this way yields a tangible benefit: false positives are reduced without decreasing the power of the test, thus decreasing the error rate in published results.

Type
Original Articles
Copyright
Copyright © The European Political Science Association 2014 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Justin Esarey is Assistant Professor of Political Science, Rice University, PO Box 1892, Houston, TX 77251-1892, USA (justin@justinesarey.com). Nathan Danneman is Data Scientist, L-3 Data Tactics, 7901 Jones Branch Dr. No. 700, McLean, VA, USA (ndanneman@gmail.com). This research was supported in part by a University Research Committee grant from Emory University. We thank Bill Berry, Jeff Staton, Tom Clark, Drew Linzer, Emily Ritter, John Gasper, Stu Jordan, Tim Salmon, David Siegel, Jackie DeMeritt, Deirdre McCloskey, Andrew Martin and respondents at presentations of this research at Emory University, Florida State University, Georgia State University, the University of Rochester, the 2010 Annual Meeting of the Society for Political Methodology, the 2010 Annual Meeting of the Midwest Political Science Association, and the 2009 Annual Meeting of the American Political Science Association for helpful comments and criticism. To view supplementary material for this article, please visit http://dx.doi.org/10.1017/psrm.2014.14.

References

Achen, Chris. 1982. Interpreting and Using Regression. Quantitative Applications in the Social Sciences. Sage University Press.Google Scholar
Brambor, Thomas, Clark, William, and Golder, Matt. 2006. ‘Understanding Interaction Models: Improving Empirical Analyses’. Political Analysis 14:6382.CrossRefGoogle Scholar
Clark, W.R., and Golder, Matthew. 2006. ‘Rehabilitating Duverger’s Theory’. Comparative Political Studies 39(6):679708.CrossRefGoogle Scholar
Clinton, Joshua. 2006. ‘Representation in Congress: Constituents and Roll Calls in the 106th House’. Journal of Politics 68(2):397409.Google Scholar
Clinton, Joshua, Jackman, Simon, and Rivers, Douglas. 2004. ‘The Statistical Analysis of Roll Call Data’. American Political Science Review 98:355370.CrossRefGoogle Scholar
DeGroot, Morris. 2004 {1970}. Optimal Statistical Decisions. Hoboken, NJ: Wiley Interscience.CrossRefGoogle Scholar
French, Simon, and Insua, David Rios. 2000. Statistical Decision Theory. London and New York: Oxford University Press.Google Scholar
Gelfand, Alan E., and Ghosh, Malay. 2000. ‘Generalized Linear Models: A Bayesian View’. In Generalized Linear Models: A Bayesian Perspective, edited by Sujit K. Ghosh and Bani K. Mallick, 322. New York: Marcel Dekker.Google Scholar
Gelman, Andrew, Carlin, John B., Stern, Hal S., and Rubin, Donald B.. 2003. Bayesian Data Analysis, Second Edition. Boca Raton, FL: Chapman & Hall/CRC.Google Scholar
Gill, Jeff. 1999. ‘The Insignificance of Null Hypothesis Significance Testing’. Political Research Quarterly 52(3):647674.CrossRefGoogle Scholar
Hunter, John E. 1997. ‘Needed: A Ban on the Significance Test’. Psychological Science 8:37.CrossRefGoogle Scholar
Lunt, Peter. 2004. ‘The Significance of the Significance Test Controversy: Comments on ‘Size Matters’’. The Journal of Socio-Economics 33:559564.Google Scholar
Manski, Charles F. 2007. Identification for Prediction and Decision. Cambridge, MA: Harvard University Press.Google Scholar
McCloskey, Deirdre. 1998. The Rhetoric of Economics. Ann Arbor: University of Wisconsin Press.Google Scholar
Miller, Jane E. 2008. ‘Interpreting the Substantive Significance of Multivariable Regression Coefficients’. In 2008 Proceedings of the American Statistical Association, Statistical Education Section. Available at http://policy.rutgers.edu/faculty/miller/2008regression_ coefficients.pdfGoogle Scholar
Pratt, John W., Raiffa, Howard, and Schlaifer, Robert. 1996. Introduction to Statistical Decision Theory. Cambridge, MA: MIT Press.Google Scholar
Siegfried, Tom. 2010. ‘Odds Are, It’s Wrong: Science Fails to Face the Shortcomings of Statistics’. Science News 177:2629.Google Scholar
Wald, Abraham. 1950. Statistical Decision Functions. Oxford: Wiley.Google Scholar
Ziliak, Steven, and McCloskey, Deirdre. 2004. ‘Size Matters: The Standard Error of Regressions in the American Economic Review’. The Journal of Socio-Economics 33:527546.Google Scholar
Ziliak, Steven, and McCloskey, Deirdre. 2008. The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. Ann Arbor: University of Michigan Press.Google Scholar
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 613 Bytes
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8.1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 4.5 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 6.4 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 11.3 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8.1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8.1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 101 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8.1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8.1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 690 Bytes
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 11.4 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 7.8 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 7.6 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 9.9 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 25.5 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 102.3 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 1.1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 9.5 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 10.7 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 2.5 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 9.5 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8.3 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 8.5 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 6 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 5.4 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 1.2 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 1.2 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 2.6 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 6.2 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 1 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 5.6 KB
Supplementary material: File

Esarey and Danneman Supplementary Material

Supplementary Material

Download Esarey and Danneman Supplementary Material(File)
File 1.3 KB