Skip to main content Accessibility help

Guarding Against False Positives in Qualitative Comparative Analysis

  • Bear F. Braumoeller (a1)


The various methodological techniques that fall under the umbrella description of qualitative comparative analysis (QCA) are increasingly popular for modeling causal complexity and necessary or sufficient conditions in medium-N settings. Because QCA methods are not designed as statistical techniques, however, there is no way to assess the probability that the patterns they uncover are the result of chance. Moreover, the implications of the multiple hypothesis tests inherent in these techniques for the false positive rate of the results are not widely understood. This article fills both gaps by tailoring a simple permutation test to the needs of QCA users and adjusting the Type I error rate of the test to take into account the multiple hypothesis tests inherent in QCA. An empirical application–a reexamination of a study of protest-movement success in the Arab Spring–highlights the need for such a test by showing that even very strong QCA results may plausibly be the result of chance.


Corresponding author

e-mail: (corresponding author)


Hide All

Author's note: Thanks to Christopher Achen, David Collier, Jirka Lewandowski, the scholars who attended the 2014 summer seminars on Boolean logit at WZB Berlin, and those who attended my sessions at the 2014 IQMR Summer Institute in Syracuse, New York, for valuable feedback, and to Andrew Rosenberg and Austin Knuppe for invaluable research assistance. Replication Data are available on the Dataverse site for this article,



Hide All
Ansani, Andrea, and Daniele, Vittorio. 2012. About a revolution: The economic motivations of the Arab Spring. International Journal of Development and Conflict 02(03): 124.
Benjamini, Yoav, and Hochberg, Yosef. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B 57(1): 289300.
Braumoeller, Bear. 2015. Replication data for: Guarding against false positives in qualitative comparative analysis. Harvard Dataverse, V2 [Version], May 18, 2015.
Braumoeller, Bear, and Goertz, Gary. 2000. The methodology of necessary conditions. American Journal of Political Science 44(4): 844–58.
Cronqvist, Lasse, and Berg-Schlosser, Dirk. 2009. Multi-value QCA (mvQCA). In Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques, eds. Rihoux, Benoît and Ragin, Charles C., 6986. Thousand Oaks, CA: Sage Publications, Inc.
Dion, Douglas. 1998. Evidence and inference in the comparative case study. Comparative Politics 30(2): 127–45.
Doyle, Michael W. 1983a. Kant, liberal legacies, and foreign affairs. Philosophy and Public Affairs 12(3): 205–35.
Doyle, Michael W 1983b. Kant, liberal legacies, and foreign affairs, part 2. Philosophy and Public Affairs 12(4): 323–53.
Eliason, S. R., and Stryker, R. 2009. Goodness-of-fit tests and descriptive measures in fuzzy-set analysis. Sociological Methods & Research 38(1): 102–46.
Goertz, Gary, Hak, Tony, and Dul, Jan. 2012. Ceilings and floors: Where are there no observations? Sociological Methods & Research 42(1): 340.
Good, Phillip I. 2005. Permutation, parametric and bootstrap tests of hypotheses, Springer Series in Statistics. 3rd ed. New York: Springer.
Holm, Sture. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6(2): 6570.
Hug, S. 2013. Qualitative comparative analysis: How inductive use and measurement error lead to problematic inference. Political Analysis 21(2): 252–65.
Hussain, Muzammil M., and Howard, Philip N. 2013. What best explains successful protest cascades? ICTs and the fuzzy causes of the Arab Spring. International Studies Review 15(1): 4866.
Krogslund, C., Choi, D. D., and Poertner, M. 2015. Fuzzy sets on shaky ground: Parameter sensitivity and confirmation bias in fsQCA. Political Analysis 23(1): 2141.
LaGraffe, Dan. 2012. The youth bulge in Egypt: An intersection of demographics, security, and the Arab Spring. Journal of Strategic Security 5(2): 6580.
Ludbrook, John, and Dudley, Hugh. 1998. Why permutation tests are superior to t and F tests in biomedical research. American Statistician 52(2): 127–32.
Marx, Axel, and Dusa, Adrian. 2011. Crisp-set qualitative comparative analysis (csQCA): Contradictions and consistency benchmarks for model specification. Methodological Innovations Online 6(2): 103–48.
Monroe, Burt, and Gold, Suzanne. 2004. A close look at the qualitative comparative analysis (QCA) family of methodologies. Paper presented at the Annual Meeting of the Midwest Political Science Association, Palmer House Hilton, Chicago, IL.
Paul, Christopher, and Clarke, Colin P. 2014. A broad approach to countering the Islamic state. Washington Post.
Ragin, Charles C. 1987. The comparative method: Moving beyond qualitative and quantitative strategies. Berkeley: University of California Press.
Ragin, Charles C 2000. Fuzzy-set social science. Chicago: University of Chicago Press.
Ragin, Charles C 2004. Between complexity and parsimony: Limited diversity, counterfactual cases, and comparative analysis. University of California. (accessed on July 10, 2015).
Ragin, Charles C 2008. Redesigning social inquiry: Fuzzy sets and beyond. Chicago: University of Chicago Press.
Ragin, Charles C., Ilene Strand, Sarah, and Rubinson, Claude. 2008. User's guide to fuzzy-set/qualitative comparative analysis. Manuscript, University of Arizona.∼cragin/fsQCA/download/fsQCAManual.pdf (accessed March 10, 2015).
Rihoux, Benoît, Alamos-Concha, Damien Bol, Axel Marx, Priscilla, and Rezsöhazy, Ilona. 2013. From niche to mainstream method? A comprehensive mapping of QCA applications in journal articles from 1984 to 2011. Political Research Quarterly 66(1): 175–84.
Schneider, Carsten Q., and Wagemann, Claudius. 2012. Set-theoretic methods for the social sciences: A guide to qualitative comparative analysis. Cambridge, UK: Cambridge University Press.
Schneider, Carsten Q., and Wagemann, Claudius. 2013. Doing justice to logical remainders in QCA: Moving beyond the standard analysis. Political Research Quarterly 66(1): 211–20.
Thygeson, N. Marcus, Peikes, Deborah, and Zutshi, Aparajita. 2013. Fuzzy-set qualitative comparative analysis: A configurational comparative method to identify multiple pathways to improve patient-centered medical home models. Technical report AHRQ, Publication No. 13-0026-EF, Agency for Healthcare Research and Quality, Rockville, MD.
Welkowitz, Joan, Cohen, Barry H., and Brooke Lea, R. 2012. Introductory statistics for the behavioral sciences. 7th ed. Hoboken, NJ: Wiley.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed