Skip to main content Accessibility help
×
×
Home

Statistical Analysis of List Experiments

  • Graeme Blair (a1) and Kosuke Imai (a1)

Abstract

The validity of empirical research often relies upon the accuracy of self-reported behavior and beliefs. Yet eliciting truthful answers in surveys is challenging, especially when studying sensitive issues such as racial prejudice, corruption, and support for militant groups. List experiments have attracted much attention recently as a potential solution to this measurement problem. Many researchers, however, have used a simple difference-in-means estimator, which prevents the efficient examination of multivariate relationships between respondents' characteristics and their responses to sensitive items. Moreover, no systematic means exists to investigate the role of underlying assumptions. We fill these gaps by developing a set of new statistical methods for list experiments. We identify the commonly invoked assumptions, propose new multivariate regression estimators, and develop methods to detect and adjust for potential violations of key assumptions. For empirical illustration, we analyze list experiments concerning racial prejudice. Open-source software is made available to implement the proposed methodology.

    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Statistical Analysis of List Experiments
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Statistical Analysis of List Experiments
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Statistical Analysis of List Experiments
      Available formats
      ×

Copyright

Corresponding author

e-mail: gblair@princeton.edu (corresponding author)
e-mail: kimai@princeton.edu (corresponding author)

Footnotes

Hide All

Edited by R. Michael Alvarez

Authors' note: Financial support from the National Science Foundation (SES-0849715) is acknowledged. All the proposed methods presented in this paper are implemented as part of the R package, “list: Statistical Methods for the Item Count Technique and List Experiment,” which is freely available for download at http://cran.r-project.org/package=list (Blair and Imai 2011a). The replication archive is available as Blair and Imai (2011b), and the Supplementary Materials are posted on the Political Analysis Web site. We thank Dan Corstange for providing his computer code, which we use in our simulation study, as well as for useful comments. Detailed comments from the editor and two anonymous reviewers significantly improved the presentation of this paper. Thanks also to Kate Baldwin, Neal Beck, Will Bullock, Stephen Chaudoin, Matthew Creighton, Michael Donnelly, Adam Glynn, Wenge Guo, John Londregan, Aila Matanock, Dustin Tingley, Teppei Yamamoto, and seminar participants at New York University, the New Jersey Institute of Technology, and Princeton University for helpful discussions.

Footnotes

References

Hide All
Andrews, D. W. K., and Soares, G. 2010. Inference for parameters defined by moment inequalities using generalized moment selection. Econometrica 78: 119–57.
Berinsky, A. J. 2004. Silent voices: Public opinion and political participation in America. Princeton, NJ: Princeton University Press.
Biemer, P., and Brown, G. 2005. Model-based estimation of drug use prevalence using item count data. Journal of Official Statistics 21: 287308.
Blair, G., and Imai, K. 2011a. list: Statistical Methods for the Item Count Technique and List Experiment. Comprehensive R Archive Network (CRAN). http://CRAN.R-project.org/package=list.
Blair, G., and Imai, K. 2011b. Replication data for: Statistical analysis of list experiments. hdl:1902.1/17040. The Dataverse Network.
Bullock, W., Imai, K., and Shapiro, J. N. 2011. Statistical analysis of endorsement experiments: Measuring support for militant groups in Pakistan. Political Analysis 19: 363–84.
Burden, B. C. 2000. Voter turnout and the national election studies. Political Analysis 8: 389–98.
Chaudhuri, A., and Christofides, T. C. 2007. Item count technique in estimating the proportion of people with a sensitive feature. Journal of Statistical Planning and Inference 137: 589–93.
Chen, X., Dempster, A. P., and Liu, J. S. 1994. Weighted finite population sampling to maximize entropy. Biometrika 81: 457–69.
Corstange, D. 2009. Sensitive questions, truthful answers? Modeling the list experiment with LISTIT. Political Analysis 17(1): 4563.
Coutts, E., and Jann, B. 2011. Sensitive questions in online surveys: Experimental results for the randomized response technique (RRT) and the unmatched count technique (UCT). Sociological Methods & Research 40: 169–93.
Dalton, D. R., Wimbush, J. C., and Daily, C. M. 1994. Using the unmatched count technique (UCT) to estimate base rates for sensitive behavior. Personnel Psychology 47: 817–29.
Dempster, A. P., Laird, N. M., and Rubin, D. B. 1977. Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society, Series B, Methodological 39(1): 138.
Droitcour, J., Caspar, R. A., Hubbard, M. L., Parsley, T. L., Visscher, W., and Ezzati, T. M. 1991. Measurement errors in surveys. In The Item Count Technique as a method of indirect questioning: A review of its development and a case study application, eds. Biemer, P. P., Groves, R. M., Lyberg, L. E., Mathiowetz, N. A., and Sudman, S., 185210. New York: John Wiley & Sons.
Ehm, W. 1991. Binomial approximation to the Poisson binomial distribution. Statistics & Probability Letters 11: 716.
Flavin, P., and Keane, M. 2009. How angry am I? Let me count the ways: Question format bias in list experiments. Technical Report, Department of Political Science, University of Notre Dame.
Gelman, A., Jakulin, A., Pittau, M. G., and Su, Y. 2008. A weakly informative default prior distribution for logistic and other regression models. Annals of Applied Statistics 2: 1360–83.
Gilens, M., Sniderman, P. M., and Kuklinski, J. H. 1998. Affirmative action and the politics of realignment. British Journal of Political Science 28: 159–83.
Gingerich, D. W. 2010. Understanding off-the-books politics: Conducting inference on the determinants of sensitive behavior with randomized response surveys. Political Analysis 18: 349–80.
Glynn, A. N. 2010. What can we learn with statistical truth serum? Design and analysis of the list experiment. Technical Report, Department of Government, Harvard University.
Gonzalez-Ocantos, E., Kiewet de Jonge, C., Melendez, C., Osorio, J., and Nickerson, D. W. 2010. Vote buying and social desirability bias: Experimental evidence from Nicaragua. Technical Report, Department of Political Science, University of Notre Dame.
Holbrook, A. L., and Krosnick, J. A. 2010. Social desirability bias in voter turnout reports: Tests using the item count technique. Public Opinion Quarterly 74 (1): 3767.
Holland, P. W. 1986. Statistics and causal inference (with discussion). Journal of the American Statistical Association 81: 945–60.
Holland, B. S., and Copenhaver, M. D. 1987. An improved sequentially rejective Bonferroni test procedure. Biometrics 43: 417–23.
Imai, K. 2011. Multivariate regression analysis for the item count technique. Journal of the American Statistical Association 106: 407–16.
Imai, K., King, G., and Stuart, E. A. 2008. Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society, Series A (Statistics in Society) 171: 481502.
Janus, A. L. 2010. The influence of social desirability pressures on expressed immigration attitudes. Social Science Quarterly 91: 928–46.
Kane, J. G., Craig, S. C., and Wald, K. D. 2004. Religion and presidential politics in Florida: A list experiment. Social Science Quarterly 85: 281–93.
Kudô, A. 1963. A multivariate analogue of the one-sided test. Biometrika 50: 403–18.
Kuklinski, J. H., Cobb, M. D., and Gilens, M. 1997a. Racial attitudes and the “New South.” Journal of Politics 59: 323–49.
Kuklinski, J. H., Sniderman, P. M., Knight, K., Piazza, T., Tetlock, P. E., Lawrence, G. R., and Mellers, B. 1997b. Racial prejudice and attitudes toward affirmative action. American Journal of Political Science 41: 402–19.
LaBrie, J. W., and Earleywine, M. 2000. Sexual risk behaviors and alcohol: Higher base rates revealed using the unmatched-count technique. Journal of Sex Research 37: 321–26.
Manski, C. F. 2007. Identification for Prediction and Decision. Cambridge, MA: Harvard University Press.
Miller, J. D. 1984. A new survey technique for studying deviant behavior. PhD diss, The George Washington University.
Newey, W. K., and McFadden, D. 1994. Large sample estimation and hypothesis testing. In Handbook of Econometrics, ed. Engle, R. F. and McFadden, D. L., volume IV, 2111–245. Amsterdam, The Netherlands: North Holland.
Perlman, M. D. 1969. One-sided testing problems in multivariate analysis. The Annals of Mathematical Statistics 40: 549–67.
Presser, S., and Stinson, L. 1998. Data collection mode and social desirability bias in self-reported religious attendance. American Sociological Review 63: 137–45.
Raghavarao, D., and Federer, W. T. 1979. Block total response as an alternative to the randomized response method in surveys. Journal of the Royal Statistical Society, Series B, Methodological 41 (1): 40–5.
Rayburn, N. R., Earleywine, M., and Davison, G. C. 2003. An investigation of base rates of anti-gay hate crimes using the unmatched-count technique. Journal of Aggression, Maltreatment & Trauma 6: 137–52.
Redlawsk, D. P., Tolbert, C. J., and Franko, W. 2010. Voters, emotions, and race in 2008: Obama as the first black president. Political Research Quarterly 63: 875–89.
Shapiro, A. 1985. Asymptotic distribution of test statistics in the analysis of moment structures under inequality constraints. Biometrika 72: 133–44.
Sniderman, P. M., and Carmines, E. G. 1997. Reaching Beyond Race. Cambridge, MA: Harvard University Press.
Sniderman, P. M., Tetlock, P. E., and Piazza, T. 1992. Codebook for the 1991 National Race and Politics Survey. Survey Research Center, Berkeley, California. http://sda.berkeley.edu/D3/Natlrace/Doc/nrac.htm.
Streb, M. J., Burrell, B., Frederick, B., and Genovese, M. A. 2008. Social desirability effects and support for a female american president. Public Opinion Quarterly 72(1): 7689.
Tourangeau, R., and Yan, T. 2007. Sensitive questions in surveys. Psychological Bulletin 133: 859–83.
Tsuchiya, T. 2005. Domain estimators for the item count technique. Survey Methodology 31(1): 4151.
Tsuchiya, T., Hirai, Y., and Ono, S. 2007. A study of the properties of the item count technique. Public Opinon Quarterly 71: 253–72.
Warner, S. L. 1965. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association 60(309): 63–9.
Wimbush, J. C., and Dalton, D. R. 1997. Base rate for employee theft: Convergence of multiple methods. Journal of Applied Psychology 82: 756–63.
Wolak, F. A. 1991. The local nature of hypothesis tests involving inequality constraints in nonlinear models. Econometrica 59: 981–95.
Zaller, J. 2002. The statistical power of election studies to detect media exposure effects in political campaigns. Election Studies 21: 297329.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax
Type Description Title
PDF
Supplementary materials

Blair and Imai supplementary material
List Experiments

 PDF (198 KB)
198 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed