Skip to main content Accessibility help
×
×
Home

List Experiments with Measurement Error

  • Graeme Blair (a1), Winston Chou (a2) and Kosuke Imai (a3)

Abstract

Measurement error threatens the validity of survey research, especially when studying sensitive questions. Although list experiments can help discourage deliberate misreporting, they may also suffer from nonstrategic measurement error due to flawed implementation and respondents’ inattention. Such error runs against the assumptions of the standard maximum likelihood regression (MLreg) estimator for list experiments and can result in misleading inferences, especially when the underlying sensitive trait is rare. We address this problem by providing new tools for diagnosing and mitigating measurement error in list experiments. First, we demonstrate that the nonlinear least squares regression (NLSreg) estimator proposed in Imai (2011) is robust to nonstrategic measurement error. Second, we offer a general model misspecification test to gauge the divergence of the MLreg and NLSreg estimates. Third, we show how to model measurement error directly, proposing new estimators that preserve the statistical efficiency of MLreg while improving robustness. Last, we revisit empirical studies shown to exhibit nonstrategic measurement error, and demonstrate that our tools readily diagnose and mitigate the bias. We conclude this article with a number of practical recommendations for applied researchers. The proposed methods are implemented through an open-source software package.

Copyright

Corresponding author

Footnotes

Hide All

Contributing Editor: Jeff Gill

Authors’ note: All the proposed methods presented in this paper are implemented as part of the R package, list: Statistical Methods for the Item Count Technique and List Experiment, which is freely available for download at http://cran.r-project.org/package=list (Blair, Chou, and Imai 2017). The replication materials are available as Blair, Chou, and Imai (2019).

Footnotes

References

Hide All
Ahlquist, John S. 2018. “List experiment design, non-strategic respondent error, and item count technique estimators.” Political Analysis 26:3453.
Ahlquist, John S., Mayer, Kenneth R., and Jackman, Simon. 2014. “Alien abduction and voter impersonation in the 2012 U.S. General Election: Evidence from a survey list experiment.” Election Law Journal 13:460475.
Aronow, Peter M., Coppock, Alexander, Crawford, Forrest W., and Green, Donald P.. 2015. “Combining list experiment and direct question estimates of sensitive behavior prevalence.” Journal of Survey Statistics and Methodology 3:4366.
Blair, Graeme, and Imai, Kosuke. 2012. “Statistical analysis of list experiments.” Political Analysis 20:4777.
Blair, Graeme, Imai, Kosuke, and Lyall, Jason. 2014. “Comparing and combining list and endorsement experiments: Evidence from Afghanistan.” American Journal of Political Science 58:10431063.
Blair, Graeme, Imai, Kosuke, and Zhou, Yang-Yang. 2015. “Design and analysis of randomized response technique.” Journal of the American Statistical Association 110:13041319.
Blair, Graeme, Chou, Winston, and Imai, Kosuke. 2017 list: Statistical methods for the item count technique and list experiment. Available at the Comprehensive R Archive Network (CRAN). https://CRAN.R-project.org/package=list.
Blair, Graeme, Chou, Winston, and Imai, Kosuke. 2019 “Replication data for: List experiments with measurement error.” https://doi.org/10.7910/DVN/L3GWNP, Harvard Dataverse.
Bullock, Will, Imai, Kosuke, and Shapiro, Jacob N.. 2011. “Statistical analysis of endorsement experiments: Measuring support for militant groups in Pakistan.” Political Analysis 19:363384.
Carroll, Raymond J., Ruppert, David, Stefanski, Leonard A., and Crainiceanu, Ciprian M.. 2006. Measurement error in nonlinear models: A modern perspective . 2nd ed. London: Chapman & Hall.
Chou, Winston. 2018. Lying on surveys: Methods for list experiments with direct questioning. Technical report, Princeton University.
Chou, Winston, Imai, Kosuke, and Rosenfeld, Bryn. 2017. “Sensitive survey questions with auxiliary information.” Sociological Methods & Research , doi:10/1177/0049124117729711.
Corstange, Daniel. 2009. “Sensitive questions, truthful answers?: Modeling the list experiment with LISTIT.” Political Analysis 17:4563.
Delgado, M. Kit, Wanner, Kathryn J., and McDonald, Catherine. 2016. “Adolescent cellphone use while driving: An overview of the literature and promising future directions for prevention.” Media and Communication 4:7989.
Dempster, Arthur P., Laird, Nan M., and Rubin, Donald B.. 1977. “Maximum likelihood from incomplete data via the EM algorithm (with discussion).” Journal of the Royal Statistical Society, Series B, Methodological 39:137.
Gelman, Andrew, Jakulin, Aleks, Pittau, Maria Grazia, and Su, Yu-Sung. 2008. “A weakly informative default prior distribution for logistic and other regression models.” Annals of Applied Statistics 2:13601383.
Gingerich, Daniel W. 2010. “Understanding off-the-books politics: Conducting inference on the determinants of sensitive behavior with randomized response surveys.” Political Analysis 18:349380.
Glynn, Adam N. 2013. “What can we learn with statistical truth serum?: Design and analysis of the list experiment.” Public Opinion Quarterly 77:159172.
Hausman, Jerry A. 1978. “Specification tests in econometrics.” Econometrica 46:12511271.
Imai, Kosuke. 2011. “Multivariate regression analysis for the item count technique.” Journal of the American Statistical Association 106:407416.
King, Gary, and Zeng, Langche. 2001. “Logistic regression in rare events data.” Political Analysis 9:137163.
Lyall, Jason, Blair, Graeme, and Imai, Kosuke. 2013. “Explaining support for combatants during wartime: A survey experiment in Afghanistan.” American Political Science Review 107:679705.
Miller, J. D.1984 The item-count/paired lists technique: An indirect method of surveying deviant behavior. PhD thesis, George Washington University.
Rosenfeld, Bryn, Imai, Kosuke, and Shapiro, Jacob. 2016. “An empirical validation study of popular survey methodologies for sensitive questions.” American Journal of Political Science 60:783802.
Schreiber, Sven. 2008. “The Hausman test statistic can be negative, even asymptotically.” Jahrbücher für Nationalökonomie und Statistik 228:394405.
Sobel, Richard. 2009. “Voter-ID Issues in Politics and Political Science: Editor’s Introduction.” PS: Political Science & Politics 42:8185.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax

Keywords

Type Description Title
PDF
Supplementary materials

Blair et al. supplementary material
Blair et al. supplementary material 1

 PDF (180 KB)
180 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed