Hostname: page-component-8448b6f56d-m8qmq Total loading time: 0 Render date: 2024-04-25T01:19:07.623Z Has data issue: false hasContentIssue false
  • English
  • Français

Modeling Guessing Components in the Measurement of Political Knowledge

Published online by Cambridge University Press:  27 September 2017

Tsung-han Tsai Open the ORCID record for Tsung-han Tsai [Opens in a new window]  and
Chang-chih Lin
Tsung-han Tsai*
Affiliation:
Assistant Professor, Department of Political Science, National Chengchi University, No. 64, Sec. 2, ZhiNan Rd., Wenshan District, Taipei City 11605, Taiwan, ROC. Email: thtsai@nccu.edu.tw
Chang-chih Lin
Affiliation:
Adjunct Assistant Professor, Department of Political Science, National Chengchi University, No. 64, Sec. 2, ZhiNan Rd., Wenshan District, Taipei City 11605, Taiwan, ROC. Email: lincc@nccu.edu.tw
*
*Email: thtsai@nccu.edu.tw
Get access Rights & Permissions [Opens in a new window]

Abstract

Due to the crucial role of political knowledge in democratic participation, the measurement of political knowledge has been a major concern in the discipline of political science. Common formats used for political knowledge questions include multiple-choice items and open-ended identification questions. The conventional wisdom holds that multiple-choice items induce guessing behavior, which leads to underestimated item-difficulty parameters and biased estimates of political knowledge. This article examines guessing behavior in multiple-choice items and argues that a successful guess requires certain levels of knowledge conditional on the difficulties of items. To deal with this issue, we propose a Bayesian IRT guessing model that accommodates the guessing components of item responses. The proposed model is applied to analyzing survey data in Taiwan, and the results show that the proposed model appropriately describes the guessing components based on respondents’ levels of political knowledge and item characteristics. That is, in general, partially informed respondents are more likely to have a successful guess because well-informed respondents do not need to guess and barely informed ones are highly seducible by the attractive distractors. We also examine the gender gap in political knowledge and find that, even when the guessing effect is accounted for, men are more knowledgeable than women about political affairs, which is consistent with the literature.

Type
Articles
Information
Political Analysis , Volume 25 , Issue 4 , October 2017 , pp. 483 - 504
Copyright
Copyright © The Author(s) 2017. Published by Cambridge University Press on behalf of the Society for Political Methodology. 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Authors’ note: The authors would like to thank Chia-yi Lee, Kevin A. Clarke, T. Y. Wang, Simon Jackman, Michelle Torres, Chi Huang, Dean Lacy, Brett Benson, Emerson Niou, Chung-li Wu, and Ching-ping Tang for helpful comments. Earlier versions of this manuscript were presented at the 2014 APSA Annual Meeting, 2015 Asian Political Methodology Conference, and 2015 APSA Annual Meeting. The research is financed by the Ministry of Science and Technology, R.O.C. under grant MOST 104-2410-H-004-091-MY2. We are also grateful to the editor, Michael Alvarez, and two anonymous reviewers for helpful suggestions. The replication materials are available at doi:10.7910/DVN/80WUQB (Tsai and Lin 2017).

Contributing Editor: R. Michael Alvarez.

References

Albert, James H. 1992. Bayesian estimation of normal ogive item response curves using Gibbs sampling. Journal of Educational and Behavioral Statistics 17(3):251–269.Google Scholar
Albert, James H., and Chib, Siddhartha. 1993. Bayesian analysis of binary and polychotomous response data. Journal of the American Statistical Association 88(442):669–679.Google Scholar
Andrich, David, and Marais, Ida. 2014. Person proficiency estimates in the dichotomous Rasch model when random guessing is removed from difficulty estimates of multiple choice items. Applied Psychological Measurement 38(6):432–449.Google Scholar
Andrich, David, Marais, Ida, and Humphry, Stephen. 2012. Using a theorem by Andersen and the Dichotomous Rasch model to assess the presence of random guessing in multiple choice items. Journal of Educational and Behavioral Statistics 37(3):417–442.Google Scholar
Baker, Frank B., and Kim, Seock-Ho. 2004. Item response theory: Parameter estimation techniques . 2 ed. Boca Raton, FL: Chapman & Hall/CRC.Google Scholar
Birnbaum, Allan. 1968. Some latent trait models and their use in inferring an examinee’s ability. In Statistical theories of mental test scores , ed. Frederic, M. Lord and Novick, Melvin R.. Reading, MA: Addison-Wesley, pp. 397–479.Google Scholar
Campbell, Angus, Converse, Philip E., Miller, Warren E., and Stokes, Donald E. 1960. The American voter . New York: John Wiley.Google Scholar
Cao, Jing, and Stokes, S Lynne. 2008. Bayesian IRT guessing models for partial guessing behaviors. Psychometrika 73(2):209–230.Google Scholar
Clinton, Joshua D., Jackman, Simon, and Rivers, Douglas. 2004. The statistical analysis of roll call data. American Political Science Review 98(2):355–370.Google Scholar
Delli Carpini, Michael X., and Keeter, Scott. 1991. Stability and change in the US public’s knowledge of politics. Public Opinion Quarterly 55(4):583–612.Google Scholar
Delli Carpini, Michael X., and Keeter, Scott. 1993. Measuring political knowledge: Putting first things first. American Journal of Political Science 37(4):1179–1206.Google Scholar
Delli Carpini, Michael X., and Keeter, Scott. 1996. What Americans know about politics and why it matters . New Haven, CT: Yale University Press.Google Scholar
Embretson, Susan E., and Reise, Steven P.. 2000. Item response theory for psychologists . Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Fox, Jean-Paul. 2010. Bayesian item response modeling: Theory and applications . New York, NY: Springer.Google Scholar
Galston, William A. 2001. Political knowledge, political engagement, and civic education. Annual Review of Political Science 4(1):217–234.Google Scholar
Gibson, James L., and Caldeira, Gregory A.. 2009. Knowing the Supreme Court? A reconsideration of public ignorance of the High Court. The Journal of Politics 71(2):429–441.Google Scholar
Hambleton, Ronald K., and Cook, Linda L.. 1977. Latent trait models and their use in the analysis of education test data. Journal of Educational Measurement 14(2):75–96.Google Scholar
Jackman, Simon. 2000. Estimation and inference are missing data problems: Unifying social science statistics via Bayesian simulation. Political Analysis 8(4):307–332.Google Scholar
Jackman, Simon. 2001. Multidimensional analysis of roll call data via Bayesian simulation: Identification, estimation, inference, and model checking. Political Analysis 9(3):227.Google Scholar
Jackman, Simon. 2009. Bayesian analysis for the social sciences . Chichester, UK: Wiley.Google Scholar
Jessee, Stephen A.2015. “Don’t know” responses, personality, and the measurement of political knowledge. Political Science Research and Method, 1–21. doi:10.1017/psrm.2015.23.Google Scholar
Johnson, Valen E., and Albert, James H.. 1999. Ordinal data modeling . New York, NY: Springer.Google Scholar
Lassen, David Dreyer. 2005. The effect of information on voter turnout: Evidence from a natural experiment. American Journal of Political Science 49(1):103–118.Google Scholar
Lizotte, Mary-Kate, and Sidman, Andrew H.. 2009. Explaining the gender gap in political knowledge. Politics & Gender 5(2):127–151.Google Scholar
Lord, Frederic M. 1968. An analysis of the verbal scholastic aptitude test using Birnbaum’s three-parameter logistic model. Educational and Psychological Measurement 28(4):989–1020.Google Scholar
Lord, Frederic M. 1970. Item characteristic curves estimated without knowledge of their mathematical form–a confrontation of Birnbaum’s logistic model. Psychometrika 35(1):43–50.Google Scholar
Lord, Frederic M., and Novick, Melvin R.. 1968. Statistical theories of mental test scores . Reading, MA: Addison-Wesley.Google Scholar
Luskin, Robert C. 1987. Measuring political sophistication. American Journal of Political Science 31(4):856–899.Google Scholar
Luskin, Robert C. 1990. Explaining political sophistication. Political Behavior 12(4):331–361.Google Scholar
Luskin, Robert C., and Bullock, John G.. 2011. “Don’t know” means “don’t know”: DK responses and the public’s level of political knowledge. Journal of Politics 73(2):547–557.Google Scholar
Martin, Andrew D., and Quinn, Kevin M.. 2002. Dynamic ideal point estimation via Markov chain Monte Carlo for the US Supreme Court, 1953–1999. Political Analysis 10(2):134–152.Google Scholar
Miller, Melissa K., and Orr, Shannon K.. 2008. Experimenting with a “third way” in political knowledge estimation. Public Opinion Quarterly 72(4):768–780.Google Scholar
Mondak, Jeffery J. 1999. Reconsidering the measurement of political knowledge. Political Analysis 8(1):57–82.Google Scholar
Mondak, Jeffery J. 2001. Developing valid knowledge scales. American Journal of Political Science 45(1):224–238.Google Scholar
Mondak, Jeffery J. 2010. Personality and the foundations of political behavior . New York, NY: Cambridge University Press.Google Scholar
Mondak, Jeffery J., and Davis, Belinda Creel. 2001. Asked and answered: Knowledge levels when we will not take “don’t know” for an answer. Political Behavior 23(3):199–224.Google Scholar
Mondak, Jeffery J., and Halperin, Karen D.. 2008. A framework for the study of personality and political behaviour. British Journal of Political Science 38(2):335–362.Google Scholar
Mondak, Jeffery J., and Anderson, Mary R.. 2004. The knowledge gap: A reexamination of gender-based differences in political knowledge. Journal of Politics 66(2):492–512.Google Scholar
Pietryka, Matthew T., and MacIntosh, Randall C.. 2013. An analysis of ANES items and their use in the construction of political knowledge scales. Political Analysis 21(4):407–429.Google Scholar
Plummer, Martyn. 2003. JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In Proceedings of the 3rd International Workshop on Distributed Statistical Computing , ed. Hornik, Kurt, Leisch, Friedrich, and Zeileis, Achim. March 20, Vienna, Austria. https://www.r-project.org/conferences/DSC-2003/Proceedings/Plummer.pdf.Google Scholar
Prior, Markus, and Lupia, Arthur. 2008. Money, time, and political knowledge: Distinguishing quick recall and political learning skills. American Journal of Political Science 52(1):169–183.Google Scholar
Rasch, Georg. 1960. Probabilistic models for some intelligence and attainment tests . Copenhagen: The Danish Institute for Educational Research.Google Scholar
San Martín, Ernesto, Guido, Del Pino, and De Boeck, Paul. 2006. IRT models for ability-based guessing. Applied Psychological Measurement 30(3):183–203.Google Scholar
Skrondal, Anders, and Rabe-Hesketh, Sophia. 2004. Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models . Boca Raton, FL: Chapman & Hall/CRC.Google Scholar
Sturgis, Patrick, Allum, Nick, and Smith, Patten. 2008. An experiment on the measurement of political knowledge in surveys. Public Opinion Quarterly 72(1):90–102.Google Scholar
Su, Yu-Sung, and Yajima, Masanao. 2012. R2jags: A package for running JAGS from R. R Package ver. 0.03-08. http://cran.r-project.org/web/packages/R2jags/.Google Scholar
Tedin, Kent L., and Murray, Richard W.. 1979. Public awareness of congressional representatives: Recall versus recognition. American Politics Research 7(4):509–517.Google Scholar
Treier, Shawn, and Jackman, Simon. 2008. Democracy as a latent variable. American Journal of Political Science 52(1):201–217.Google Scholar
Tsai, Tsung-han, and Lin, Chang-chih. 2017. Replication data for: Modeling guessing components in the measurement of political knowledge. doi:10.7910/DVN/80WUQB, Harvard Database, V1, UNF:6:OLHx9ShFd1/kZz7kxPYXZ==.Google Scholar
Tsai, Tsung-han, and Gill, Jeff. 2012. Superdiag: A comprehensive test suite for markov chain non-convergence. The Political Methodologist 19(2):12–18.Google Scholar
Waller, Michael I. 1976. Estimating parameters in the Rasch model: Removing the effects of random guessing . Princeton, NJ: Educational Testing Service.Google Scholar
Waller, Michael I. 1989. Modeling guessing behavior: A comparison of two IRT models. Applied Psychological Measurement 13(3):233–243.Google Scholar
Supplementary material: File

Tsai and Lin supplementary material

Tsai and Lin supplementary material 1

Download Tsai and Lin supplementary material(File)
File 4.4 MB