Skip to main content Accessibility help
×
Home

Nonresponse and Mode Effects in Self- and Interviewer-Administered Surveys

  • Lonna Rae Atkeson (a1), Alex N. Adams (a2) and R. Michael Alvarez (a3)

Abstract

We examine the quality of two probability-based polls, one interviewer administered (telephone) and one self-administered (Internet and mail mixed mode survey). The polls use the same sampling frame (registered voters) and the same questions. First, we examine the representativeness of both surveys using information known about the population, and although we find important differences between the two in terms of sampling and nonresponse bias, we also find that both surveys represent the underlying population despite low response rates. We also test for mode effects between surveys due to social desirability and how it influences nondifferentiation or satisficing. Using a variety of methods (t-tests, multivariate regression, and genetic propensity matching), we find evidence that the presence of an interviewer alters response patterns on ego-driven questions. The implications of our work are important, due to the increasing popularity of mixed mode surveys. Researchers need to be methodologically sensitive to these differences when analyzing surveys that allow for different response modes.

Copyright

Corresponding author

e-mail: atkeson@unm.edu (corresponding author)

Footnotes

Hide All

Authors' note: Supplementary materials for this article are available on the Political Analysis Web site.

Footnotes

References

Hide All
Adams, Alex, Atkeson, Lonna, and Karp, Jeffrey. 2013. Measurement error in online survey panels: Panel conditioning and data quality. Presented at the American Panel Survey (TAPS) Workshop, Washington University, St. Louis, November 8–9.
American Association for Public Opinion Research. 2010. AAPOR report on online panels. http://www.aapor.org/AAPOR_Releases_Report_on_Online_Survey_Panels/2263.htm (accessed January 10, 2014).
The American Association for Public Opinion Research. 2011. Standard definitions: Final dispositions of case codes and outcome rates for surveys, 7th edition. AAPOR. http://aapor.org/Content/NavigationMenu/AboutAAPOR/StandardsampEthics/StandardDefinitions/StandardDefinitions2011.pdf (accessed January 10, 2014).
Alexander, C. H. Jr., and Wetrogan, S. 2000. Integrating the American Community Survey and the Intercensal Demographic Estimates Program. Proceedings of the American Statistical Association at www.amstat.org/sections/srms/proceedings/papers/2000_046.pdf (accessed January 10, 2014).
Aquilino, W. 1994. Interview mode effects in surveys of drug and alcohol use. Public Opinion Quarterly 58: 210-40.
Atkeson, Lonna Rae, Adams, Alex N., and Michael Alvarez, R. 2013. Replication data for: Nonresponse and mode effects in self- and interviewer-administered surveys. http://dx.doi.org/10.7910/DVN/23773 UNF:5:QH1dvPfRoJ+lbQkO32oeww== IQSS Dataverse Network [Distributor] V1 [Version].
Atkeson, Lonna Rae, and Adams, Alex N. 2010. Mixed mode (Internet and Mail) probability samples and survey representativeness: The case of New Mexico 2008. San Francisco, CA: Presented at the Western Political Science Association, April 1–4.
Atkeson, Lonna Rae, Adams, Alex N., Bryant, Lisa A., Zilberman, Luciana, and Saunders, Kyle L. 2011. Considering mixed mode surveys for questions in political behavior: Using the Internet and mail to get quality data at reasonable costs. Political Behavior 33 (1): 161-78.
Atkeson, Lonna Rae, and Tafoya, Lori. 2008. Surveying political activists: An examination of the effectiveness of a mixed- mode (Internet and mail) survey design. Journal of Elections Public Opinion and Parties 18 (4): 367-86.
Atkeson, Lonna Rae, Michael Alvarez, R., Adams, Alex N., and Bryant, Lisa. 2011. The 2010 New Mexico election administration report. Typescript, University of New Mexico. http://polisci.unm.edu/common/documents/c-sved/papers/nm-2010-general-election.pdf (accessed January 10, 2014).
Atkeson, Lonna Rae, Bryant, Lisa A., and Adams, Alex N. 2013. The 2012 Bernalillo County election administration report. Typescript, University of New Mexico. http://polisci.unm.edu/common/documents/c-sved/papers/BernCoFinalReport.pdf (accessed January 10, 2014).
Atkeson, Lonna Rae, Michael Alvarez, R., and Hall, Thad E. 2010. Assessing electoral performance in New Mexico using an eco-system approach: New Mexico 2008. Typescript, University of New Mexico. http://polisci.unm.edu/common/documents/c-sved/papers/nm08pew.pdf (accessed January 10, 2014).
Balden, W. 2004. Multi-mode data collection: Benefits and downsides. Paper presented at the 2004 Conference of the Great Lakes. Cancun, Mexico: Chapter of the Marketing Research Association.
Belli, Robert F., Traugott, Michael W., and Beckmann, Matthew N. 2001. What leads to voting overreports? Contrasts of overreporters and admitted nonvoters in the American National Election Studies. Journal of Official Statistics 17 (4): 479-98.
Blair, Graeme, and Imai, Kosuke. 2012. Statistical analysis of list experiments. Political Analysis 20: 4777.
Blumberg, Stephen J., and Luke, Julian V. 2009. Wireless substitution: Early release of estimates from the National Health Interview Survey. July-December 2008. http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless200905.htm.
Blumberg, Stephen J., and Luke, Julian V. 2013. Wireless substitution: Early release of estimates from the National Health Interview Survey. July-December 2012. http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless201306.pdf.
Burden, Barry C. 2000. Voter turnout and the National Election Studies. Political Analysis 8 (4): 389-98.
Burden, Barry C. 2003. Internal and external effects on the accuracy of NES turnout: Reply. Political Analysis 11 (2): 193-95.
Campbell, Donald T. 1950. The indirect assessment of social attitudes. Psychological Bulletin 47):1538.
Cannell, C. F., Miller, P. V., Oksenberg, and L. 1981. Research on interviewing techniques. In Sociological Methodology, ed. Leinhardt, S., 389437. San Francisco, CA: Jossey-Bass.
Carley-Baxter, Lisa R., Peytchev, Andy, and Black, Michele C. 2010. Comparison of cell phone and landline surveys: A design perspective. Field Methods 22 (1): 315.
Chang, LinChiat, and Krosnick, Jon. 2009. National surveys via RDD telephone interviewing versus the Internet: Comparing sample representativeness and response quality. Public Opinion Quarterly 73 (4): 641-78.
Chang, LinChiat, and Krosnick, Jon. 2010. Comparing oral interviewing with self-administered computerized questionnaires: An experiment. Public Opinion Quarterly 74 (1): 154-67.
Christian Leah, M., and Dillman, Don A. 2004. The influence of graphical and symbolic language manipulations on responses to self-administered questions. Public Opinion Quarterly 68 (1): 5780.
Curtin, Richard, Presser, Stanley, and Singer, Eleanor. 2000. The effects of response rate changes on the index of consumer sentiment. Public Opinion Quarterly 64: 413-28.
Davis, James Allan, Smith, Tom, and Marsden, Peter V. 2009. General social surveys, 1972–2008: Cumulative codebook. Chicago: National Opinion Research Center.
Day, N. A., Dunt, D. R., Day, and S. 1995. Maximizing response to surveys in health program evaluation at minimum cost using multiple methods. Evaluation Review 19 (4): 436-50.
de Leeuw, Edith. 2005. To mix or not to mix: Data collection modes in surveys. Journal of Official Statistics 21: 233-55.
de Leeuw, E., Hox, J., Korendijk, E., Lensvelt-Mulders, G., Callegaro, and M. 2004. The influence of advance letters on response in telephone surveys: A meta-analysis. Maastricht: Paper presented at the 15 International Workshop on Household Survey Nonresponse.
de Leeuw, Edith D., and de Heer, W. 2002. Trends in household survey nonresponse: A longitudinal and international comparison. In Survey nonresponse, eds. Groves, Robert M., Dillman, Don A., Eltinge, John L., and Little, Roderick J. A. New York: Wiley.
de Leeuw, Edith, and Van der Zouwen, J. 1988. Data quality in telephone and face-to-face surveys: A comparative meta- analysis. In Telephone survey methodology, eds. Groves, R., Bimer, P. P., Lyberg, L., Massey, I. T., Nicholls, W. L., Waksberg, and J. New York: John Wiley & Sons.
DeBell, Mathew, Wilson, Catherine, Segura, Gary, Jackman, Simon, and Hutchings, Vincent. 2011. Methodology report and user's guide for the ANES 2010–2012 Evaluations of Government and Society Study. Palo Alto, CA, and Ann Arbor: Stanford University and University of Michigan.
Diamond, Alexis, and Sekhon, Jasjeet S. 2012. Genetic matching for estimating causal effects: A general multivariate matching method for achieving balance in observational studies. Review of Economics and Statistics. http://sekhon.berkeley.edu/papers/GenMatch.pdf.
Dillman, Don A. 2000. Mail and Internet surveys: The tailored design method. 2nd ed. New York: Wiley.
Dillman, Don A., Sangster, R., Tarnai, John, and Rockwood, T. 1996. In Current issues in survey research: New directions for evaluation, #70. Chapter 4, eds. Braverman, Mark T., and Kay Slater, Jana, 4562. San Francisco, CA: Jossey-Bass.
Dillman, Don A., and Malani Christian, Leah. 2005. Survey mode as a source of instability in responses across surveys. Field Methods 17 (1): 3052.
Dillman, Don A., Smyth, Jolene, and Melani Christian, Leah. 2009. Internet, mail, and mixed-mode surveys: The tailored design method. New York: Wiley.
Fowler, Floyd Jackson. Jr, Roman, Anthony M., and Xiao Di, Zhu. 1998. Mode effects in a survey of Medicare prostate surgery patients. Public Opinion Quarterly 62 (1): 2946.
Fricker, S., Galesic, M., Tourangeau, R., Yan, and T. 2005. An experimental comparison of web and telephone surveys. Public Opinion Quarterly 69 (3): 370-92.
Gingerich, D. W. 2010. Understanding off-the-books politics: Conducting inference on the determinants of sensitive behavior with randomized response surveys. Political Analysis 18: 349-80.
Groves, Robert M. 1989. Survey errors and survey costs. New York: John Wiley and Sons.
Groves, R. M., Kahn, and R. L. 1979. Surveys by telephone: A national comparison with personal interviews. New York: Academic Press.
Groves, R. M., Peytcheva, and E. 2008. The impact of nonresponse rates on nonresponse bias. Public Opinion Quarterly 72: 123.
Groves, Robert M., Presser, Stanley, and Dipko, Sarah. 2004. The role of topic interest in survey participation decisions. Public Opinion Quarterly 68 (1): 231.
Groves, Robert, Singer, Eleanor, and Corning, Amy. 2000. Leverage-salience theory of survey participation. Public Opinion Quarterly 64: 299308.
Ho, Daniel, Imai, Kosuke, King, Gary, and Stuart, Elizabeth. 2007a. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis 15 (3): 199236.
Ho, Daniel, Imai, Kosuke, King, Gary, and Stuart, Elizabeth. 2007b. Matchit: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software. http://gking.harvard.edu/matchit/.
Holbrook, A. L., Green, M. C., Krosnick, and J. A. 2003. Telephone versus face-to-face interviewing of national probability samples with long questionnaires. Public Opinion Quarterly 67: 79125.
Holbrook, Allyson L., and Krosnick, Jon A. 2010. Social desirability bias in voter turnout reports: Tests using the item count technique. Public Opinion Quarterly 74 (1): 3767.
Hox, J. J., and de Leeuw, E. D. 1994. A comparison of nonresponse in mail, telephone, and face-to-face surveys: Applying multilevel models to meta-analysis. Quality and Quantity 28: 329-44.
Iannacchione, Vincent. 2011. The changing role of address-based sampling in survey research. Public Opinion Quarterly 75 (3): 556-75.
Keeter, Scott, Miller, Carolyn, Kohut, Andrew, Groves, Robert M., and Presser, Stanley. 2000. Consequences of reducing non-response in a national telephone survey. Public Opinion Quarterly 64: 125-48.
Kreuter, F., Presser, S., Tourangeau, and R. 2008. Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity. Public Opinion Quarterly 72: 847-65.
Krosnick, J. A. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5: 213-36.
Krosnick, J. A. 1999. Maximizing questionnaire quality. In Measures of political attitudes, eds. Robinson, J. P., Shaver, P. R., Wrightsman, and L. S. New York: Academic Press.
London, K., Williams, and L. 1990. A comparison of abortion underreporting in an in-person interview and self-administered questionnaire. Toronto: Paper presented at the Annual Meeting of the Population Association of America.
Lupia, Arthur, Krosnick, Jon, Luevano, Pat, DeBell, Matthew, and Donakowski, Darrell. 2009. Users Guide to the ANES 2008 Time Series Study. Ann Arbor, MI, and Palo Alto, CA: University of Michigan and Stanford University. http://www.electionstudies.org/studypages/2008prepost/2008prepost_UsersGuide.pdf.
Martinez, Michael D. 2003. Comment on voter turnout and the National Election Studies. Political Analysis 11 (2): 187-92.
McDonald, Michael P. 2003. On the overreport bias of the National Election Study turnout rate. Political Analysis 11 (2): 180-86.
McDonald, Michael P. 2007. The true electorate: A cross validation of voter registration files and election survey demographics. Public Opinion Quarterly 71 (4): 588602.
Mebane, Walter R. Jr, and Sekhon, Jasjeet S. 1998. GENetic Optimization Using Derivatives (GENOUD). Software package. http://sekhon/berkeley.edu/rgenoud/.
Mensch, Barbara S., and Kandel, Denise B. 1988. Underreporting of substance use in a National Longitudinal Youth Cohort. Public Opinion Quarterly 52 (Spring): 100-24.
Merkle, Daniel M., and Edelman, Murray. 2002. Non-response in exit polls: A comprehensive analysis. In Survey non- response, eds. Groves, Robert M., Dillman, Don A., Eltinge, John L., and Little, Roderick J. A., 243-58. New York: John Wiley & Sons Inc.
Mulligan, Kenneth, Krosnick, Jon A., Smith, Wendy, Green, Melanie, and Bizer, George. 2001. Nondifferentiation on attitude rating scales: A test of survey satisficing theory. Unpublished manuscript.
O'Muircheartaigh, Colm, English, Ned, Latterner, Michael, Eckman, Stephanie, and Dekker, Katie. 2009. Modeling the need for traditional vs. commercially available address listings for in-person surveys: Results from a National Validation of Addresses JSM. Proceedings of the American Statistical Association, Section on Survey Research Methods 6193–202.
Peterson, Robert A., and Kerin, Roger A. 1981. The quality of self-report data: Review and synthesis. In Review of marketing, eds. Enis, Ben M. and Roering, Kenneth J., 520. Chicago: American Marketing Association.
Pew Research Center for the People and the Press. 2004. Polls face growing resistance, but still representative: Survey experiment shows. http://people-press.org/reportsdisplay.php3?ReportID=211.
Presser, Stanley, Traugott, Michael W., and Traugott, Santa. 1990. Vote “over” reporting in surveys: The records or the respondents. Technical Report No. 39. Ann Arbor, MI: National Election Studies.
Rivers, Douglas. n.d. Sample matching: Representative sampling from Internet polling. Polimetrix.
Rosen, R., Gomes, and T. 2004. Converting CES reporters from TDE to web data collection. Toronto, Canada: Paper presented at the Joint Statistical Meetings.
Schuman, Howard, and Presser, Stanley. 1981. Questions and answers in attitude surveys: Experiments on question form, wording and context. New York: Academic Press.
Schuman, Howard. 1992. Context effects: State of the past/state of the art. In Context effects in social and psychological research, eds. Schwarz, N. and Sudman, S., 520. New York: Springer.
Schwarz, N. 1996. Cognition and communication judgmental biases, research methods, and the logic of conversation. Mahwah, NJ: Lawrence Erlbaum.
Schwarz, N., Strack, and E. 1985. Cognitive and affective processes in judgments of subjective well-being: A preliminary model. In Economic psychology, eds. Brandstatter, H. and Kirehler, E., 439-47. Linz: Austria: R. Trauner.
Sekhon, Jasjeet Singh, and Mebane, Walter R. Jr. 1998. Genetic optimization using derivatives: Theory and application to nonlinear models. Political Analysis 7: 189203.
Sekhon, Jasjeet S. 2011. Multivariate and propensity score matching software with automated balance optimization: The matching package for R. Journal of Statistical Software 42 (7): 152.
Shettle, Carolyn, and Mooney, Geraldine. 1999. Monetary incentives in government surveys. Journal of Official Statistics 15: 231-50.
Smyth, Jolene D., Dillman, Don, Malani Christian, Leah, and Stern, Michael J. 2006. Effects of using visual design principles to group response options in web surveys. International Journal of Internet Science 1: 616.
Stoop, I. A. L. 2005. The hunt for the last respondent. The Hague: Social and Cultural Planning Office (SCP).
Seymour, Sudman, Bradburn, Norman M., and Schwarz, Norbert. 1996. Thinking about answers. San Francisco, CA: Josey-Bass.
Tourangeau, Roger, Couper, Mick, and Conrad, Frederick. 2004. Spacing, position and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly 68 (3): 368-93.
Tourangeau, R., Rasinski, and K. A. 1988. Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin 103: 299314.
Tourangeau, Roger, and Yan, Ting. 2007. Sensitive questions in surveys. Psychological Bulletin 133: 859-83.
Traugott, Santa. 1989. Validating self-reported vote: 1964–1988. ANES Technical Report Series, No. nes010152. Ann Arbor, MI: National Election Studies.
U.S Census Bureau. American community survey 2006–2008 three-year estimates. Data profile: New Mexico: Bernalillo County: Hispanics (All).
Visser, Penny S., Krosnick, Jon A., Marquette, Jesse, and Curtin, Michael. 1996. Mail surveys for election forecasting? An evaluation of the Columbus Dispatch Poll. Public Opinion Quarterly 60 (2): 181227.
Zickhur, K., Smith, and A. 2012. Digital differences. Washington, DC: Pew Internet and American Life Project. http://pewinternet.org/Reports/2012/Digital-differences.aspx (accessed August 7, 2012).
MathJax
MathJax is a JavaScript display engine for mathematics. For more information see http://www.mathjax.org.

Related content

Powered by UNSILO
Type Description Title
PDF
Supplementary materials

Atkeson et al. supplementary material
Tables S1-S4

 PDF (100 KB)
100 KB

Nonresponse and Mode Effects in Self- and Interviewer-Administered Surveys

  • Lonna Rae Atkeson (a1), Alex N. Adams (a2) and R. Michael Alvarez (a3)

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed.