Skip to main content Accessibility help
×
×
Home

How Much Should We Trust Estimates from Multiplicative Interaction Models? Simple Tools to Improve Empirical Practice

  • Jens Hainmueller (a1), Jonathan Mummolo (a2) and Yiqing Xu (a3)
  • Please note a correction has been issued for this article.

Abstract

Multiplicative interaction models are widely used in social science to examine whether the relationship between an outcome and an independent variable changes with a moderating variable. Current empirical practice tends to overlook two important problems. First, these models assume a linear interaction effect that changes at a constant rate with the moderator. Second, estimates of the conditional effects of the independent variable can be misleading if there is a lack of common support of the moderator. Replicating 46 interaction effects from 22 recent publications in five top political science journals, we find that these core assumptions often fail in practice, suggesting that a large portion of findings across all political science subfields based on interaction models are fragile and model dependent. We propose a checklist of simple diagnostics to assess the validity of these assumptions and offer flexible estimation strategies that allow for nonlinear interaction effects and safeguard against excessive extrapolation. These statistical routines are available in both R and STATA.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      How Much Should We Trust Estimates from Multiplicative Interaction Models? Simple Tools to Improve Empirical Practice
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      How Much Should We Trust Estimates from Multiplicative Interaction Models? Simple Tools to Improve Empirical Practice
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      How Much Should We Trust Estimates from Multiplicative Interaction Models? Simple Tools to Improve Empirical Practice
      Available formats
      ×

Copyright

This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike licence (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits noncommercial re-use, distribution, and reproduction in any medium, provided the same Creative Commons licence is included and the original work is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use.

Corresponding author

Footnotes

Hide All

Authors’ note: We thank Licheng Liu for excellent research assistance. We thank David Broockman, Daniel Carpenter, James Fowler, Justin Grimmer, Erin Hartman, Seth Hill, Macartan Humphreys, Kosuke Imai, Dorothy Kronick, Gabe Lenz, Adeline Lo, Neil Malhotra, John Marshall, Marc Ratkovic, Molly Roberts, Jas Sekhon, Vera Troeger, Sean Westwood and participants at the PolMeth, APSA and MPSA annual meetings and at methods workshops at Massachusetts Institute of Technology, Harvard University, Princeton University, Columbia University and University of California, San Diego for helpful feedback. We also thank the authors of the studies we replicate for generously sharing code and data. Replication data available in Hainmueller, Mummolo, and Xu (2018).

Contributing Editor: Jonathan Nagler

Footnotes

References

Hide All
Adams, James, Clark, Michael, Ezrow, Lawrence, and Glasgow, Garrett. 2006. Are niche parties fundamentally different from mainstream parties? The causes and the electoral consequences of western European parties’ policy shifts, 1976–1998. American Journal of Political Science 50(3):513529.
Aiken, Leona S., West, Stephen G., and Reno, Raymond R.. 1991. Multiple regression: Testing and interpreting interactions . London: Sage Publications.
Aklin, Michaël, and Urpelainen, Johannes. 2013. Political competition, path dependence, and the strategy of sustainable energy transitions. American Journal of Political Science 57(3):643658.
Anderson, James H.2013 Sunshine works–comment on ‘the adverse effects of sunshine: A field experiment on legislative transparency in an authoritarian assembly’. World Bank Policy Research Working Paper (6602).
Athey, Susan, and Wager, Stefan. 2018. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association , doi:10.1080/01621459.2017.1319839.
Banks, Antoine J., and Valentino, Nicholas A.. 2012. Emotional substrates of white racial attitudes. American Journal of Political Science 56(2):286297.
Beck, Nathaniel, King, Gary, and Zeng, Langche. 2000. Improving quantitative studies of international conflict: A conjecture. American Political Science Review 94(1):2135.
Berry, William D., Golder, Matt, and Milton, Daniel. 2012. Improving tests of theories positing interaction. Journal of Politics 74(3):653671.
Bodea, Cristina, and Hicks, Raymond. 2015a. International finance and central bank independence: Institutional diffusion and the flow and cost of capital. The Journal of Politics 77(1):268284.
Bodea, Cristina, and Hicks, Raymond. 2015b. Price stability and central bank independence: Discipline, credibility, and democratic institutions. International Organization 69(1):3561.
Brambor, Thomas, Clark, William Roberts, and Golder, Matt. 2006. Understanding interaction models: Improving empirical analyses. Political Analysis 14:6382.
Braumoeller, Bear F. 2004. Hypothesis testing and multiplicative interaction terms. International Organization 58(04):807820.
Cameron, A. Colin, and Miller, Douglass L.. 2015. A Practitioner’s guide to cluster-robust inference. The Journal of Human Resources 50(2):317372.
Carpenter, Daniel, and Moore, Colin D.. 2014. When canvassers became activists: Antislavery petitioning and the political mobilization of American women. American Political Science Review 108(3):479498.
Chapman, Terrence L. 2009. Audience beliefs and international organization legitimacy. International Organization 63(04):733764.
Clark, Michael, and Leiter, Debra. 2014. Does the ideological dispersion of parties mediate the electoral impact of valence? A cross-national study of party support in nine western European democracies. Comparative Political Studies 47(2):171202.
Clark, William Roberts, and Golder, Matt. 2006. Rehabilitating Duverger’s theory testing the mechanical and strategic modifying effects of electoral laws. Comparative Political Studies 39(6):679708.
Cleveland, William S., and Devlin, Susan J.. 1988. Locally weighted regression: An approach to regression analysis by local fitting. Journal of the American Statistical Association 83(403):596610.
Fan, Jianqing, Heckman, Nancy E., and Wand, Matt P.. 1995. Local polynomial kernel regression for generalized linear models and quasi-likelihood functions. Journal of the American Statistical Association 90(429):141.
Friedrich, Robert J. 1982. In defense of multiplicative terms in multiple regression equations. American Journal of Political Science 26(4):797833.
Grimmer, Justin, Messing, Solomon, and Westwood, Sean J.. 2014 Estimating heterogeneous treatment effects and the effects of heterogeneous treatments with ensemble methods.
Hainmueller, Jens, and Hazlett, Chad. 2014. Kernel regularized least squares: Reducing misspecification bias with a flexible and interpretable machine learning approach. Political Analysis 22(2):143168.
Hainmueller, Jens, Mummolo, Jonathan, and Xu, Yiqing. 2018. Replication data for: How much should we trust estimates from multiplicative interaction models? Simple tools to improve empirical practice. https://doi.org/10.7910/DVN/Q1V0OG, Harvard Dataverse, V1.
Hastie, Trevor, and Tibshirani, Robert. 1986. Generalized additive models. Statistical Science 1(3):297318.
Hellwig, Timothy, and Samuels, David. 2007. Voting in open economies the electoral consequences of globalization. Comparative Political Studies 40(3):283306.
Hicken, Allen, and Simmons, Joel W.. 2008. The personal vote and the efficacy of education spending. American Journal of Political Science 52(1):109124.
Hosking, J. R. M. 1990. L-moments: Analysis and estimation of distributions using linear combinations of order statistics. Journal of the Royal Statistical Society. Series B (Methodological) 52(1):105124.
Huddy, Leonie, Mason, Lilliana, and Aarøe, Lene. 2015. Expressive partisanship: Campaign involvement, political emotion, and partisan identity. American Political Science Review 109(1):117.
Imai, Kosuke, Keele, Luke, and Yamamoto, Teppei. 2010. Identification, inference and sensitivity analysis for causal mediation effects. Statistical Science 25(1):5171.
Imai, Kosuke, and Ratkovic, Marc. 2013. Estimating treatment effect heterogeneity in randomized program evaluation. Annals of Applied Statistics 7(1):443470.
Jaccard, James, and Turrisi, Robert. 2003. Interaction effects in multiple regression , vol. 72. London: Sage Publications.
Kam, Cindy D., and Franzese, Robert J. Jr. 2007. Modeling and interpreting interactive hypotheses in regression analysis . Ann Arbor: The University of Michigan Press.
Kim, Henry A., and LeVeck, Brad L.. 2013. Money, reputation, and incumbency in US house elections, or why marginals have become more expensive. American Political Science Review 107(3):492504.
King, Gary, and Zeng, Langche. 2006. The dangers of extreme counterfactuals. Political Analysis 14(2):131159.
Li, Qi, and Racine, Jeffrey S.. 2010. Smooth varying-coefficient estimation and inference for qualitative and quantitative data. Econometric Theory 26:131.
Lin, Winston. 2013. Agnostic notes on regression adjustments to experimental data: Reexamining Freedman?s critique. The Annals of Applied Statistics 7(1):295318.
Malesky, Edmund, Schuler, Paul, and Tran, Anh. 2012. The adverse effects of sunshine: A field experiment on legislative transparency in an authoritarian assembly. American Political Science Review 106(04):762786.
Miratrix, Luke W., Sekhon, Jasjeet S., and Yu, Bin. 2013. Adjusting treatment effect estimates by post-stratification in randomized experiments. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 75(2):369396.
Mosteller, Frederick, and Tukey, John Wilder. 1977. Data analysis and regression: a second course in statistics . Reading, MA: Addison-Wesley Publishing Co.
Neblo, Michael A., Esterling, Kevin M., Kennedy, Ryan P., Lazer, David M.J., and Sokhey, Anand E.. 2010. Who wants to deliberate—and why? American Political Science Review 104(3):566583.
Open Science Collaboration et al.2015. Estimating the reproducibility of psychological science. Science 349(6251): aac4716.
Pelc, Krzysztof J. 2011. Why do some countries get better WTO accession terms than others? International Organization 65(4):639672.
Petersen, Michael Bang, and Aarøe, Lene. 2013. Politics in the mind’s eye: Imagination as a link between social and political cognition. American Political Science Review 107(2):275293.
Somer-Topcu, Zeynep. 2009. Timely decisions: The effects of past national elections on party policy change. The Journal of Politics 71(1):238248.
Tavits, Margit. 2008. Policy positions, issue importance, and party competition in new democracies. Comparative Political Studies 41(1):4872.
Truex, Rory. 2014. The returns to office in a ‘rubber stamp’ parliament. American Political Science Review 108(2):235251.
Vernby, Kåre. 2013. Inclusion and public policy: Evidence from Sweden’s introduction of noncitizen suffrage. American Journal of Political Science 57(1):1529.
Williams, Laron K. 2011. Unsuccessful success? Failed no-confidence motions, competence signals, and electoral support. Comparative Political Studies 44(11):14741499.
Wood, Simon N. 2003. Thin plate regression splines. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 65(1):95114.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax

Keywords

Type Description Title
UNKNOWN
Supplementary materials

Hainmueller et al. supplementary material
Hainmueller et al. supplementary material 1

 Unknown (263 KB)
263 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed

A correction has been issued for this article: