Hostname: page-component-848d4c4894-hfldf Total loading time: 0 Render date: 2024-04-30T21:30:46.471Z Has data issue: false hasContentIssue false
  • English
  • Français

Navigating Potential Pitfalls in Difference-in-Differences Designs: Reconciling Conflicting Findings on Mass Shootings’ Effect on Electoral Outcomes

Published online by Cambridge University Press:  12 April 2024

HANS J. G. HASSELL Open the ORCID record for HANS J. G. HASSELL [Opens in a new window]  and
JOHN B. HOLBEIN Open the ORCID record for JOHN B. HOLBEIN [Opens in a new window]
HANS J. G. HASSELL*
Affiliation:
Florida State University, United States
JOHN B. HOLBEIN*
Affiliation:
University of Virginia, United States
*
Hans J. G. Hassell, Professor, Department of Political Science, Florida State University, United States, hans.hassell@fsu.edu.
Corresponding author: John B. Holbein, Associate Professor of Public Policy, Politics, and Education, Frank Batten School of Leadership and Public Policy, University of Virginia, United States, holbein@virginia.edu.
Get access Rights & Permissions [Opens in a new window]

Abstract

Work on the electoral effects of gun violence in the U.S. relying on difference-in-differences designs has produced findings ranging from null to substantively large effects. However, as difference-in-difference designs, on which this research relies, have exploded in popularity, scholars have documented several methodological issues including potential violations of parallel-trends and unaccounted for treatment effect heterogeneity. These pitfalls (and their solutions) have not been fully explored in political science. We apply these advancements to the unresolved debate on gun violence’s effects on U.S. electoral outcomes. We show that studies finding a large positive effect of gun violence on Democratic vote shares are a product of a failure to properly specify difference-in-differences models when underlying assumptions are unlikely to hold. Once these biases are corrected, shootings show little evidence of sparking large electoral change. Our work clarifies an unresolved debate and provides a cautionary guide for scholars currently employing difference-in-differences designs.

Type
Research Article
Information
American Political Science Review , First View , pp. 1 - 21
Copyright
© The Author(s), 2024. Published by Cambridge University Press on behalf of American Political Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Abadie, Alberto, Athey, Susan, Imbens, Guido W., and Wooldridge, Jeffrey M.. 2023. “When Should You Adjust Standard Errors for Clustering?” Quarterly Journal of Economics 138 (1): 1–35.CrossRefGoogle Scholar
Abadie, Alberto, Diamond, Alexis, and Hainmueller, Jens. 2015. “Comparative Politics and the Synthetic Control Method.” American Journal of Political Science 59 (2): 495–510.CrossRefGoogle Scholar
Angrist, Joshua D., and Pischke, Jörn-Steffen. 2008. Mostly Harmless Econometrics. Princeton, NJ: Princeton University Press.CrossRefGoogle Scholar
Angrist, Joshua D., and Pischke, Jörn-Steffen. 2010. “The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con Out of Econometrics.” Journal of Economic Perspectives 24 (2): 3–30.CrossRefGoogle Scholar
Arceneaux, Kevin, Gerber, Alan S., and Green, Donald P.. 2006. “Comparing Experimental and Matching Methods Using a Large-Scale Voter Mobilization Experiment.” Political Analysis 14 (1): 37–62.CrossRefGoogle Scholar
Arkhangelsky, Dmitry, Athey, Susan, Hirshberg, David A., Imbens, Guido W., and Wager, Stefan. 2021. “Synthetic Difference-in-Differences.” American Economic Review 111 (12): 4088–118.CrossRefGoogle Scholar
Armitage, Seth. 1995. “Event Study Methods and Evidence on Their Performance.” Journal of Economic Surveys 9 (1): 25–52.CrossRefGoogle Scholar
Bai, Jushan, and Ng, Serena. 2021. “Matrix Completion, Counterfactuals, and Factor Analysis of Missing Data.” Journal of the American Statistical Association 116 (536): 1746–63.CrossRefGoogle Scholar
Baker, Andrew C., Larcker, David F., and Wang, Charles C. Y.. 2022. “How Much Should We Trust Staggered Difference-in-Differences Estimates?.” Journal of Financial Economics 144 (2): 370–95.CrossRefGoogle Scholar
Barney, David J., and Schaffner, Brian F.. 2019. “Reexamining the Effect of Mass Shootings on Public Support for Gun Control.” British Journal of Political Science 49 (4): 1555–65.CrossRefGoogle Scholar
Beck, Nathaniel L., Katz, Jonathan N., and Mignozzetti, Umberto G.. 2014. “Of Nickell Bias and Its Cures: Comment on Gaibulloev, Sandler, and Sul.” Political Analysis 22 (2): 274–8.CrossRefGoogle Scholar
Berge, Laurent, Krantz, Sebastian, and McDermott, Grant. 2022. “Fixest: Fast Fixed-Effects Estimations.” CRAN.Google Scholar
Bilinski, Alyssa, and Hatfield, Laura A.. 2018. “Nothing to See Here? Non-Inferiority Approaches to Parallel Trends and Other Model Assumptions.” Preprint, arXiv:1805.03273.Google Scholar
Binder, John. 1998. “The Event Study Methodology since 1969.” Review of Quantitative Finance and Accounting 11 (2): 111–37.CrossRefGoogle Scholar
Borusyak, Kirill, Jaravel, Xavier, and Spiess, Jann. 2021. “Revisiting Event Study Designs: Robust and Efficient Estimation.” Preprint, arXiv:2108.12419.Google Scholar
Broderick, Tamara, Giordano, Ryan, and Meager, Rachael. 2020. “An Automatic Finite-Sample Robustness Metric: Can Dropping a Little Data Change Conclusions?” Preprint, arXiv:2011.14999.Google Scholar
Callaway, Brantly, and Li, Tong. 2019. “Quantile Treatment Effects in Difference in Differences Models with Panel Data.” Quantitative Economics 10 (4): 1579–618.CrossRefGoogle Scholar
Callaway, Brantly, and Sant’Anna, Pedro H. C.. 2021. “Difference-in-Differences with Multiple Time Periods.” Journal of Econometrics 225 (2): 200–30.CrossRefGoogle Scholar
Cameron, A. Colin, and Miller, Douglas L.. 2015. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources 50 (2): 317–72.CrossRefGoogle Scholar
Clarke, Damian, and Tapia-Schythe, Kathya. 2021. “Implementing the Panel Event Study.” Stata Journal 21 (4): 853–84.CrossRefGoogle Scholar
Cunningham, Scott. 2021. Causal Inference: The Mixtape. New Haven, CT: Yale University Press.Google Scholar
De Chaisemartin, Clément, and d’Haultfoeuille, Xavier. 2020. “Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects.” American Economic Review 110 (9): 2964–96.CrossRefGoogle Scholar
De Chaisemartin, Clément, and D’Haultfoeuille, Xavier. 2023. “Two-Way Fixed Effects and Differences-in-Differences with Heterogeneous Treatment Effects: A Survey.” The Econometrics Journal 26 (3): C1–C20.CrossRefGoogle Scholar
de Chaisemartin, Clément, D’Haultfoeuille, Xavier, and Guyonvarch, Yannick. 2019. “DID_MULTIPLEGT: Stata Module to Estimate Sharp Difference-in-Difference Designs with Multiple Groups and Periods.” Statistical Software Components S458643, Boston College Department of Economics.Google Scholar
DeSilver, Drew. 2016. “The Growing Democratic Domination of Nation’s Largest Counties.” Pew Research Center, July 21.Google Scholar
Ding, Peng, and Li, Fan. 2019. “A Bracketing Relationship between Difference-in-Differences and Lagged-Dependent-Variable Adjustment.” Political Analysis 27 (4): 605–15.CrossRefGoogle Scholar
Freyaldenhoven, Simon, Hansen, Christian, Pérez, Jorge Pérez, and Shapiro, Jesse. 2022. “Xtevent: Stata Module to Estimate and Visualize Linear Panel Event-Study Models.” Working Paper.Google Scholar
Freyaldenhoven, Simon, Hansen, Christian, Pérez, Jorge Pérez, and Shapiro, Jesse M.. 2021. “Visualization, Identification, and Estimation in the Linear Panel Event-Study Design.” Working Paper, National Bureau of Economic Research.CrossRefGoogle Scholar
Freyaldenhoven, Simon, Hansen, Christian, and Shapiro, Jesse M.. 2019. “Pre-Event Trends in the Panel Event-Study Design.” American Economic Review 109 (9): 3307–38.CrossRefGoogle Scholar
García-Montoya, Laura, Arjona, Ana, and Lacombe, Matthew. 2022. “Violence and Voting in the United States: How School Shootings Affect Elections.” American Political Science Review 116 (3): 807–26.CrossRefGoogle Scholar
Goodman-Bacon, Andrew. 2019. “So You’ve Been Told to Do My Difference-in-Differences Thing: A Guide.” Working Paper, Vanderbilt University.Google Scholar
Goodman-Bacon, Andrew. 2021. “Difference-in-Differences with Variation in Treatment Timing.” Journal of Econometrics 225 (2): 254–77.CrossRefGoogle Scholar
Goodman-Bacon, Andrew, Goldring, Thomas, and Nichols, Austin. 2019. “BACONDECOMP: Stata Module to Perform a Bacon Decomposition of Difference-in-Differences Estimation.” Statistical Software Components S458676, Boston College Department of Economics.Google Scholar
Goss, Kristin. 2010. Disarmed: The Missing Movement for Gun Control in America. Princeton, NJ: Princeton University Press.Google Scholar
Green, Donald P., Leong, Terence Y., Kern, Holger L., Gerber, Alan S., and Larimer, Christopher W.. 2009. “Testing the Accuracy of Regression Discontinuity Analysis Using Experimental Benchmarks.” Political Analysis 17 (4): 400–17.CrossRefGoogle Scholar
Grimmer, Justin, Hersh, Eitan, Meredith, Marc, Mummolo, Jonathan, and Nall, Clayton. 2018. “Obstacles to Estimating Voter ID Laws’ Effect on Turnout.” Journal of Politics 80 (3): 1045–51.CrossRefGoogle Scholar
Hansen, Ben B., and Bowers, Jake. 2008. “Covariate Balance in Simple, Stratified and Clustered Comparative Studies.” Statistical Science 23(2): 219–36.CrossRefGoogle Scholar
Hartman, Erin, and Hidalgo, F. Daniel. 2018. “An Equivalence Approach to Balance and Placebo Tests.” American Journal of Political Science 62 (4): 1000–13.CrossRefGoogle Scholar
Hartman, Todd K., and Newman, Benjamin J.. 2019. “Accounting for Pre-Treatment Exposure in Panel Data: Re-Estimating the Effect of Mass Public Shootings.” British Journal of Political Science 49 (4): 1567–76.CrossRefGoogle Scholar
Hasegawa, Raiden B., Webster, Daniel W., and Small, Dylan S.. 2019. “Evaluating Missouri’s Handgun Purchaser Law: A Bracketing Method for Addressing Concerns about History Interacting with Group.” Epidemiology 30 (3): 371–9.CrossRefGoogle Scholar
Hassell, Hans J. G., Holbein, John B., and Baldwin, Matthew. 2020. “Mobilize for Our Lives? School Shootings and Democratic Accountability in US Elections.” American Political Science Review 114 (4): 1375–85.CrossRefGoogle Scholar
Healy, Andrew, and Lenz, Gabriel S.. 2017. “Presidential Voting and the Local Economy: Evidence from Two Population-Based Data Sets.” Journal of Politics 79 (4): 1419–32.CrossRefGoogle Scholar
Holbein, John B., and Hassell, Hans J. G.. 2024. “Replication Data for: Navigating Potential Pitfalls in Difference-in-Differences Designs: Reconciling Conflicting Findings on Mass Shootings’ Effect on Electoral Outcomes.” Harvard Dataverse. Dataset. https://doi.org/10.7910/DVN/GH69TI.CrossRefGoogle Scholar
Hollenbach, Florian. 2021. “Comparing Staggered DiD.” https://github.com/fhollenbach/did_compare/blob/main/ComparingDiD.md.Google Scholar
Kahn-Lang, Ariella, and Lang, Kevin. 2020. “The Promise and Pitfalls of Differences-in-Differences: Reflections on 16 and Pregnant and Other Applications.” Journal of Business & Economic Statistics 38 (3): 613–20.CrossRefGoogle Scholar
Keele, Luke. 2015. “The Statistics of Causal Inference: A View from Political Methodology.” Political Analysis 23 (3): 313–35.CrossRefGoogle Scholar
Keele, Luke, and Minozzi, William. 2013. “How Much Is Minnesota Like Wisconsin? Assumptions and Counterfactuals in Causal Inference with Observational Data.” Political Analysis 21 (2): 193–216.CrossRefGoogle Scholar
Kreif, Noémi, Grieve, Richard, Hangartner, Dominik, Turner, Alex James, Nikolova, Silviya, and Sutton, Matt. 2016. “Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units.” Health Economics 25 (12): 1514–28.CrossRefGoogle ScholarPubMed
Liu, Licheng, Wang, Ye, and Xu, Yiqing. 2024. “A Practical Guide to Counterfactual Estimators for Causal Inference with Time-Series Cross-Sectional Data.” American Journal of Political Science 68 (1): 160–76.CrossRefGoogle Scholar
Luca, Michael, Malhotra, Deepak, and Poliquin, Christopher. 2020. “The Impact of Mass Shootings on Gun Policy.” Journal of Public Economics 181: 104083.CrossRefGoogle Scholar
MacKinnon, James G., Nielsen, Morten Ørregaard, and Webb, Matthew D.. 2023. “Testing for the Appropriate Level of Clustering in Linear Regression Models.” Journal of Econometrics 235 (2): 2027–56.CrossRefGoogle Scholar
Manski, Charles F., and Pepper, John V.. 2018. “How Do Right-to-Carry Laws Affect Crime Rates? Coping with Ambiguity Using Bounded-Variation Assumptions.” Review of Economics and Statistics 100 (2): 232–44.CrossRefGoogle Scholar
Marcus, Michelle, and Sant’Anna, Pedro H. C.. 2021. “The Role of Parallel Trends in Event Study Settings: An Application to Environmental Economics.” Journal of the Association of Environmental and Resource Economists 8 (2): 235–75.CrossRefGoogle Scholar
Marsh, Wayde Z. C. 2022. “Trauma and Turnout: The Political Consequences of Traumatic Events.” American Political Science Review 117 (3): 1036–52.CrossRefGoogle Scholar
Mou, Hongyu, Liu, Licheng, and Yiqing, Xu. 2022a. “Package ‘panelView.’”Google Scholar
Mou, Hongyu, Liu, Licheng, and Yiqing, Xu. 2022b. “panelView: Panel Data Visualization in R and Stata.” SSRN 4202154.CrossRefGoogle Scholar
Musu-Gillette, Lauren, Zhang, Anlan, Wang, Ke, Kemp, Jana, Diliberti, Melissa, and Oudekerk, Barbara A.. 2018. “Indicators of School Crime and Safety: 2017.” Report, National Center for Educational Statistics (NCES 2018-036).Google Scholar
Olea, Montiel, Luis, José, and Plagborg-Møller, Mikkel. 2019. “Simultaneous Confidence Bands: Theory, Implementation, and an Application to SVARs.” Journal of Applied Econometrics 34 (1): 1–17.CrossRefGoogle Scholar
Porreca, Zachary. 2022. “Synthetic Difference-in-Differences Estimation with Staggered Treatment Timing.” Economics Letters 220: 110874.CrossRefGoogle Scholar
Rambachan, Ashesh, and Roth, Jonathan. 2021. “An Honest Approach to Parallel Trends.” Unpublished Manuscript, Harvard University.Google Scholar
Rambachan, Ashesh, Roth, Jonathan, and Bravo, Mauricio Caceres. 2021. “HonestDiD.” https://github.com/asheshrambachan/HonestDiD/tree/bc576d5f338dd01ebb641f30215792c0605bc08a.Google Scholar
Rogowski, Jon C., and Tucker, Patrick D.. 2019. “Critical Events and Attitude Change: Support for Gun Control after Mass Shootings.” Political Science Research and Methods 7 (4): 903–11.CrossRefGoogle Scholar
Rossin-Slater, Maya, Schnell, Molly, Schwandt, Hannes, Trejo, Sam, and Uniat, Lindsey. 2020. “Local Exposure to School Shootings and Youth Antidepressant Use.” Proceedings of the National Academy of Sciences 117 (38): 23484–9.CrossRefGoogle ScholarPubMed
Roth, Jonathan. 2022. “Pre-Test with Caution: Event-Study Estimates after Testing for Parallel Trends.” American Economic Review: Insights 4(3): 305–22.Google Scholar
Roth, Jonathan, Sant’Anna, Pedro H. C., Bilinski, Alyssa, and Poe, John. 2022. “What’s Trending in Difference-in-Differences? A Synthesis of the Recent Econometrics Literature.” Journal of Econometrics 235 (2): 2218–44.CrossRefGoogle Scholar
Schmidheiny, Kurt, and Siegloch, Sebastian. 2023. “On Event Studies and Distributed Lags in Two-Way Fixed Effects Models: Identification, Equivalence, and Generalization.” Journal of Applied Econometrics 38(5), 695–713.CrossRefGoogle Scholar
Sides, John, Vavreck, Lynn, and Warshaw, Christopher. 2022. “The Effect of Television Advertising in United States Elections.” American Political Science Review 116 (2): 702–18.CrossRefGoogle Scholar
Stock, James H., and Watson, Mark W.. 2020. Introduction to Econometrics. New York: Pearson.Google Scholar
Sun, Liyang. 2021. “EVENTSTUDYINTERACT: Stata Module to Implement the Interaction Weighted Estimator for an Event Study.” Statistical Software Components S458978, Boston College Department of Economics.Google Scholar
Sun, Liyang, and Abraham, Sarah. 2021. “Estimating Dynamic Treatment Effects in Event Studies with Heterogeneous Treatment Effects.” Journal of Econometrics 225 (2): 175–99.CrossRefGoogle Scholar
U.S. Government Accountability Office. 2020. “K-12 Education: Characteristics of School Shootings.” GAO-20-455.Google Scholar
Wing, Coady, Simon, Kosali, and Bello-Gomez, Ricardo A.. 2018. “Designing Difference in Difference Studies: Best Practices for Public Health Policy Research.” Annual Review of Public Health 39: 453–69.CrossRefGoogle ScholarPubMed
Xu, Yiqing. 2023. “Causal Inference with Time-Series Cross-Sectional Data: A Reflection.” In Oxford Handbook of Engaged Methodological Pluralism in Political Science, Vol. 1, eds. Janet M. Box-Steffensmeier, Dino P. Christenson, and Valeria Sinclair-Chapman, online edition. https://doi.org/10.1093/oxfordhb/9780192868282.013.30.CrossRefGoogle Scholar
Yousaf, Hasin. 2021. “Sticking to One’s Guns: Mass Shootings and the Political Economy of Gun Control in the United States.” Journal of the European Economic Association 19 (5): 2765–802.CrossRefGoogle Scholar
Zhang, Shuo. 2022. “DIDmultiplegt: Estimation in DID with Multiple Groups and Periods.” CRAN.Google Scholar
Supplementary material: File

Hassell and Holbein supplementary material

Hassell and Holbein supplementary material
Download Hassell and Holbein supplementary material(File)
File 2.1 MB