Hostname: page-component-76fb5796d-dfsvx Total loading time: 0 Render date: 2024-04-26T12:24:04.770Z Has data issue: false hasContentIssue false
  • English
  • Français

A Permutation-Based Changepoint Technique for Monitoring Effect Sizes

Published online by Cambridge University Press:  05 January 2021

Daniel Kent Open the ORCID record for Daniel Kent [Opens in a new window] ,
James D. Wilson  and
Skyler J. Cranmer
Daniel Kent*
Affiliation:
Department of Political Science, The Ohio State University, 2140 Derby Hall, 154 North Oval Mall, Columbus, OH 43210, USA. Email: kent.249@osu.edu
James D. Wilson
Affiliation:
Department of Mathematics and Statistics, University of San Francisco, San Francisco, CA 94117, USA. Email: jdwilson4@usfca.edu
Skyler J. Cranmer
Affiliation:
Department of Political Science, The Ohio State University, 2140 Derby Hall, 154 North Oval Mall, Columbus, OH 43210, USA. Email: cranmer.12@osu.edu
*
Corresponding author Daniel Kent
Get access Rights & Permissions [Opens in a new window]

Abstract

Across the social sciences, scholars regularly pool effects over substantial periods of time, a practice that produces faulty inferences if the underlying data generating process is dynamic. To help researchers better perform principled analyses of time-varying processes, we develop a two-stage procedure based upon techniques for permutation testing and statistical process monitoring. Given time series cross-sectional data, we break the role of time through permutation inference and produce a null distribution that reflects a time-invariant data generating process. The null distribution then serves as a stable reference point, enabling the detection of effect changepoints. In Monte Carlo simulations, our randomization technique outperforms alternatives for changepoint analysis. A particular benefit of our method is that, by establishing the bounds for time-invariant effects before interacting with actual estimates, it is able to differentiate stochastic fluctuations from genuine changes. We demonstrate the method’s utility by applying it to a popular study on the relationship between alliances and the initiation of militarized interstate disputes. The example illustrates how the technique can help researchers make inferences about where changes occur in dynamic relationships and ask important questions about such changes.

Keywords

changepoint analysispermutation teststime series cross-sectional data
Type
Article
Information
Political Analysis , Volume 30 , Issue 2 , April 2022 , pp. 167 - 178
Copyright
© The Author(s) 2021. Published by Cambridge University Press on behalf of the Society for Political Methodology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Edited by Sunshine Hillygus

References

Aminikhanghahi, S., and Cook, D. J.. 2017. “A Survey of Methods for Time Series Change Point Detection.” Knowledge and Information Systems 51(2):339–367.CrossRefGoogle ScholarPubMed
Barry, D., and Hartigan, J. A.. 1993. “A Bayesian Analysis for Change Point Problems.” Journal of the American Statistical Association 88(421):309–319.Google Scholar
Beck, N. 1983. “Time-Varying Parameter Regression Models.” American Journal of Political Science 27:557–600.CrossRefGoogle Scholar
Beck, N. 2001. “Time-Series–Cross-Section Data: What Have We Learned In The Past Few Years?” Annual Review of Political Science 4(1):271–293.CrossRefGoogle Scholar
Beck, N. 2008. “Time-Series-Cross-Section Methods.” In Oxford Handbook of Political Methodology, edited by J. M. Box-Steffensmeier, H. E. Brady, and D. Collier, 475–493. Oxford: Oxford University Press.Google Scholar
Beck, N., Katz, J. N., and Tucker, R.. 1998. “Taking Time Seriously: Time-Series-Cross-Section Analysis with a Binary Dependent Variable.” American Journal of Political Science 42:1260–1288.CrossRefGoogle Scholar
Blackwell, M. 2018. “Game Changers: Detecting Shifts in Overdispersed Count Data.” Political Analysis 26(2):230–239.CrossRefGoogle Scholar
Box-Steffensmeier, J. M., Freeman, J. R., Hitt, M. P., and Pevehouse, J. C.. 2014. Time Series Analysis for the Social Sciences. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Box-Steffensmeier, J. M., and Jones, B. S.. 2004. Event History Modeling: A Guide for Social Scientists. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Braumoeller, B. F. 2013. The Great Powers and the International System: Systemic Theory in Empirical Perspective. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Carter, D. B., and Signorino, C. S.. 2010. “Back to the Future: Modeling Time Dependence in Binary Data.” Political Analysis 18(3):271–292.CrossRefGoogle Scholar
Cranmer, S. J., and Desmarais, B. A.. 2016. “A Critique of Dyadic Design.” International Studies Quarterly 60(2):355–362.CrossRefGoogle Scholar
Cranmer, S. J., Heinrich, T., and Desmarais, B. A.. 2014. “Reciprocity and the Structural Determinants of the International Sanctions Network.” Social Networks 36:5–22.CrossRefGoogle Scholar
De Boef, S., and Keele, L.. 2008. “Taking Time Seriously.” American Journal of Political Science 52(1):184–200.CrossRefGoogle Scholar
Erdman, C., and Emerson, J. W.. 2007. “bcp: An R Package for Performing a Bayesian Analysis of Change Point Problems.” Journal of Statistical Software 23(3):1–13.CrossRefGoogle Scholar
Ge, Z., Song, Z., and Gao, F.. 2013. “Review of Recent Research on Data-Based Process Monitoring.” Industrial & Engineering Chemistry Research 52(10):3543–3562.CrossRefGoogle Scholar
Gelman, A., and Hill, J.. 2007. Data Analysis Using Regression and Multilevel Hierarchical Models, vol. 1. New York: Cambridge University Press.Google Scholar
Gill, J. (2014). Bayesian Methods: A Social and Behavioral Sciences Approach, vol. 20. Boca Raton, FL: CRC Press.CrossRefGoogle Scholar
Golub, J. 2008. “Survival Analysis.” In The Oxford Handbook of Political Methodology. Oxford, UK: Oxford University Press.Google Scholar
Hastie, T., and Tibshirani, R.. 1993. “Varying-Coefficient Models.” Journal of the Royal Statistical Society. Series B (Methodological) 55:757–796.CrossRefGoogle Scholar
Haynes, K., Eckley, I. A., and Fearnhead, P.. 2017. “Computationally Efficient Changepoint Detection for a Range of Penalties.” Journal of Computational and Graphical Statistics 26(1):134–143.CrossRefGoogle Scholar
Huckfeldt, R. R., Kohfeld, C. W., and Likens, T. W.. 1982. Dynamic Modeling: An Introduction. New York: Sage.CrossRefGoogle Scholar
Jenke, L., and Gelpi, C.. 2016. “Theme and Variations: Historical Contingencies in the Causal Model of Interstate Conflict.” Journal of Conflict Resolution 63(7): 2262–2284.Google Scholar
Kent, D., Wilson, J. D., and Cranmer, S. J.. 2020a. “A Permutation-Based Changepoint Technique for Monitoring Effect Sizes.” Code Ocean. https://doi.org/10.24433/CO.5350515.v1.CrossRefGoogle Scholar
Kent, D., Wilson, J. D., and Cranmer, S. J.. 2020b. “Replication Data for: A Permutation-Based Changepoint Technique for Monitoring Effect Sizes.” https://doi.org/10.7910/DVN/NS91FD, Harvard Dataverse, V1, UNF:6:Yww4QlNWYChGp21XaM1P2g== [fileUNF].Google Scholar
Killick, R., and Eckley, I.. 2014. “Changepoint: An R Package for Changepoint Analysis.” Journal of Statistical Software 58(3):1–19.CrossRefGoogle Scholar
King, G. 1998. Unifying Political Methodology: The Likelihood Theory of Statistical Inference. Ann Arbor: University of Michigan Press.CrossRefGoogle Scholar
Leeds, B. A. 2003. “Do Alliances Deter Aggression? The Influence of Military Alliances on the Initiation of Militarized Interstate Disputes.” American Journal of Political Science 47(3):427–439.CrossRefGoogle Scholar
Mitchell, S. M., Gates, S., and Hegre, H.. 1999. “Evolution in Democracy-War Dynamics.” Journal of Conflict Resolution 43(6):771–792.CrossRefGoogle Scholar
Montgomery, D. C. 2013. Introduction to Statistical Quality Control. 7th ed. New York: John Wiley and Sons, Inc..Google Scholar
Montgomery, D. C., and Keats, J. B.. 1991. Statistical Process Control in Manufacturing. New York: Marcel Dekker.Google Scholar
Nieman, M. D. 2016. “Moments in Time: Temporal Patterns in the Effect of Democracy and Trade on Conflict.” Conflict Management and Peace Science 33(3):273–293.CrossRefGoogle Scholar
Park, J. H. 2012. “A Unified Method for Dynamic and Cross-Sectional Heterogeneity: Introducing Hidden Markov Panel Models.” American Journal of Political Science 56(4):1040–1054.CrossRefGoogle Scholar
Thurner, P. W., Schmid, C. S., Cranmer, S. J., and Kauermann, G.. 2018. “Network Interdependencies and the Evolution of the International Arms Trade.” Journal of Conflict Resolution 63(7): 1736–1764.Google Scholar
Wawro, G. J., and Katznelson, I.. 2014. “Designing Historical Social Scientific Inquiry: How Parameter Heterogeneity can Bridge the Methodological Divide Between Quantitative and Qualitative Approaches.” American Journal of Political Science 58(2):526–546.CrossRefGoogle Scholar
Wilson, J. D., Stevens, N. T., and Woodall, W. H.. 2019. “Modeling and Detecting Change in Temporal Networks via the Degree Corrected Stochastic Block Model.” Quality and Reliability Engineering International 35(5):1363–1378.CrossRefGoogle Scholar
Woodall, W. H., and Montgomery, D. C.. 1999. “Research Issues and Ideas in Statistical Process Control.” Journal of Quality Technology 31(4):376–386.CrossRefGoogle Scholar
Woodall, W. H., Zhao, M. J., Paynabar, K., Sparks, R., and Wilson, J. D.. 2017. “An Overview and Perspective on Social Network Monitoring.” IISE Transactions 49(3):354–365.CrossRefGoogle Scholar
Zorn, C. J. 2001. “Generalized Estimating Equation Models for Correlated Data: A Review with Applications.” American Journal of Political Science 45:470–490.CrossRefGoogle Scholar
Supplementary material: Link

Kent et al. Dataset

Dataset

https://doi.org/10.7910/DVN/NS91FD
Link