Hostname: page-component-8448b6f56d-gtxcr Total loading time: 0 Render date: 2024-04-23T11:37:41.258Z Has data issue: false hasContentIssue false
  • English
  • Français

Validity of the National Health Security Preparedness Index as a Predictor of Excess COVID-19 Mortality

Published online by Cambridge University Press:  05 January 2021

Mark E. Keim Open the ORCID record for Mark E. Keim [Opens in a new window]  and
Alex P. Lovallo
Mark E. Keim*
Affiliation:
DisasterDoc LLC, Atlanta, GeorgiaUSA Beth Israel Deaconess Medical Center Disaster Medicine Fellowship, Harvard University, Boston, MassachusettsUSA Rollins School of Public Health, Emory University, Atlanta, GeorgiaUSA
Alex P. Lovallo
Affiliation:
DisasterDoc LLC, Atlanta, GeorgiaUSA
*
Correspondence: Mark Keim, MD, MBA, 141 Chantilly Lane, Lawrenceville, Georgia30043USA, E-mail: mark@disasterdoc.org
Rights & Permissions [Opens in a new window]

Abstract

Objective:

This study compared 2019 values for the National Health Security Preparedness Index (NHSPI) with 2020 rates of coronavirus disease 2019 (COVID-19)-related mortality as reported by the 50 US states and Puerto Rico during the first six months of the US pandemic (March 1 - August 31, 2020).

Methods:

Data regarding provisional death counts and estimates of excess deaths for COVID-19 according to state and territory were downloaded from the Centers for Disease Control and Prevention (CDC) National Center for Health Statistics website. Reporting included the six-month-long period of March 1 - August 31, 2020. Excess mortality rates were calculated as the number of excess deaths per 100,000 persons in each state population using 2019 US Census Bureau data. Mean values for state and territorial NHSPI domain indices were compared to state and territorial rates of COVID-19-related excess mortality using multiple linear regression, including analysis of variance. Correlations between the 51 state and territorial NHSPI values and corresponding COVID-19 excess mortality rates were calculated using Pearson’s correlation coefficient.

Results:

These calculations revealed a high degree of variance (adjusted r square = 0.02 and 0.25) and poor correlation (P = .16 and .08) among values for the overall NHSPI as compared to low and high estimates of excess COVID-19 mortality rates for 50 US states and Puerto Rico.

There was also a high degree of variance (adjusted r square = 0.001 and 0.03) and poor correlation (P values ranging from .09 to .94) for values for the six individual domains of the NHSPI as compared to low and high estimates of excess COVID-19 mortality rates for 50 US states and Puerto Rico.

Conclusion:

The NHSPI does not appear to be a valid predictor of excess COVID-19 mortality rates for 50 US states and Puerto Rico during the first six months of the pandemic.

Keywords

coronavirushealth securitymortalitypublic health preparedness
Type
Original Research
Information
Prehospital and Disaster Medicine , Volume 36 , Issue 2 , April 2021 , pp. 141 - 144
Check for updates
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
© The Author(s), 2021. Published by Cambridge University Press on behalf of World Association for Disaster and Emergency Medicine

Introduction

Over the past 20 years, US local, state, and federal agencies have implemented a wide range of measures for public health emergency preparedness; “but these efforts have not resulted in a clear picture of the nation’s preparedness owing to ambiguous and uncertain preparedness goals, a lack of agreement about what the measures should aim at and how they should be interpreted, and a weak system of accountability for producing results.” Reference Lurie, Wasserman and Nelson1,Reference Nelson, Lurie, Wasserman and Zakowski2

Health security is a recent concept that encompasses “activities and measures across sovereign boundaries that mitigates public health incidents to ensure the health of populations.” 3 However, like preparedness, there remains limited consensus regarding the definition and scope of health security. Reference Kamradt-Scott4

In 2012, the US Centers for Disease Control and Prevention (CDC; Atlanta, Georgia USA) initiated development of the National Health Security Preparedness Index (NHSPI) for “measuring the nation’s progress in preparing for, responding to, and recovering from disasters and other large-scale emergencies that pose risks to health and well-being in the United States.” 5

As a measurement tool, the NHSPI is intended to “summarize levels of preparedness achieved within individual states and for the nation as a whole” and has been used for this purpose. Health impact is implied as the domains of the NHSPI are reportedly considered to have been “shown to be important in protecting people from the health consequences of disasters and other large-scale hazards and emergencies.” 5

Validity is central to determining the utility of any scale. Validity describes a test’s ability to produce results consistent with other measures of the same characteristic and it requires external criteria. Reference Karras6 Predictive validity is considered to be achieved when there is significant correlation between an experimental test (eg, the NHSPI) and a reference criterion standard (eg, excess coronavirus disease 2019 [COVID-19] mortality). This study evaluated the validity of the NHSPI for predicting excess COVID-19-related mortality.

Methods

This study compared 2019 values for the NHSPI with rates of COVID-19-related morbidity and mortality as reported by the 50 US states and Puerto Rico during the first six months of the pandemic (March 1 - August 31, 2020).

Data Collection

State data summaries for the 2019 NHSPI were downloaded from the 2019 Preparedness Index Data Preview website. 7 Table 1 lists the 2019 NHSPI indicators grouped into six domains and 19 sub-domains.

Table 1. Domains and Sub-Domains of the NHSPI

Abbreviation: NHSPI, National Health Security Preparedness Index.

Data regarding provisional death counts and excess deaths for COVID-19 (according to state and territory) were downloaded from the CDC National Center for Health Statistics website. 8 Reports were downloaded for the six-month-long period of March 1 - August 31, 2020. The CDC defined excess deaths as the difference between the observed numbers of deaths in specific time periods and expected numbers of deaths in the same time periods. Estimates of excess deaths were calculated by CDC using Farrington surveillance algorithms. 8 The lower and upper estimates of excess mortality were calculated as the difference between the observed count and one of two thresholds (either the average expected count or the upper bound of the 95% prediction interval) by jurisdiction. Excess mortality rates were calculated as the number of excess deaths per 100,000 persons in each state population using 2019 US Census Bureau (Suitland, Maryland USA) data. 9

Data Analysis

Mean values for state and territorial NHSPI domain indices were compared to state and territorial rates of COVID-19-related excess mortality using multiple linear regression, including analysis of variance. Correlations between the 51 (state and territorial) NHSPI values and corresponding COVID-19 excess mortality rates were calculated using Pearson’s correlation coefficient.

Results

In general, these calculations failed to demonstrate any significant association between values of the overall NHSPI (or its six domains) and excess COVID-19 mortality rates for 50 US states and Puerto Rico.

Table 2 reveals a high degree of variance and poor correlation of values for the overall NHSPI as compared to low and high estimates of excess COVID-19 mortality rates for 50 US states and Puerto Rico.

Table 2. Linear Regression Statistics for 2019 Values of the Overall NHSPI, as Compared to Low and High Estimates of Excess COVID-19 Mortality Rates for 50 US States and Puerto Rico

Abbreviation: NHSPI, National Health Security Preparedness Index.

Table 3 also indicates a high degree of variance and poor correlation of values for the six individual domains of the NHSPI as compared to low and high estimates of excess COVID-19 mortality rates for 50 US states and Puerto Rico.

Table 3. Linear Regression Statistics for NHSPI Domains, as Compared to Low and High Estimates of Excess COVID-19 Mortality Rates for 50 US States and Puerto Rico

Abbreviation: NHSPI, National Health Security Preparedness Index.

Discussion

A research-practice gap exists across all fields of public health, including disaster-related health science. Reference Brownson, Eyler, Harris, Moore and Tabak10,Reference Kirsch and Keim11 Public health has moved forward in recent years to bridge this gap. Evidence-based public health calls for knowledge of the determinants and consequences of disease, as well as the efficacy, effectiveness, and costs of interventions. Reference Lhachimi, Bala and Vanagas12,Reference Brownson, Fielding and Maylahn13 And yet, despite repeated urging of public health leadership, disaster epidemiology remains chiefly concerned with etiological, rather than evaluative, hypotheses. Reference Lurie, Manolio, Patterson, Collins and Frieden14,15

Scale development and validation are critical to much of the work in public health. However, scale development is not a straightforward endeavor. There are many steps to scale development. There is significant jargon within these techniques. The work can be costly, time consuming, and complex statistical analysis is often required. Despite the availability of a large amount of technical literature on scale theory and development, many incomplete scales remain. Reference Boateng, Neilands, Frongillo, Melgar-Quiñonez and Young16

Part of the challenge in establishing a valid scale for preparedness stems from the reliance upon expert opinion as compared to a set of specific empirical observations to reach the overarching conclusion.

Future developments are not always predicted correctly by Delphi consensus, especially those involving complex forecasts (eg, disasters) with multiple factors (eg, 140 separate measures). The Delphi method is therefore used most successfully in forecasting single scalar indicators.

Practical application of the NHSPI is unlikely to occur if it is not directly tied to indicators of health outcome (ie, mortality). This challenge is compounded by the polysemous nature of both of the key phrases in question, “preparedness” and “health security,” making predicted health outcomes difficult to define, let alone measure. There would appear to be limited practical utility for a health-related index that has no proven association with actual health outcomes.

Study Limitations

While this study does raise an important and reportable question regarding the criterion-related validity of the NHSPI, the investigation does not, however, prove the null hypothesis. Further study is necessary to further detail the association between the NHSPI and COVID-19-related mortality. Subsequent studies should also include a systematic analysis of content validity considering the relatively ambiguous and polysemous set of definitions associated with terms such as “health security” and “preparedness.”

Conclusion

The NHSPI does not appear to be a valid predictor of excess COVID-19 mortality rates for 50 US states and Puerto Rico during the first six months of the pandemic.

Conflicts of interest

none

References

Lurie, N, Wasserman, J, Nelson, C. Public health emergency preparedness: evolution or revolution? Health Affairs. 2006;25(4):935–945.CrossRefGoogle ScholarPubMed
Nelson, C, Lurie, N, Wasserman, J, Zakowski, S. Conceptualizing and defining public health emergency preparedness. Am J Public Health. 2007;97(Suppl 1):S9–S11.CrossRefGoogle ScholarPubMed
World Health Organization. Health Security. Health Topics Web site. https://www.who.int/health-topics/health-security/#tab=tab_1. Published 2020. Accessed October 16, 2020.Google Scholar
Kamradt-Scott, A. Securing Indo-Pacific health security: Australia’s approach to regional health security. Australian Journal of International Affairs. 2018;72(6):500–519.CrossRefGoogle Scholar
National Health Security Preparedness Index. Methodology for the 2020 Release of the National Health Security Preparedness Index. Methodology Web site. https://nhspi.org/wp-content/uploads/2020/06/NHSPI_2020_Methodology.pdf. Published 2020. Accessed October 16, 2020.Google Scholar
Karras, D. Statistical methodology: reliability and validity assessment in study design. Acad Emerg Med. 1997;4(4):144–149.CrossRefGoogle ScholarPubMed
National Health Security Preparedness Index. 2019 Preparedness Index Data Preview. https://nhspi.org/wp-content/uploads/2019/05/NHSPI_2019_Data_Download.xlsx. Published 2019. Accessed October 16, 2020.Google Scholar
Centers for Disease Control and Prevention. Excess Deaths Associated with COVID-19. COVID-19 Death Data and Resources Web site. https://www.cdc.gov/nchs/nvss/vsrr/covid19/excess_deaths.htm. Published 2020. Accessed October 16, 2020.Google Scholar
Bureau USC. 2019 US Census Bureau data. Explore Census Data Web site. https://data.census.gov/cedsci/. Published 2020. Accessed October 16, 2020.Google Scholar
Brownson, R, Eyler, A, Harris, J, Moore, J, Tabak, R. Getting the word out: new approaches for disseminating public health science. J Public Health Manag Pract. 2018;24(2):102–111.CrossRefGoogle ScholarPubMed
Kirsch, TD, Keim, M. US governmental spending for disaster-related research, 2011-2016: characterizing the state of science funding across 5 professional disciplines. Disaster Med Public Health Prep. 2019;13(5-6):912–919.CrossRefGoogle ScholarPubMed
Lhachimi, S, Bala, M, Vanagas, G. Evidence-based public health. Biomed Res Int. 2016;2016:569019.CrossRefGoogle ScholarPubMed
Brownson, R, Fielding, J, Maylahn, C. Evidence-Based public health: a fundamental concept for public health practice. Ann Rev Public Health. 2009;30:175–201.CrossRefGoogle ScholarPubMed
Lurie, N, Manolio, T, Patterson, A, Collins, F, Frieden, T. Research as a part of public health emergency response. N Eng J Med. 2013;368(13):1251–1255.CrossRefGoogle ScholarPubMed
Institute of Medicine. Enabling Rapid and Sustainable Public Health Research During Disasters: Summary of a Joint Workshop by the Institute of Medicine and the US Department of Health and Human Services. Washington, DC USA: National Academies Press; 2015.Google Scholar
Boateng, G, Neilands, T, Frongillo, E, Melgar-Quiñonez, H, Young, S. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health. 2018;6:149.CrossRefGoogle ScholarPubMed
Figure 0

Table 1. Domains and Sub-Domains of the NHSPI

Figure 1

Table 2. Linear Regression Statistics for 2019 Values of the Overall NHSPI, as Compared to Low and High Estimates of Excess COVID-19 Mortality Rates for 50 US States and Puerto Rico

Figure 2

Table 3. Linear Regression Statistics for NHSPI Domains, as Compared to Low and High Estimates of Excess COVID-19 Mortality Rates for 50 US States and Puerto Rico