Hostname: page-component-788cddb947-w95db Total loading time: 0 Render date: 2024-10-14T12:59:37.812Z Has data issue: false hasContentIssue false
  • English
  • Français

Whose Variance Is It Anyway? Interpreting Empirical Models with State-Level Data

Published online by Cambridge University Press:  25 January 2021

Jeff Gill
Get access Rights & Permissions [Opens in a new window]

Abstract

Researchers commonly apply inferential statistical procedures to population data from the 50 U.S. states as if they were estimating population parameters from sample statistics. This method is incorrect because with population data there is no need to make inferences about quantities that are already known. Instead, authors should simply provide evidence that their specified model provides a good fit to the data. Summary measures of variance as well as the full engine of Bayesian statistics perform this function. This research note demonstrates why the current practice of making inferences from population data with the null hypothesis significance test is wrong, provides some specific examples of problems in the literature, and gives prescriptive advice about correctly assessing and conveying empirical model results.

Type
The Practical Researcher
Information
State Politics & Policy Quarterly , Volume 1 , Issue 3 , Fall 2001 , pp. 318 - 338
Copyright
Copyright © 2001 The American Political Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Akaike, H. 1969. “Fitting Autoregressive Models for Prediction.” Annals of the Institute of Statistical Mathematics 21:243–247.CrossRefGoogle Scholar
Bakan, David. 1960. “The Test of Significance in Psychological Research.” Psychological Bulletin 66:423–437.CrossRefGoogle Scholar
Barnett, Vic. 1973. Comparative Statistical Inference. New York: John Wiley & Sons.Google Scholar
Becker, Gary S., Grossman, Michael, and Murphy, Kevin M.. 1994. “An Empirical Analysis of Cigarette Addiction.” American Economic Review 84:396–418.Google Scholar
Berger, James O. 1985. Statistical Decision Theory and Bayesian Analysis. 2nd ed. New York: Springer-Verlag.CrossRefGoogle Scholar
Besley, Timothy, and Case, Anne. 1995. “Incumbent Behavior: Vote-Seeking, Tax-Setting, and Yardstick Competition.” American Economic Review 85:25–45.Google Scholar
Blalock, Hubert M. 1960. Social Statistics. New York: McGraw Hill.Google Scholar
Blalock, Hubert M. 1961. Causal Inferences in Nonexperimental Research. Chapel Hill, NC: University of North Carolina Press.Google Scholar
Bonjean, Charles M., and Lineberry, Robert L.. 1970. “The Urbanization-Party Competition Hypothesis: A Comparison of All United States Countries.” Journal of Politics 32:305–321.CrossRefGoogle Scholar
Box, George E. P. 1980. “Sampling and Bayes' Inference in Scientific Modeling and Robustness.” Journal of the Royal Statistical Society A 143:383–430.CrossRefGoogle Scholar
Box, G. E. P., and Jenkins, G. M.. 1976. Time Series Analysis: Forecasting and Control. San Francisco, CA: Holden-Day.Google Scholar
Box, George E. P., and Tiao, George C.. 1992. Bayesian Inference in Statistical Analysis. New York: John Wiley & Sons.CrossRefGoogle Scholar
Brace, Paul, and Jewett, Aubrey. 1995. “The State of State Politics Research.” Political Research Quarterly 48:643–681.CrossRefGoogle Scholar
Broemeling, Lyle, and Shaarawy, Samir. 1988. “Time Series: A Bayesian Analysis in the Time Domain.” In Bayesian Analysis of Time Series Models and Dynamic Models, ed. Spall, James C.. New York: Marcell Dekker.Google Scholar
Carmines, Edward. 1974. “The Mediating Influence of State Legislatures on the Linkage Between Interparty Competition and Welfare Policies.” American Political Science Review 68:1118–1124.CrossRefGoogle Scholar
Casella, George, and Berger, Roger L.. 1990. Statistical Inference. Belmont, CA: Wadsworth.Google Scholar
Cohen, Jacob. 1992. “A Power Primer.” Psychological Bulletin 112:115–159.CrossRefGoogle ScholarPubMed
Cohen, Jacob. 1994. “The Earth is Round (p<.05).” American Psychologist 12:997–1003.CrossRefGoogle Scholar
Davidson, R., and MacKinnon, J.. 1993. Estimation and Inference in Econometrics. New York: Oxford University Press.Google Scholar
Dawson, Richard E., and Robinson, James A.. 1963. “Inter-Party Competition, Economic Variables, and Welfare Policies in the American States.” Journal of Politics 25:265–289.CrossRefGoogle Scholar
Dennis, Brian. 1996. “Discussion: Should Ecologists Become Bayesians?” Ecological Applications 6:1095–1103.CrossRefGoogle Scholar
Dye, Thomas R. 1961. “A Comparison of Constituency Influences in the Upper and Lower Chambers of a State Legislature.” Western Political Quarterly 14:473–481.CrossRefGoogle Scholar
Dye, Thomas R. 1965. “Malapportionment and Public Policy in the States.” Journal of Politics 27:586–601.CrossRefGoogle Scholar
Dye, Thomas R. 1984. “Party and Policy in the States” Journal of Politics 46:1097–1116.CrossRefGoogle Scholar
Edwards, Don. 1996. “Comment: The First Data Analysis Should Be Journalistic.” Ecological Applications 6: 1090–1094.CrossRefGoogle Scholar
Edwards, George. 1990. Presidential Influence in Congress. San Francisco, CA: W.H. Freeman and Company.Google Scholar
Ellison, Aaron M. 1996. “An Introduction to Bayesian Inference for Ecological Research and Environmental Decision-Making.” Ecological Applications 64: 1036–1046.CrossRefGoogle Scholar
Fiorina, Morris P. 1991. “Divided Government in the States.” PS: Political Science & Politics 24:646–650.Google Scholar
Fisher, Ronald A. 1925a. Statistical Methods for Research Workers. Edinburgh, Scotland: Oliver and Boyd.Google Scholar
Fisher, Ronald A. 1925b. “Theory of Statistical Estimation.” Proceedings of the Cambridge Philosophical Society 22:700–725.CrossRefGoogle Scholar
Fisher, Ronald A. 1934. The Design of Experiments. Edinburgh, Scotland: Oliver and Boyd.Google Scholar
Fisher, Ronald A. 1955. “Statistical Methods and Scientific Induction.” Journal of the Royal Statistical Society B 17: 69–78.Google Scholar
Fomby, Thomas B., Carter Hill, R., and Johnson, Stanley R.. 1980. Advanced Econometric Methods. New York: Springer-Verlag.Google Scholar
Gary, Lawrence E. 1973. “Policy Decisions in the Aid to Families with Dependent Children Program: A Comparative State Analysis.” Journal of Politics 35: 886–923.CrossRefGoogle Scholar
Gelman, Andrew, Carlin, John B., Stern, Hal S., and Rubin, Donald B.. 1995. Bayesian Data Analysis. New York: Chapman & Hall.CrossRefGoogle Scholar
Gill, Jeff. 1999. “The Insignificance of Null Hypothesis Significance Testing.” Political Research Quarterly 52:647–674.CrossRefGoogle Scholar
Gill, Jeff, and Meier, Kenneth J.. 2000. “Public Administration Research and Practice: A Methodological Manifesto.” Journal of Public Administration Research and Theory 10:157–200.CrossRefGoogle Scholar
Godfrey, Leslie G. 1987. “Discriminating Between Autocorrelation and Misspecification in Regression Analysis: An Alternative Test Strategy.” Review of Economics and Statistics 69:128–134.CrossRefGoogle Scholar
Good, I. J. 1950. Probability and the Weighing of Evidence. New York: Hafner.Google Scholar
Gray, Virginia. 1974. “Expenditures and Innovation as Dimensions of ‘Progressivism’: A Note on the American States” American Journal of Political Science 18:693–699.CrossRefGoogle Scholar
Gray, Virginia. 1976. “Models of Comparative State Politics: A Comparison of Cross-Sectional Time Series Analyses.” American Journal of Political Science 20:235–256.CrossRefGoogle Scholar
Hedlund, Ronald D., and Paul Friesema, H.. 1972. “Representatives' Perception of Constituency Opinion.” Journal of Politics 34:730–752.CrossRefGoogle Scholar
Heer, David M. 1966. “Economic Development and Fertility.” Demography 3:423–444.CrossRefGoogle ScholarPubMed
Hero, Rodney, and Tolbert, Caroline J.. 1996. “A Racial/Ethnic Diversity Interpretation of Politics and Policy in the States of the U.S.” American Journal of Political Science 40:851–871.CrossRefGoogle Scholar
Herson, Lawrence J. R. 1957. “The Lost World of Municipal Government.” American Political Science Review 51:330–345.CrossRefGoogle Scholar
Hofferbert, Richard I. 1966. “Ecological Development and Policy Change.” Midwest Journal of Political Science 10:464–486.CrossRefGoogle Scholar
Hopkins, Anne H., and Weber, Ronald E.. 1976. “Dimensions of Public Policies in the American States.” Polity 8:475–489.CrossRefGoogle Scholar
Howson, Colin, and Urbach, Peter. 1993. Scientific Reasoning: The Bayesian Approach. 2nd ed. Chicago: Open Court.Google Scholar
Hunter, John E. 1997. “Needed: A Ban on the Significance Test.” Psychological Science January, Special Section 8:3–7.CrossRefGoogle Scholar
Jeffreys, Harold. 1961. The Theory of Probability. Oxford, England: Clarendon Press.Google Scholar
Jennings, Edward T. Jr. 1979. “Competition, Constituencies, and Welfare Policies in American States.” American Political Science Review 73:414–429.CrossRefGoogle Scholar
Jewell, Malcom E. 1982. “The Neglected World of State Politics.” Journal of Politics 44:638–657.CrossRefGoogle Scholar
Jones, Ruth S., and Miller, Warren E.. 1984. “State Polls: Promising Data Sources for Political Research.” Journal of Politics 46:1182–1192.CrossRefGoogle Scholar
Kass, R. E., and Raftery, A. E.. 1989. “Bayes Factors.” Journal of the American Statistical Association 90:773–795.CrossRefGoogle Scholar
King, Gary. 1989. Unifying Political Methodology: The Likelihood Theory of Statistical Inference. Cambridge, England: Cambridge University Press.Google Scholar
Knottnerus, Paul. 1985. “A Test Strategy for Discriminating between Autocorrelation and Misspecification in Regression Analysis: A Critical Note.” The Review of Economics and Statistics 67:175–177.CrossRefGoogle Scholar
Leamer, Edward E. 1978. Specification Searches: Ad Hoc Inference with Nonexperimental Data. New York: John Wiley & Sons.Google Scholar
Lee, Peter M. 1989. Bayesian Statistics: An Introduction. New York: John Wiley & Sons.Google Scholar
LeLoup, Lance T. 1978. “Reassessing the Mediating Impact of Legislative Capability.” American Political Science Review 72:616–621.CrossRefGoogle Scholar
Leonard, Thomas, and Hsu, John S.J.. 1999. Bayesian Methods. Cambridge, England: Cambridge University Press.Google Scholar
Lindsey, James K. 1997. Applying Generalized Linear Models. New York: Springer-Verlag.Google Scholar
Ludwig, D. 1996. “Uncertainty and the Assessment of Extinction Probabilities.” Ecological Applications 6:1067–1076.CrossRefGoogle Scholar
MacKinnon, James G. 1992. “Model Specification Tests and Artificial Regressions.” Journal of Economic Literature 30:102–146.Google Scholar
Markus, Gregory B. 1974. “Electoral Coalitions and Senate Roll Call Behavior: An Ecological Analysis.” American Journal of Political Science 18:595–607.CrossRefGoogle Scholar
Marquette, Jesse F., and Hinckley, Katherine A.. 1981. “Competition, Control and Spurious Covariation: A Longitudinal Analysis of State Spending.” American Journal of Political Science 25:362–375.CrossRefGoogle Scholar
Meehl, Paul E. 1978. “Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology.” Journal of Counseling and Clinical Psychology 46:806–834.CrossRefGoogle Scholar
Miller, Alan J. 1990. Subset Selection in Regression. New York: Chapman & Hall.CrossRefGoogle Scholar
Neyman, Jerzy, and Pearson, Egon S.. 1928a. “On the Use and Interpretation of Certain Test Criteria for Purposes of Statistical Inference. Part I.” Biometrika 20(A):175–240.Google Scholar
Neyman, Jerzy, and Pearson, Egon S.. 1928b. “On the Use and Interpretation of Certain Test Criteria for Purposes of Statistical Inference. Part II.” Biometrika 20(A):263–294.Google Scholar
Neyman, Jerzy, and Pearson, Egon S.. 1936. “Contributions to the Theory of Testing Statistical Hypotheses.” Statistical Research Memorandum 1:1–37.Google Scholar
Nice, David C. 1994. Policy Innovation in State Government. Ames, IA: Iowa State University Press.Google Scholar
Pollard, W. E. 1986. Bayesian Statistics for Evaluation Research. Newbury Park, CA: Sage.Google Scholar
Pollard, P., and Richardson, J. T. E.. 1987. “On the Probability of Making Type One Errors.” Psychological Bulletin 102:159–163.CrossRefGoogle Scholar
Poskitt, D. S. 1986. “A Bayes Procedure for the Identification of Univariate Time Series Models.” Annals of Statistics 14:502–516.CrossRefGoogle Scholar
Press, S. James. 1989. Bayesian Statistics: Principles, Models and Applications. New York: John Wiley & Sons.Google Scholar
Raftery, Adrian E. 1995. “Bayesian Model Selection in Social Research.” In Sociological Methodology, ed. Marsden, Peter V.. Cambridge, MA: Blackwells.Google Scholar
Reckhow, Kenneth H. 1990. “Bayesian Inference in Non-Replicated Ecological Studies.” Ecology 71(6): 2053–2059.CrossRefGoogle Scholar
Rivers, Douglas, and Rose, Nancy L. 1985. “Passing the President's Program: Public Opinion and Presidential Influence in Congress.” American Journal of Political Science 29:183–196.CrossRefGoogle Scholar
Ringquist, Evan J. 1993. “Does Regulation Matter?: Evaluating the Effects of State Air Pollution Control Programs.” Journal of Politics 55:1022–1045.Google Scholar
Robinson, W.S. 1950. “Ecological Correlations and the Behavior of Individuals.” American Sociological Review 15:351–357.CrossRefGoogle Scholar
Rozeboom, William W. 1960. “The Fallacy of the Null Hypothesis Significance Test.” Psychological Bulletin 57:416–428.CrossRefGoogle ScholarPubMed
Russell, Bertrand. 1929. Mysticism and Logic and Other Essays. New York: W.W. Norton & Company.Google Scholar
Schmidt, Frank L. 1996. “Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for the Training of Researchers.” Psychological Methods 1:115–129.CrossRefGoogle Scholar
Schmidt, Frank L., and Hunter, John E.. 1977. “Development of a General Solution to the Problem of Validity Generalization.” Journal of Applied Psychology 62:529–540.CrossRefGoogle Scholar
Serlin, Ronald C., and Lapsley, Daniel K.. 1993. “Rational Appraisal of Psychological Research and the Good-enough Principle.” In A Handbook for Data Analysis in the Behavioral Sciences: Methodological Issues, eds. Keren, G. and Lewis, C.. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
Sharkansky, Ira. 1968. Spending in the American States. Chicago: Rand McNally.Google Scholar
Sharkansky, Ira, and Hofferbert, Richard I.. 1969. “Dimensions of States Politics, Economics, and Public Policy.” American Political Science Review 63:867–879.CrossRefGoogle Scholar
Shibata, R. 1980. “Asymptotically Efficient Selection of the Order of the Model for Estimating Parameters of a Linear Process.” Annals of Statistics 8:147–164.CrossRefGoogle Scholar
Sims, Christopher. 1988. “Bayesian Skepticism on Unit Root Econometrics.” Journal of Economic Dynamics and Control 12:463–474.CrossRefGoogle Scholar
Solow, Andrew R. 1993. “Inferring Extinction from Sighting Data.” Ecology 74(3): 962–964.CrossRefGoogle Scholar
Taylor, B. L., Wade, P. R., Stehn, R. A., and Cochrane, J. F.. 1996. “A Bayesian Approach to Classification Criteria for Spectacled Eiders.” Ecological Applications 6:1077–1089.CrossRefGoogle Scholar
Thursby, Jerry G. 1981. “A Test Strategy For Discriminating Between Autocorrelation and Misspecification in Regression Analysis.” Review of Economics and Statistics 63:117–123.CrossRefGoogle Scholar
Tiao, George C., and Zellner, Arnold. 1964. “Bayes's Theorem and the Use of Prior Knowledge in Regression Analysis.” Biometrika 51:219–230.CrossRefGoogle Scholar
West, Mike, and Harrison, Jeff. 1989. Bayesian Forecasting and Dynamic Models. New York: Springer-Verlag.CrossRefGoogle Scholar
Wiggins, Charles W., Hamm, Keith E., and Bell, Charles G.. 1992. “Interest Group and Party Influence Agents in the Legislative Process: A Comparative State Analysis.” Journal of Politics 54:82–100.CrossRefGoogle Scholar
Winters, Richard. 1976. “Party Control and Policy Change.” American Journal of Political Science 20:597–636.CrossRefGoogle Scholar
Wolfson, L. J., Kadane, J. B., and Small, M. J.. 1996. “Bayesian Environmental Policy Decisions: Two Case Studies.” Ecological Applications 6:1056–1066.CrossRefGoogle Scholar