Hostname: page-component-78c5997874-lj6df Total loading time: 0 Render date: 2024-11-17T14:57:45.111Z Has data issue: false hasContentIssue false
  • English
  • Français

Uncle Wuffle’s Reflections on Political Science Methodology

Published online by Cambridge University Press:  31 December 2014

A Wuffle
A Wuffle*
Affiliation:
University of California, Irvine
Rights & Permissions [Opens in a new window]

Abstract

This essay consists of idiosynratic reflections on research methodology based on a long career.

Type
The Teacher
Information
PS: Political Science & Politics , Volume 48 , Issue 1 , January 2015 , pp. 176 - 182
Copyright
Copyright © American Political Science Association 2015 

The set of numbered items below reflect, in no particular order, reflections about what to do and what not to do in research and comments on various current disputes in the discipline.

  1. 1. Albert Einstein once almost said, “Our models should be as simple as possible—but no simpler.”Footnote 1 That remains good advice in developing statistical models.Footnote 2 Unfortunately, the plethora of so-called control variables in political-science regressions suggests that most of us have no real idea of what matters, or why, or exactly how. It is rather like a football pile-up with many bodies: somewhere underneath the players is the ball, but we cannot be sure where until the rubble has been removed.Footnote 3

  2. 2. Data analysis has its own version of Gresham’s Law: namely, “Easily available data drive out the painful, time-consuming, and often very costly effort to collect your own data”—even though those data may be far more relevant to the problem(s) about which you (and others) care.Footnote 4

  3. 3. Good methodology and good research design, although essential to good science, are not a substitute for good theory and well-thought-out concepts.Footnote 5 Neither do they make you smart or prevent you from making stupid mistakes in making sense of the world, although they can help you to avoid certain kinds of errors. Any tool must be used correctly to be of any real use, and no tool is smarter than the person who uses it. For example, to get Gary King’s programs to present useful answers, you need questions worth answering, and if your ideas are all a-muddle, improving your graphics will not necessarily help much.Footnote 6 Similarly, if Richard Fenno “soaks and pokes,” he derives important insights into how Congress members think and behaveFootnote 7; if I were to soak and poke, all I likely would get is irritated interviewees and dirty bath water.

  4. 4. Theory-building and empirical work go hand in hand.Footnote 8

  5. 5. Every new methodological technique or tool promises more for those who use it than it actually delivers.Footnote 9 However, that is not to say that we have not seen great improvements in methodology in my lifetime.Footnote 10

  6. 6. It has been said, “Give a graduate student a hammer and he’ll discover that everything is a nail.”Footnote 11 Unfortunately, as Karl Marx once said, there is no single “royal road to truth.”Footnote 12 Those who jump on a methodological bandwagon very, very early might be lucky enough to have an article published in a top journal while the bloom has not yet left the rose and the methodology is still being seen as on the cutting edge. After that, the article probably has to make an actual substantive contribution. Relatedly, Gary King observed that there are tradeoffs between a focus on teaching “cutting-edge” methods and teaching all of the methods that might be useful to graduate students in their subsequent careers. The first has the limitation that by the time you leave graduate school, it is likely that the methods taught will be old hat; the second has the limitation that by the time you leave graduate school, it is likely that you will be quite old.Footnote 13 King’s solution to obsolescence is to teach the fundamentals—that is, the underlying theory of inference from which all models are developed.Footnote 14

  7. 7. We should applaud the recent push to use methodology that allows us to get a better handle on causalityFootnote 15—most important, methods that require us to directly examine change over time.Footnote 16 As I opined some years ago: “Trying to get at causality with cross-sectional methods is like trying to tell time with a stopped watch; you might get it right, but only by accident.”Footnote 17

  8. 8. Most political scientists are too lazy to do historical analysis or they lack the necessary training to do it well. One of the few advantages of being older is that you recognize that things were not always as they are now (e.g., African Americans were not always overwhelmingly Democrats; the parties were not always polarized around abortion). Thus, growing older is the ordinary scholar’s substitute for studying history.Footnote 18

  9. 9. The search for mechanism appears to be the present-day methodological substitute for the search for the Holy Grail.Footnote 19 Here, a warning made by Jon Elster some decades ago (1989, 3–12) seems worth repeating, although the rephrasing is my own: “For any mechanism one can suggest, it is likely that there is another (not necessarily equal) mechanism also at play in the same setting that would give rise to a quite different set of outcomes.” Although it is important to identify mechanisms that might be involved in any social process, deciding why some turn out to be more important than others in given settings is the real trick.Footnote 20

  10. 10. Most of the supposed distinctions between qualitative and quantitative research are wrong.

  11. 10A. I do not have a problem with a distinction between qualitative and quantitative work defined in terms of level of measurement of key variables—that is, calling research qualitative if it involves nominal or ordinal variables and quantitative if it involves interval or ratio variables.Footnote 21 However, it is important to recognize that such a divide is far from hard and fast. There is almost always an issue of choice in how to operationalize a variable. For example, do we use nominal categories for religious groupings or do we take into account “levels of religiosity”? If the latter, do we seek only an ordinal coding or do we look for quantitative measures? Even more important, however, it is a mistake to think that because you are initially doing work involving only nominal or ordinal variables, that standard statistical tools are therefore useless.Footnote 22 You just have to be sure to use the right statistical tools.Footnote 23

  12. 10B. If the qualitative-quantitative distinction is in terms of whether the hypotheses being tested are couched in quantitative as opposed to ordinal terms, most work that claims to be quantitative really is not. It is, at best, ordinal. Taagepera (Reference Taagepera2008) reminded us that almost never does continuing research in an area of political science examine whether the (regression) parameters estimated from earlier work are being reproduced; all that is asked is whether the signs on given variables are in the predicted direction.Footnote 24

  13. 10C. Another reason to be suspicious of the qualitative-quantitative distinction is suggested by Taagepera (Reference Taagepera2008), who was originally trained as a physicist. He observed that if you look at theory in physics, you will discover that it involves relatively few variables; draws on closely interlinked sets of theories involving those variables; that the “dimensionality” on both the right-hand and the left-hand sides of the equation is the same (i.e., E = IR); and that some of the parameters that it estimates (e.g., the speed of light and the force of gravity on Earth’s surface) are rather “fundamental.” From the perspective of a physicist, there is virtually nothing in political science that counts as theory, regardless of whether it calls itself quantitative or qualitative.Footnote 25

If the qualitative-quantitative distinction is in terms of whether the hypotheses being tested are couched in quantitative as opposed to ordinal terms, most work that claims to be quantitative really is not. It is, at best, ordinal.

  1. 10D. Sometimes the qualitative-quantitative divide is defined in terms of how many cases are in the dataset. However, that also is not a very useful a distinction—although it is interesting that some qualitatively oriented scholars use this essentially quantitative distinction (i.e., “how many”) to decide where a particular piece of research falls.Footnote 26 The more cases you have—assuming that they comprise a random sample from some posited set—the easier it is to determine when the relationship(s) you find can be ruled out as the effects of chance. However, there is no magic cutoff; ceteris paribus, the clearer the pattern, the fewer the cases needed to figure out that what you see is probably not due to chance or to reject a hypothesis as false. Sometimes, an N of 1 will work if the hypothesis is deterministic rather than probabilistic in form. If you posit that A always implies B and you find A without finding B, then the hypothesis is contradicted by the evidence.Footnote 27

  2. 10E. Sometimes the qualitative-quantitative divide is defined in terms of concepts that are supposedly “inherently” qualitative because they involve the attribution of meaning to social constructs. In my view, there are no inherently qualitative or quantitative concepts; there are only issues of level of measurement and of which tools we use to generate data. Consider, for example, social identity, considered to involve perceptions of self (in particular, regarding oneself as a member of a particular group and identifying with the collective interests of the group). Are political scientists who study party identification, which is defined by Campbell et al. (1960) as a form of social identity, socially transmitted across generations, and—like other social identities, such as religion—relatively fixed, doing qualitative work? If not, why not? Does the answer change if we observe that many studies of voting behavior use open-ended survey questions that then are recoded? Does the answer change if we observe that partisan identification is normally coded as a seven-point ordinal scale? What about quantitatively oriented research that addresses the question of whether party identification has the same “meaning” in different countries (see, e.g., various essays in Bartle and Bellucci Reference Bartle and Bellucci2009)? Is it qualitative because it is concerned with meaning? Or, consider norms, another area often singled out as uniquely qualitative. Is the work of Bicchieri (Reference Bicchieri2006) and Axelrod’s classic essay (1986) on “meta-norms” somehow not about norms because they use formal models to study the properties of norms or the stability of norms? Is the game-theory-inspired work of Chwe (Reference Chwe2001) on “rituals” somehow not kosher because it does not rely on thick description?

  3. 10F. Sometimes the qualitative-quantitative distinction is drawn in terms of a whole slew of expectations about what kinds of models are needed to make sense of the world, of which perhaps the most important is that qualitative scholars recognize that different mechanisms may be at play in different settings.Footnote 28 Bennett and Elman (2006, 455), for example, asserted that “qualitative methodologists tend to believe that the social world is complex, characterized by path dependence, tipping points, interaction effects, strategic interaction, two-directional causality or feedback loops, and equifinality (many different paths to the same outcome) or multifinality (many different outcomes from the same value of an independent variable, depending on context).”Footnote 29 I think it is fair to interpret the implication of these remarks as being that quantitatively oriented political scientists do not share these views about social reality. My response: “Any researcher with any sense agrees with you. Now let’s see how well some particular piece of research lives up to these standards of sophistication.” Any scholar with any sense who is concerned about generality tries to formulate explanatory mechanisms and models in a way that will maximize the scope of their applicability, recognizing that the same mechanisms can produce different outcomes in different contexts.Footnote 30 Good quantitative work is sensitive to possible interaction effects.Footnote 31 Good quantitative work is sensitive to possible two-way causality.Footnote 32 Good formal modeling is sensitive to the possibility of tipping-point effects.Footnote 33 Good quantitative work is sensitive to case-selection issues.Footnote 34 Similarly, path dependence is inherent in most game-theory models (in extensive form), where what branch of the game tree has already been chosen conditions what outcomes are feasible (see, e.g., Brams Reference Brams1975). Etc.Footnote 35

  4. 10G. Sometimes the qualitative-quantitative distinction is taken to be more epistemological, with qualitative researchers supposedly uninterested in (or skeptical about the feasibility of) empirical generalizations that apply widely or, perhaps, even uninterested in explanation or causality per se but instead concerned with developing detailed knowledge and insight about particular cases.Footnote 36 If that is the distinction, I do not have any problem with recognizing it as one that can meaningfully be drawn, but I am highly suspicious that there are that many researchers who eschew all claims to generality or explanatory power.Footnote 37 Also, if a leading scholar of Absurdistan said that he had identified mechanisms that perfectly explained that country’s behavior—but, of course, they applied nowhere else except AbsurdistanFootnote 38—it is unlikely that he would be nominated for the Skytte Prize.Footnote 39

  5. 10H. Only Americanists have been allowed to regularly get away with treating their case as sui generis, although the claim for “my country” exceptionalism seems to be pervasive.Footnote 40 Personally, I adhere to David Easton’s dictum from almost a half-century ago that “All political science is/needs to be comparative politics” (Easton, personal communication, 1968). I find compelling a line from Rudyard Kipling that is frequently quoted by my colleague Russ Dalton: “And what should they know of England who only England know?”Footnote 41 However, on the flip side, there is one commonsense piece of methodological advice for every graduate student who wants to be a comparativist, one that works regardless of their epistemological proclivities, namely: become a real expert on at least one case.Footnote 42 That way, you can test the generalities from larger-N studies or from experiments or from formal models or from other case studies to determine whether they really make sense. However, I also emphasize a point that I made in earlier work (quoted in Grofman Reference Grofman, Grofman, Lee, Winckler and Woodall1999) regarding how we should think about “comparative” politics in terms of the “TNT principle” that I see as defining comparative politics, namely, “Comparison across Time, Nations, or Types of institutions, persons, or processes.” In that framework, within-nation studies also can be comparative. Consider, for example, Posner’s (Reference Posner2007) work on when (a limited number of) linguistic cleavages or (a larger number of) tribal cleavages will form the basis of party competition in deeply divided societies. He showed that in Zambia, a shift from multiparty competition to single-party rule back to multiparty competition shifted incentives for campaigning strategies from linguistic to tribal and then back to linguistic (see also Posner Reference Posner2005).

  6. 11. Mixed methods that combine large-N studies with case methods are a current methodological panacea and, subject to the previous caveats, one to which I am quite sympathetic. Bennett and Elman (2006, 458) are clearly correct that “even when there are enough observations to allow statistical analysis, conducting in-depth case studies can still offer separate inferential advantages.” My main note of caution is simply that whereas combining knowledge and insight derived from case studies and/or experiments with larger-N analyses—to rule out spurious relationships and get at mechanisms and causality—is (and always was) the best way to go, doing so is not at all easy.Footnote 43 Moreover, mixed-methods training is sometimes merely a way to learn to do two or more methods badly rather than one very well, perhaps taught by someone who is more interested in convincing students of the existence of flaws in a disfavored method (or set of methods) than in teaching them how to use it as well as might be possible.

  7. 12. A different way to sidestep qualitative-quantitative debates was offered by Grofman (Reference Grofman2001). He suggested that we think of political science as involving three different types of puzzle solving—who-dunnits, how-dunnits, and why-dunnits—with a focus on particular situations to be analyzed and competing hypotheses or models. The first calls attention to competing notions about explanatory factors; the second, to a search for how particular factors achieve their effects; and the third, to explanations that are rooted in beliefs and values. However, these questions cut across the more usual qualitative-quantitative distinctions discussed previously.Footnote 44 In this vision, let whoever has the best answer to a particular empirical or theoretical puzzle catch the gold ring.Footnote 45

  8. 13. A far more useful distinction than the qualitative-quantitative one is between good work and not-so-good work, a distinction that—as far as I can assess—is very close to orthogonal to the qualitative-quantitative divide. The first rule of research is simple: “98.6%Footnote 46 of everything is crap.”Footnote 47 However, no study—no matter how good—is perfect. The second rule of research is: “No single study can address, much less answer, all questions.” The third rule of research is: “No single study can answer even one question definitively.”

  9. 14. The fourth rule of research is equally simple: “It is far easier to criticize than it is to do better.”Footnote 48

  10. 15. The search for a single master cause or master mechanism, of course, is silly. However, the number of trees sacrificed to debates about whether nations act the way they do because actors within them are “really” pursuing national interest (however defined), as opposed to “really” responding to an evolving international-norms regime, as opposed to “really” engaging in cooperative exchange behavior, perhaps under the shadow of a hegemon, is inherently very amusing to those of us who are not international-relations theorists and therefore not to be totally discouraged. Luckily, however, most good international-relations scholars do not waste their time in such debates but instead focus their brain power on the very difficult but quite intriguing set of questions about which factors will matter more in which contexts.Footnote 49

Thus, in honor of the Founding Fathers, I suggest this motto: “No reputation without replication.”

  1. 16. Correlation is not causation but it is a hell of a lot easier to report correlations than to plausibly demonstrate causation; therefore, correlations in political science are not going away any time soon.

  2. 17. The January 2014 issue of PS: Political Science and Politics included a debate about whether journals should require, after acceptance for publication, that datasets and codebooks and perhaps also a “diary” with sufficient detail be filed with articles to allow determination of exactly how reported analyses were done. Although the existence of “easy”-to-access datasets for reanalysis will divert certain students from doing some hard thinking on their own, and it will impose numerous costs on those who must prepare the materials from their publication for archiving and some nontrivial costs on the journals that store the materials, I think the benefits outweigh the costs. I view archived data as a public good.Footnote 50 I also am sympathetic to the idea that empirically oriented graduate students who claim statistical skills should not graduate until they show the ability to replicate (and critique) one major published study.Footnote 51 The other benefit is to the honesty of the profession. The only way to really know for sure whether to believe a published quantitative analysis is to rerun the data yourself or have a disinterested but inherently skeptical party do it.Footnote 52 There are too many ways that data can be “massaged” and methods tweaked.Footnote 53 Thus, in honor of the Founding Fathers, I suggest this motto: “No reputation without replication.”Footnote 54

  3. 18. To publish empirical work, it is probably sufficient to be NICE—that is, to have all four of the following conditions, or at least most of themFootnote 55:

    New data and/or New findings and/or New theory and/or New methodology

    Important and/or Interesting question(s)

    Clearly written discussion

    Evidence that is credible

    However, whereas being NICEFootnote 56 almost certainly will guarantee publication somewhere, being NICE does not guarantee being published in a top journal. Accomplishing that is much more of a crapshoot.Footnote 57

  4. 19. Reality is like an unbelievably enormous jigsaw puzzle: if, as a social scientist, you fill in one itsy-bitsy piece before you die, then you will have done far more than most.Footnote 58

ACKNOWLEDGMENTS

Wuffle is indebted to Diana Kapizewski and Gary King for helpful suggestions on an earlier draft of this article and to two reviewers for useful comments, some of which are incorporated in the present version. However, perceived errors remaining in the article may best be attributed solely to the reader.

Footnotes

1. This improvement on Einstein’s original formulation is attributed to Rein Taagepera (Reference Taagepera2008).

2. The KISS principle (i.e., “Keep it simple, Stupid”) is good advice, especially if you are smart.

3. Note that I am not suggesting that it is never appropriate to use control variables, only that you need a clear justification for those that you choose, or which interaction effects you take into account, and a very good reason to believe that they are not obscuring the causal paths in which you are most interested. As one of the reviewers of this article stated in a comment that I wholeheartedly endorse: “What we want is theoretically informed models.”

4. If restaurant prices are very low, ceteris paribus, more people eat at restaurants–even though, in principle, they could fix more nutritious meals at home. Of course, even with all of the goodwill in the world, the data we ourselves collect may suffer from all kinds of problems: from poor questionnaire design, to low completion rates and biased samples, to strong interviewer or participant–observer effects, and so on. On the other hand, we tend to be even blinder to the problems in the datasets collected by others, especially if they have already been used in the published literature. Consider, for example, the widely used Economic Freedom of the World Index created by the Heritage Foundation (available at www.heritage.org/index/ranking). By some of its indicators, countries such as Denmark are unfree because they do evil things such as having various forms of worker protection and strong trade unions.

5. It is all well and good to avoid preconceptions and to learn from your data as you proceed, but it is almost impossible to do good research without having at least some reasonably clear conceptions at the beginning.

6. Gary King’s computer programs for analyzing and displaying data are mostly marvelous, but they work best in helping to answer interesting questions or presenting results clearly and cogently when used by someone (nearly) as smart as Gary King. I leave it to readers to decide if they are in that category.

7. See Fenno Reference Fenno1978.

8. Conversely, no matter how good the theory, if you cannot actually measure its key concepts at some level (i.e., nominally, ordinally, or in terms of an interval or ratio scale), you are never going to get beyond armchair theorizing. Relatedly, even the most highly sophisticated theory needs to be checked against the real world. For example, whereas mathematical “theorems” are (presumably) true, those intended to be relevant to the social sciences (e.g., Downsian ideological convergence in two-party plurality competition); rest on many assumptions (see Grofman Reference Grofman2004). Only when you check whether the supposed implications of the theorem actually hold in your data can you begin to evaluate how critical some of those assumptions are for the applicability of the theorem to the world.

9. Methodological innovations come with some regularity and, in their initial incarnations, either disappear or are relegated to a relatively minor role, or—perhaps most often—are replaced with new tools to do almost the same job (although usually better) but with more bells and whistles. For example, path analysis is replaced with structural-equation models that then turn into nonparametric methods of various types to get at the idea (or the various ideas) of causality (Gary King, personal communication, February 2, 2014).

10. To provide examples from my own personal experience: We know far more than we did when I was a graduate student about how to deal with dichotomous or bounded variables. There have been major innovations in areas such as scaling theory and ecological inference. New computer-assisted tools for research are many, with major substantive contributions in areas such as content analysis and computer-mediated survey experiments. Maximum-likelihood methods were in their infancy then, as were Bayesian statistics, and tools for combining information at the individual and aggregate level (e.g., MLE were not yet being taught in political science departments. Of course, when I was a graduate student, we brought our Hollerith punched cards to a computer the size of a large whale in the early evening and waited until the next morning for a result from the SPSS Version 0 program, hoping against hope that we had not made a coding error and would not have to start all over again the next night.

11. However, exactly who said it is not clear to me. I thought it was Mo Fiorina, but he said “no” (personal communication, February 12, 2014). I note, however, that even the most serious of methodologists seem to have a lighter side. For example, Gary King recently observed (via Facebook) that “The best answer to most methods questions seems to involve changing the question.” In the dialogue that followed this posting, King later opined: “Sometimes it also helps to change the questioner” (January 27, 2014).

12. The full quotation is: “There is no royal road to science, and only those who do not dread the fatiguing climb of its steep paths have a chance of gaining its luminous summits.” (Yes, Virginia, Marx really said that.)

13. Relatedly, Mark Twain once observed: “Life is too short to learn German.” Lord Keynes pointed out that “in the long run, we are all dead”—which is the only theoretical claim by a macroeconomist that has ever won universal acceptance in economics.

14. “Good methods training helps students understand how to keep up with this scholarly literature, and how to pick up new methods, and…interpret the results” (Gary King, personal communication, February 2, 2014).

15. Currently, there is renewed interest in (and new computer tools for) experiments, and I am sympathetic to the claim that series of well-designed experiments can be helpful in pinning down the specifics of causal linkages in a way that no single experiment can; however, experimentation, too, is no universal “solvent.” Relatedly, matched-case analysis, when done well (see, especially, the discussion in Persson and Tabellini Reference Persson and Tabellini2003, chap. 5), can be highly informative, but what we mean by a “matched case” is related to the theory that we are trying to test. For example, when is the US House a matched case for the US Senate?

16. Pooled cross-sectional data are, in most of the uses that I have seen, cross-sectional data still. To the extent that process tracing involves careful analysis of sequences of events, it may indeed help us to define causality and mechanism. (As one reviewer noted, “You can treat time as a nuisance, or you can model the dynamics.”) However, in some dissertation prospectuses, a plan to engage in process tracing seems shorthand for “I want an excuse to spend some time in Country X, but I do not have the faintest idea exactly what I am going to do when I get there other than to perhaps find out whether A happened before or after B did.” At a conference I recently attended, a graduate student whose dissertation involved historical archival analysis (name withheld to protect her/his job prospects) characterized process tracing as “a respectable way of saying ‘I’m going to tell you a story.’” Of course, some stories are both less boring and more compelling than others.

17. For a more optimistic view of what can be done with cross-sectional data, see Persson and Tabellini (2003, chap. 5).

18. Scholars who lack the natural advantage of age must instead avail themselves of the remarkable and still growing availability of various kinds of time-series data spanning many decades—or they might simply emulate Rip Van Winkle and take a very, very long nap.

19. What is meant by “mechanism” varies among scholars. The key intuition seems to be that we need to go beyond correlations to identify some mental or physical process (or set of processes) that could lead to a change in one variable affecting a change in another variable in a way that seemed plausible (or, better yet, that could be directly tested). Although I appreciate the argument that unless we can specify a mechanism, we cannot be certain of which causal factors are at play, these days, mechanisms are sought after with much the same zeal that Norman Mailer idealized the “apocalyptic orgasm” in his writings of the 1960s and 1970s.

20. For example, Downs (Reference Downs1957) identified a powerful mechanism that, under specified assumptions, can foster candidate convergence in two-party plurality based on competition in a single-seat constituency. However, in the real world, candidates of opposite parties within the same constituency do not (always or even often) converge to the same policy views. However, that fact does not deny the truth that centripetal forces are at play; there simply are strong centrifugal as well as strong centripetal mechanisms that affect party competition (e.g., party primaries, the role of activists and interest groups, and the need to run party candidates in multiple constituencies that may differ in the location of their median voter). (See, for example, the long inventory of centrifugal forces in political competition in Grofman Reference Grofman2004.) Relatedly, Taagepera et al. (forthcoming) observe that the existence of forces pushing in opposite directions may lead to curvilinear relationships.

21. In my view, one of the reasons that King, Keohane, and Verba (Reference King, Keohane and Verba1994) received a hostile rejection from scholars who were more qualitatively oriented—despite their stated goal of bridging the qualitative-quantitative divide by explicating issues of research design and methods of analysis that transcended that distinction—was that the bulk of their examples used quantitative variables (or dichotomies treated as quantitative variables). It is unfortunate for present-day methods training that King et al. provided relatively few illustrations of research methods and research dos and don’ts drawn from research based on hypotheses involving nominal or ordinal variables—and with those applications rather buried in the text so as to make them difficult to find (see, e.g., 1994, 124–8). Another feature of research that is too little emphasized in King et al. is the creative process of hypothesis/theory formation. However, having made these criticisms, I hasten to add that the King et al. book needs to be read and its lessons taken to heart by any political scientist who is serious about doing empirical research.

22. Consider the fieldwork involved in personally interviewing people and asking open-ended questions with follow-up questions based on the initial responses. Once answers are coded, tabulated, and reported as percentages, a percentage is a number that can be treated as a quantitative variable, even if case coding is based on a dichotomy.

23. If you have nominal-level data, relationships can be examined with tools such as chi-square. Indeed, even if you are looking at relationships among dichotomous variables—specified in terms of sufficiency and/or necessity rather than those specified in terms of “more or less”—it does not mean that you are doing something to which statistical tools do not apply. If you do have more than one case and you are interested in how much support a hypothesis involving two dichotomous variables specified in terms of sufficiency and/or necessity has, asymmetric measures of association (e.g., Lambda) are intended to address that issue. Furthermore, there is a plethora of statistical tools intended for use with ordinal variables, including some rather fancy ones. Also, if you are interested in approaches to necessity and sufficiency that permit multivariable analyses, qualitative classification analysis (QCA) is another option. (For a nontechnical introduction to crisp-set QCA with a brief comparison to other methods, such as a regression with interaction effects, see Grofman and Schneider Reference Grofman and Schneider2009. Unfortunately, QCA at present lacks clear statistical underpinnings.)

24. Most of the uses of regression analysis in political science (including my own) can be described best as a way to estimate some variable at some particular point in time by treating it as a linear combination of some other huge list of other variables. In any case, maximizing the explained variance in a regression model is not a good test of theoretical success (see, especially, Taagepera Reference Taagepera2008). Furthermore, as numerous scholars have reminded us (without much success), the statistical significance of a variable should not be confused with the magnitude of its substantive impact. Although my observation of some years ago (Wuffle Reference Wuffle1989) that “political science is the only discipline in which regression is progress” is clearly hyperbolic, there is enough truth in it that I am willing to repeat it here.

25. Also, the key variables in theories in physics are usually sufficiently powerful in their effects to be visible even without control variables “complicating” the analysis. (Although I think that James March, the first Dean of the School of Social Sciences at the University of California, Irvine, exaggerated when he said “God gave all the easy problems to the physicists” (quoted in Wuffle Reference Wuffle1986), like many hyperbolic statements, there is a kernel of truth in it to make it worthy of reflection.)

26. Of course, in fairness, they might actually be using the ordinal distinction made famous in anthropology: “one, two, many.”

27. Here, I refer readers to Eckstein’s (Reference Eckstein, Greenstein and Polsby1975) classic essay on the theoretical case study.

28. Sometimes this aspect of the qualitative-quantitative divide is defined in terms of whether the analysis involves the study of “causes-of-effects” (qualitative) as opposed to “effects-of-causes” (quantitative). I find this to be a very elegant distinction in terms of both its near palindromic structure and its genuine insights, but I am skeptical that it really distinguishes qualitative from nonqualitative work. Yes, the same variable may not always have the same causal effect, depending on its interaction with other variables, and having more cases simply for the sake of having more cases is stupid, if there is good reason to think that the proposed mechanisms may not apply to those cases. (The best discussion of case selection that I know of is Geddes Reference Geddes2003, chap. 3. Indeed, it is the best book on the comparative method—period.)

29. The trick is to figure out a way to frame the mechanism such that it allows for variability in its implications for particular case circumstances. For example, we know that in some countries, the rich vote at much higher rates than the poor, whereas in other countries, there is almost no difference in turnout rates by income. We could say that the mechanisms that link voting turnout and class are different in the different countries. However, it also might be the case that there is a similar mechanism at play in each country but with outcomes varying according to contextual factor(s). (For an effort to explain cross-national variation not in turnout per se but rather in the relationship that we expect to find between income and turnout, see Grofman Reference Grofman2010, which links this variation to different levels of turnout in different countries.) Posner’s Reference Posner2004 study, which examines why Chewas and Tumbukas are allies in Zambia and adversaries in Malawi, is a good example of a very general theory that is, nonetheless, context specific.

30. For example, I would not expect the same consequences of electoral-system choice in democracies and dictatorships because elections do not have the same meaning in the two settings. However, electoral-law manipulation can be studied in both democracies and autocracies (compare, for example, Owen and Grofman, on the one hand, and Tan and Grofman Reference Tan and Grofman2014 on the other.)

31. In this context, it is useful to remind readers that one of the most basic of statistical techniques, ANOVA, is all about the search for interaction effects.

32. See, for example, any standard econometrics textbook and the discussion of two- and three-stage regressions. Of course, the fact that economists see multistage regression as a tool for developing better causal models does not mean that, in practice, they are correct. In general, I have never met an instrumental variables analysis that I found convincing.

33. See, for example, the work by the economist Thomas Schelling (Reference Schelling1978) and the mathematical sociologist Mark Granovetter (Reference Granovetter1978) or virtually any work in evolutionary game theory, such as Skyrms (Reference Skyrms1996).

34. See footnote 28.

35. As Gerring (2006, 676–9) stated: “Inquiry of a scientific nature… aims to be cumulative, evidence-based (empirical), falsifiable, generalizing, nonsubjective, replicable, rigorous, skeptical, systematic, transparent, and grounded in rational argument.” He continued: “There are differences of opinion over whether, or to what extent, science lives up to these high ideals. Even so, these are the ideals to which natural and social scientists generally aspire, and they help to define the enterprise in a general way and to demarcate it from other realms.” However, that the world is complex does not mean that we are always required to deal with all of its complexities at once. (See the rules of research in a subsequent discussion.)

36. In its extreme form, this skepticism asserts that the phenomena the understanding of which we care most about are unique events and thus not amenable to being fitted within broader explanatory schema. My response is to agree that things the understanding of which we care most about often are unique. An example is my wife, about whom I care deeply and whom I certainly do regard as unique. I have never ceased to try to understand her and I continually fail. However, I do not take her uniqueness and her unique personal history to prevent doctors from studying the (mal)function of her colon or social scientists from studying gender discrimination, using her attitudes and life experiences as a case.

37. Furthermore, if those insights really apply only to that particular case, why should anyone not interested in that particular case care?

38. I also note that because a major complaint against rational-choice models is that the only place that they apply is in Absurdistan, what is sauce for the goose also should be sauce for the gander.

39. The Johan Skytte Prize in Political Science was established in 1995 by the Johan Skytte Foundation at Uppsala University. It is arguably the closest thing political science has to a Nobel Prize.

40. In my view, every empirically oriented graduate student should be a comparativist (see following definition) and that especially includes Americanists. (Taylor et al. Reference Taylor, Shugart, Lijphart and Grofman2014, which builds on Lijphart Reference Lijphart1999 and a course jointly taught some years ago by Arend Lijphart and Bernard Grofman, is an interesting example of work that treats the United States in comparative cross-national perspective.)

41. For a set of justifications for thinking comparatively that combines theoretical arguments with a concern for professional success, see Lees (Reference Lees2006).

42. Note: The case you choose as the one for you to become a world-class expert on should not be that large. I suggest something on the order of magnitude of Ashtabula, Ohio. (Learning about, say, China is best thought of as several lifetimes of work. Thus, if you choose to study China, it helps to believe in reincarnation.)

43. On this point, see, for example, Verba Reference Verba1967.

44. For example, values (or beliefs) can be studied in many ways, from participant observation to survey research to studying behavior in experimental games such as the dictator game.

45. For social scientists (as opposed to philosophers), Grofman (2001, 1) asserted, “The proof is in the pudding, not in abstract debates about methodology or epistemology. A key task of empirically oriented social scientists is to find interesting features of the world and try to tell us something insightful that will help us explain/understand them better. If—whatever your methodology—you can do that, more power to you. If you can’t, find another line of work.… A puzzle-solving approach forces…real debates about real issues, in particular, the power of competing explanations to explain real-world puzzles.”

46. Beware specious precision. No matter how many decimal places are reported in your computer output, do not report more significant digits than make sense for your data.

47. It is critical early on in a scholar’s career to learn to tell the difference between good work and bad work. Unfortunately, contra Tolstoy, it is not true that all good work is alike and all bad work is bad in its own way. Read really good work (aided by older scholars who can help you understand why it is good). Read lots of bad work, too, and learn to understand exactly how it is flawed. Exposure to all kinds and qualities of research (including work not in your own area) is important in developing your own methodological skills; however, even more important, arguably, is actually doing research.

48. The fifth rule of research is: “Don’t bite off more than you can chew.” The sixth rule of research is: “Choose your methods so that they are appropriate to the question you are dealing with; do not let your methodological predilections/limited methodological skills determine the questions you choose to address.” The seventh rule of research is: “It’s never too late to learn.” Indeed, my own view is that even if your heart and lungs are still in perfect working order and your lips are still moving, the day you stopped learning is the day you died. The eighth rule of research is: “The more different ways/methods you use to reach the same conclusion, the more likely it is that it is the right conclusion.” Of course, “likely” is not the same as “certain.” (See the second and third rules of research.)

49. In like manner, I find almost nothing useful in the abstract debates over the merits and demerits of rational-choice theory, such as the one about “how many rational-choice theorists can dance on the edge of a protractor?” (My own perspectives, as a “reasonable-choice” modeler, are found in Wuffle Reference Wuffle1999. There, I point out a number of commonsense truths, such as that people usually, although not always, do things for reasons—although rarely for only a single reason and sometimes for reasons that they do not fully understand.)

50. Having archived one humongous dataset with the Interuniversity Consortium for Political and Social Research many years ago, I can tell you that providing all of the needed information is not easy. Also, it is unrealistic to expect journals to maintain proper archives to ensure format compatibility for anything more than a relatively short period. Indeed, having lived much of my academic life in the century known as B.E.K (i.e., before the existence of Gary King), I have numerous datasets that I can no longer read. (Unfortunately, even Gary King is probably not immortal, even if his immense productivity suggests that he may have already secretly mastered the art of cloning—and it is far from clear that his data-archiving website will outlive him.)

51. When I teach graduate-level statistics, I invariably assign published articles or research reports for replication, and the same is true for others who regularly teach methodology—most famously Gary King. Having the data easily accessible would make such assignments so much easier. However, I agree fully with the comment of one reviewer that although doing one replication is good training, doing more than one means graduate students may well lack the time to develop their own ideas.

52. Indeed, just as war is too important to leave to the privates, replication is too important to cumulative knowledge building to leave entirely to graduate students; faculty must be involved as well. Making it easier to access the data to replicate major studies will make that involvement more likely.

53. Of course, there are just as many reasons (if slightly different ones) for us to be suspicious of the findings of field research. Anthropologists are still arguing about how accurate Margaret Mead’s descriptions of Samoan sexual practices were, based on what her interviewees told her. More generally, the debate continues about how a fieldworker’s own views on topics such as “nature versus nurture” and “cultural relativism” color the interpretation of participant-observation studies.

54. Those with a more religious slant may wish to substitute the slogan: “Be fruitful and replicate.” Here, of course, there is a note of caution: If the news come off the UPI wire, then reading the news in several different newspapers does not actually tell us anything new. Similarly, reanalyses of bad data—no matter how sophisticated the reanalysis—probably will not help that much in getting at the truth (cf. Adcock and Collier Reference Adcock and Collier2001).

55. Indeed, one reviewer of this article made the important and plausible claim that “New data and New findings and New theory and New methodology all at once is virtually a guarantee that you will NOT get published. The work will be too different or too innovative for most reviewers to recognize.”

56. Note that the NICE principle involves a four-way interaction effect, not an additive model. Thus, for example, as the experimental economist Catherine Eckels (personal communication, June 13, 2014) observed: “The mere fact that something has never been done before does not make it therefore worth doing.”

57. In this context, it also is well to remember a point I made in earlier work (Wuffle Reference Wuffle1989), namely, “articles that are not submitted are very unlikely to be published.”

58. Being me, I have more to say, but observation 19 seemed a good place to end because, were I Luther, I then would have exactly 76 theses/trombones still to go.

References

REFERENCES

Adcock, Robert, and Collier, David. 2001. “Measurement Validity: A Shared Standard for Qualitative and Quantitative Research.” American Political Science Review 95: 529–45.Google Scholar
Axelrod, Robert. 1986. “An Evolutionary Approach to Norms.” American Political Science Review 80 (4): 1095–1111.CrossRefGoogle Scholar
Bartle, John, and Bellucci, Paolo, eds. 2009. Political Parties and Partisanship: Social Identity and Individual Attitudes. London: Routledge.Google Scholar
Bennett, Andrew, and Elman, Colin. 2006. “Qualitative Research: Recent Developments in Case-Study Methods.” Annual Review of Political Science 9: 455–76.Google Scholar
Bicchieri, Cristina. 2006. The Grammar of Society: The Nature and Dynamics of Social Norms. New York: Cambridge University Press.Google Scholar
Brams, Steven. 1975. Game Theory and Politics. New York: Free Press.Google Scholar
Campbell, Angus, Converse, Philip E., Miller, Warren E., and Stokes, Donald. 1960. The American Voter. New York: Wiley & Sons.Google Scholar
Chwe, Michael Suk-Young. 2001. Rational Ritual: Culture, Coordination, and Common Knowledge. Princeton, NJ: Princeton University Press.Google Scholar
Downs, Anthony. 1957. An Economic Theory of Democracy. New York: Harper and Row.Google Scholar
Eckstein, Harry. 1975. “Case Study and Theory in Political Science.” In Political Science: Scope and Theory (Handbook of Political Science, Volume I), ed. Greenstein, Fred I. and Polsby, Nelson W.. Reading, MA: Addison-Wesley (reprinted in Eckstein, 1992, Regarding Politics:Essays on Political Theory, Stability and Change. Berkeley: University of California Press, 117–78).Google Scholar
Elster, Jon. 1989. Nuts and Bolts for the Social Sciences. New York: Cambridge University Press.Google Scholar
Fenno, Richard. 1978. Home Style. Boston: Little, Brown.Google Scholar
Geddes, Barbara. 2003. Paradigms and Sand Castles: Theory Building and Research Design in Comparative Politics. Ann Arbor: University of Michigan Press.CrossRefGoogle Scholar
Gerring, John. 2006. Social Science Methodology: Strategies for Social Inquiry. Cambridge: Cambridge University Press (Kindle edition).Google Scholar
Granovetter, Mark. 1978. “Threshold Models of Collective Behavior.” American Journal of Sociology 83 (6): 1420–43.Google Scholar
Grofman, Bernard. 1999. “Introduction.” In Elections in Japan, Korea and Taiwan under the Single Non-Transferable Vote: The Comparative Study of an Embedded Institution, ed. Grofman, Bernard, Lee, Sung-Chull, Winckler, Edwin, and Woodall, Brian. 1–20. Ann Arbor: University of Michigan Press.Google Scholar
Grofman, Bernard. 2004. “Downs and Two-Party Convergence.” Annual Review of Political Science 7: 25–46.CrossRefGoogle Scholar
Grofman, Bernard. 2010. “Constraints on the Turnout Gap Between High and Low Knowledge (or Income) Voters: Combining the Duncan–Davis Method of Bounds with the Taagepera Method of Bounds.” Electoral Studies 29 (4): 673–7.Google Scholar
Grofman, Bernard, ed. 2001. “Editor’s Introduction.” Political Science as Puzzle Solving. Ann Arbor: University of Michigan Press.Google Scholar
Grofman, Bernard, and Schneider, Carsten. 2009. “An Introduction to Crisp-Set QCA, with a Comparison to Binary Logistic Regression.” Political Research Quarterly 62 (4): 662–72.Google Scholar
King, Gary, Keohane, Robert, and Verba, Sidney. 1994. Designing Social Inquiry. Princeton, NJ: Princeton University Press.Google Scholar
Lees, Charles. 2006. “We Are All Comparativists Now: Why and How Single-Country Scholarship Must Adapt and Incorporate the Comparative Politics Approach.” Comparative Political Studies 39 (9): 1084–108.Google Scholar
Lijphart, Arend. 1999. Patterns of Democracy. New Haven, CT: Yale University Press.Google Scholar
Owen, Guillermo, and Grofman, Bernard. 1988. “Optimal Partisan Gerrymandering.” Political Geography Quarterly 7 (1): 5–22.Google Scholar
Persson, Torsten, and Tabellini, Guido. 2003. The Economic Effects of Constitutions. Cambridge, MA: MIT Press.Google Scholar
Posner, Daniel N. 2004. “The Political Salience of Cultural Difference: Why Chewas and Tumbukas Are Allies in Zambia and Adversaries in Malawi.” American Political Science Review 98 (4): 529–45.Google Scholar
Posner, Daniel N. 2005. Institutions and Ethnic Politics in Africa. New York: Cambridge University Press.Google Scholar
Posner, Daniel N. 2007. “Regime Change and Ethnic Cleavages in Africa.” Comparative Political Studies 40 (11): 1302–27.Google Scholar
Schelling, Thomas C. 1978. Micromotives and Macrobehavior. New York: Norton.Google Scholar
Skyrms, Brian. 1996. Evolution of the Social Contract. New York: Cambridge University Press.Google Scholar
Taagepera, Rein. 2008. Making Social Sciences More Scientific: The Need for Predictive Models. New York and Oxford: Oxford University Press.Google Scholar
Taagepera, Rein, Selb, Peter, and Grofman, Bernard. Forthcoming. “How Turnout Depends on the Number of Parties/District Magnitude.” Journal of Elections, Public Opinion and Parties.Google Scholar
Tan, Netina, and Grofman, Bernard. 2014. “Multiseat Plurality Bloc Voting in Electoral Authoritarian Regimes: Comparing Singapore, Cameroon, Djibouti, and Chad.” Prepared for delivery at the Annual Meeting of the Asian Studies Association, Singapore, July 17–19.Google Scholar
Taylor, Steven, Shugart, Matthew, Lijphart, Arend, and Grofman, Bernard. 2014. U.S. Government in Perspective: A Thirty-Nation Comparison. New Haven, CT: Yale University Press.Google Scholar
Verba, Sidney. 1967. “Some Dilemmas in Comparative Research.” World Politics 20 (1): 111–27.Google Scholar
Wuffle, A. 1986. “Reflections on Academia.” PS: Political Science and Politics 19 (1): 57–61.Google Scholar
Wuffle, A. 1989. “Uncle Wuffle’s Advice to the Advanced Graduate Student.” PS: Political Science and Politics 22 (4): 838–9.Google Scholar
Wuffle, A. 1999. “Credo of a ‘Reasonable Choice’ Modeler.” Journal of Theoretical Politics 11 (2): 203–6.Google Scholar