Hostname: page-component-7c8c6479df-5xszh Total loading time: 0 Render date: 2024-03-29T14:17:30.728Z Has data issue: false hasContentIssue false

Making the implicit quality standards and performance expectations for traditional legal scholarship explicit

Published online by Cambridge University Press:  13 March 2019

Abstract

Scholars in search of quality standards for traditional legal scholarship could well end up disappointed. By answering the question concerning what standards legal academics use for evaluating such works—through reviewing the international literature on evaluative standards and interviews with forty law professors—this Article aims at filling this gap. This Article recommends that traditional legal scholarship is judged by using the following criteria: (1) the conceptual design—a clearly formulated research question that is both original and significant and the adequacy of the methods proposed to answer that question; (2) the composition of a particular line of reasoning—does the researcher adhere to principles of accountability, accuracy, balance, and credibility?; and (3) the overall characteristics of a work of scholarship—the readability and persuasiveness of the whole and the extent to which the researcher managed to identify and clarify the presuppositions that may have potentially affected her inquiry.

Type
Article
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
© 2019 The Author. Published by Cambridge University Press on behalf of the German Law Journal

A. Introduction

As legal academics, we constantly engage in the process of evaluating works of legal scholarship. In a variety of contexts and with different purposes, we either explicitly or unconsciously put quality labels on the work we come across and produce ourselves. We consider some works to be outstanding, inspiring us to start new research projects, or making us want to cooperate with or hire the author. We encounter pretty good works which we believe should be published in our journals, or that we decide to build on when we conduct our own research. And we—all too often, according to some—encounter works that we instantly dismiss as poor quality, sometimes even making us seriously doubt the capabilities of those that authorized publication.Footnote 1 While it makes perfect sense that we pass such judgments, it does raise the question of how, and especially on what grounds we do so: What standards do legal academics use for evaluating works of legal scholarship?

Given the long history of legal scholarship, one might expect an answer to that question to be quite highly developed.Footnote 2 One is likely to be disappointed, however. Unlike their colleagues operating in other academic disciplines, legal academics rarely engage, individually or collectively, in a more general and thorough analysis of their quality standards.Footnote 3 Consequently, what counts as evidence of “outstanding” legal scholarship has remained largely inexplicit, mysterious, and ill-defined. For a variety of reasons, explained in paragraph B, I believe this to result in a suboptimal research environment.Footnote 4 This Article is an effort to take a leap forward through, first, a systematic and critical review of the scarce and fragmented contributions that have proposed quality standards—paragraph C—and, second, a presentation of the results of an empirical study that was aimed at verifying, supplementing, and bringing nuance and deepness to the insights present within that literature—paragraph D.Footnote 5

Because very different forms of legal scholarship exist—from the normative to the descriptive, from narrow doctrinal analyses to large sample size statistical investigations, from case notes to elaborate monographs—and it would be presumptuous to suggest that all of that scholarship should or could hold on to one set of views about quality, I was—like others who have addressed the topic beforeFootnote 6—forced to confine myself to discussing standards for one type of legal scholarship only. As it still lies at the heart of our discipline, is practiced by legal scholars all over the world every day, and has no other academic disciplines to directly rely on,Footnote 7 my focus rests on what is usually called traditional legal scholarship.Footnote 8 In short, traditional legal scholarship is the type of scholarship that focuses on the way in which judges, legislators, or administrators have decided or should decide on legal questions from a legal—thus, not economic, social, moral, etc.—perspective.Footnote 9

Right from the start, I do have to temper expectations. If the debates on quality standards present in other academic disciplines teach us anything, it is that developing a comprehensive, all-encompassing, operationalized set of quality standards is simply impossible.Footnote 10 Any proposition of such a set is necessarily open-ended, assailable, non-absolute, and no more than “a snapshot in time of a set of emergent ideas.”Footnote 11 That, however, has not withheld scholars operating in other disciplines from having a vibrant debate about their quality standards, and it does not keep me from trying to breathe life into a debate about criteria for considering traditional legal scholarship now. What it does mean, though, is that the reader is encouraged to consider the insights presented in this Article critically, to treat the proposed performance expectations with a healthy form of suspicion, and to actively engage in a further debate about the rules of the game we all like to play.

B. The need for explicating standards or performance expectations

Prior to presenting what I found, let me first express why it is that I—and I am not alone hereFootnote 12—think our discipline should be involved in a more conscious dialogue about its quality standards, and, as our community has operated for so long without any, why now would be a good time to begin doing so.

The absence of clearly articulated criteria made several scholars conclude earlier that our method of evaluating the work of our colleagues is essentially intuitiveFootnote 13 and/or based on criteria of meritocracy.Footnote 14 Besides the fact that intuitionism of this sort—scholars acting as oracles—and the reliance on merit seem seriously out of place in academia, it also—and more crucially—creates specific dangers. Most importantly, when relying on intuition or merit rather than agreed-upon criteria when evaluating works of scholarship, any scholar, no matter how conscientious, may more easily fall prey to personal opinion, bias, or even idiosyncrasy that cannot be justified in general terms.Footnote 15 That is especially problematic if one realizes that far-reaching consequences can be attached to the quality judgements we render.Footnote 16 An articulated, coherent theory of evaluation may help here, as it can both create altitude and a means of motivating and justifying our judgements.Footnote 17

Articulated criteria may, however, also be helpful in the sense that they can guide us in seeking to make our own work the best example of what we understand our scholarship to be, or at least contribute to the avoidance of sloppy science. Here, rather than evaluatively, criteria are used “aspirationally” as a means for members of the scholarly community to improve their work.Footnote 18 If we have a clear idea about what evidences outstanding research, we may more consciously focus our research process accordingly. While one could hardly deny that all who are in one way or another pursuing legal research could in this way benefit from articulated criteria, the aspirational function of criteria is especially affecting students and scholars that stand at the beginning of their academic careers.Footnote 19 To get acquainted with the standards legal scholars are expected to adhere to in another way than finding out by trial and error what supervisors, peer reviewers, or journal editors apparently expect, would certainly be a welcoming development.Footnote 20

A final reason for having a debate about our standards that I would like to mention, is that the articulation of criteria may have political value for our discipline.Footnote 21 The importance of providing evidence to governments and other funding agencies that their investment in research provides good value for money in terms of quality and impact has increased in nearly all countries.Footnote 22 Therefore, whether we like it or not, our sponsors will, and already are, measuring the quality of the products in which they invest. To insulate our work from the imposition of inappropriate criteria developed by other academic disciplines when such assessments are made, we cannot escape from developing and/or clarifying our own standards. As doing so would help us defend the validity of our scholarly activities and objectively demonstrate the quality of the work we produce,Footnote 23 we may, moreover, eventually even be able to reverse the diminishing position of traditional legal scholarship on the market for research funds.Footnote 24

C. State of the art

The absence of a vivid scholarly debate on standards for traditional legal scholarship does not alter the fact that, from various angles, the legal academic community has put effort in the development of explicated quality criteria. Repeatedly, quality standards are the subject of discussion in committees appointed to decide on the (re-)accreditation and/or performance of universities and their respective research groups, programs, and institutes.Footnote 25 Also, quality standards are being discussed among editorial boards of law journals as they grapple with the question of how to determine whether a scholarly work qualifies for publication. Earlier studies of the instruments used for these purposes (assessment protocolsFootnote 26 respectively author guidelinesFootnote 27) revealed that, if quality standards are mentioned all mentioned, three in particular stand out: Originality, thoroughness, and profundity. In nearly all instances, however, it remains unclear what these exactly mean. All of the studies analyzing assessment protocols and/or author guidelines conclude that standards—if at all mentioned—mostly remain “extremely broad,” “extortionately vague,” and “multi-interpretable.” In these assessment protocols, Van Gestel and Vranken, for instance, did not find much more information than that originality refers to contributing “something new to the scholarly debate on a subject,” thoroughness “describes the extent to which authors should try to find underlying explanations or theories for developments in law (asking why questions instead of how questions),” and profundity is associated with visible competence and the methodological or analytical rigor of a particular inquiry.Footnote 28

Greater insight into the standards that may be used to judge works of traditional legal scholarship comes from scarce, but available, scholarly contributions.Footnote 29 Around 1990, the desire of several American legal scholars operating in minority disciplines—feminism and critical race theory—to develop objective standards for judging their work resulted in a small debate in the United States on the different quality standards that apply to traditional and non-traditional legal scholarship. From this debate, two works stand out in particular:Footnote 30 Rubin’s On Beyond Truth: A Theory for Evaluating Legal Scholarship, and Kissam’s The Evaluation of Legal Scholarship. Inspired by the famous philosophers of science, Jürgen Habermas and Hans-Georg Gadamer, Rubin argues that a piece of traditional legal scholarship should meet standards of “normative clarity,” “persuasiveness,” “significance,” and “applicability.” In order to reach normative clarity, Rubin argues that the scholar is required to clearly identify so-called “controlling norms”Footnote 31—the premises running through a work—and to ensure that specific arguments are appropriately related to, or flow from, these controlling norms.Footnote 32 Persuasiveness refers to both the extent to which an evaluator believes the work should convince the public decisionmakers to whom it addressesFootnote 33 and the authenticityFootnote 34 of the interpretations of legal texts—case law, statutory provisions, or parliamentary documents—present in the work.Footnote 35 Significance, then, relates to the extent to which the work is part of the ongoing developments in a particular field; the extent to which the contribution is of interest and importance to the community and influences the field by changing the way people think.Footnote 36 Finally, applicability refers to the extent to which the evaluator believes that the work contains an identifiable thought that can be used by other legal scholars.Footnote 37

Kissam approaches the question of quality standards from a different angle. Starting from the more general values or motives for conducting academic scholarship, Kissam considers the achievement of an originality and competence standard necessary for conducting “good scholarship.” According to him, the originality standard may be satisfied if a scholar discovers new knowledge or if the scholar manages to disseminate existing knowledge in an original manner so that it becomes more useful to professional audiences.Footnote 38 Competence, then, implies that a piece of scholarly work “must be factually accurate, written in a comprehensible manner, and be based on appropriate methods.”Footnote 39 If scholarly work is both original and competent, Kissam continues, it may be indicated as important or even outstanding depending on the extent to which it affects the lives, works, and thoughts of professionals and academic colleagues.Footnote 40

In Europe, mostly as a response to the increasing focus in universities on the quantitative evaluation of scholarly output,Footnote 41 several scholars and committees have—mainly in policy documents—called for a re-focus on the substantive quality of legal publications. Verbeke’s proposition, one of the most elaborate publicly accessible among these types of contributions, is exemplary.Footnote 42 He proposes to differentiate more consciously between academic and non-academic research. Therefore, he suggests one cardinal criterion—”scientificness”—and four additional criteria—”analytical,” “original,” “critical,” and “clear.” According to Verbeke, being scientific means that the research should be solid—for example proper referencing, all claims supported by references or argumentation, all relevant sources checked—comprehensive—for example all aspects of research question covered, all perspectives checked, arguments pro and contra considered and analyzed, open for falsification—and transparent—for example clear research question, source selection strategy clarified, explication of underlying legal, philosophical, ethical, sociological frameworks. With the analytical and critical criteria,Footnote 43 Verbeke seems to mean that the scholar should move beyond mere description of relevant source material—abstraction, working with a theoretical and conceptual framework, critical assessment, among others. Originality refers to some novelty in content, ideas, approach, and critical assessment, but also in structure or methodology. Finally, being clear means that the publication should present its analysis and argumentation in a clear and stringent style, offering a high degree of lucidity and clarity.

The most recent and elaborate contribution on quality standards for traditional legal research I came across, is a Dutch book written by Vranken. Vranken explores the similarities and differences in the rules of the game that apply to judges and legal academics.Footnote 44 Interestingly, he—more so than Rubin, Kissam, and Verbeke—distinguishes between quality standards related to the development and embedment of a research question and the quality of the composition of an answer to that question. With respect to the research question, Vranken identifies “novelty”—in the sense that the question posed should be able to generate new knowledge—as its most important requirement.Footnote 45 According to Vranken, this also implies that a scholar is required to situate her question against the “backdrop of existing knowledge.”Footnote 46 The newness criterion closely mirrors, but is distinguishable from, the requirement that the question must be worth answering from a scientific and societal perspective—the requirement of “relevance.” Besides being original and relevant, Vranken argues that a research question should also be “precise”; It must allow the reader to get to know exactly what the author means by the core concepts used and what aspects are going to be included and excluded from the project.Footnote 47 The standards Vranken considers to apply to the construction of a line of reasoning in answering a particular research question, seem to be mainly inspired by the Dutch and European “code of conduct” for academic research:Footnote 48 The materials on which the answer is based should be “complete” and up-to-date or—if completeness is not possible because there are simply too many relevant sources—”representative” and including justification thereof; statements should be accompanied by “correct references”; the answer must be “reliable”;Footnote 49 the research materials used by the scholar must be able to “carry the conclusions”; and, the answer must be “persuasive” and “easy to read.”Footnote 50

The problem these accounts share, is that however thoughtful and considerate they may be, it remains unclear whether they represent the ideas and views of the author concerned or those of the scholarly community as a whole.Footnote 51 As different authors appear to focus on slightly or partly different standards or maintain different interpretations thereof, I believe it is important that we start approaching the search for standards differently; in a way that allows us to capture the conceptions—and the patterns and variations therein—of the legal scholarly community as a whole. Initiatives hereto are already taken. Scholars operating in a relatively new subdiscipline aimed at empirically studying research assessment methods and techniques have started to take legal scholarship as an object of study.Footnote 52 As far as I know, the first results of these projects are yet to be published. At first sight, however, the current initiatives seem to be designed in such a way that, while they do allow for revealing abstract quality standards that the legal community considers applicable to certain forms of traditional legal research, they do not aim to operationalize those standards in any depth.

D. Empirical insights

Preceded by providing them with the opportunity to openly express their views about standards and performance expectations for traditional legal scholarship, I presented the insights from the literature—in semi-structured interviews—to forty Dutch law professors who regularly publish internationallyFootnote 53 for commentary, discussion, and further operationalization—what do these standards mean for daily research practice?Footnote 54 That allowed me to achieve—exploratory, but still—insights in the standards the legal scholarly community as a whole considers important and the way in which these standards are to be interpreted and operationalized. Interestingly, the data that resulted from the interviews showed that the respondents—like Vranken—felt a need to distinguish among standards that relate to the embeddedness of the research problem or question and the conceptual design (paragraph D.I), the composition of an answer to a research problem or question (paragraph D.II), and the publication as a whole (paragraph D.III). Despite the fact that the respondents recognized and emphasized the importance of most of the standards I encountered within the literature, the data did give rise to the formulation of a set of standards that does not seamlessly coincide with one of the sets proposed before. A brief overview of that set is presented in Table 1. Before addressing the individual standards in more detail, let me remark, though, that while they are distinguishable for analytical purposes, the different standards are, of course, interrelated and sometimes even partly overlapping. Where I found such to be the case, I reported it.

Table 1: Overview of Quality Standards

I. Requirements related to the embedment of the research question and the conceptual design

According to the respondents, an academic traditional legal publication must commence with a “clear problem definition,” preferably shaped as a “concretely and precisely formulated research question or hypothesis.” Here, the scholar should “indicate as precisely as possible what it is that is going to be researched.”Footnote 55 It is important that this question is “linguistically correct,” “easy to understand,” and “does not contain internal contradictions or inconsistencies.” Apart from that, the scholar also has to make sure that “possibly ambiguous or multi-interpretable terms used in the question are conceptualized” before, in, or shortly after the question is presented, allowing author and reader to remain on the same page throughout the publication.Footnote 56

Subsequently, it is expected from the scholar that her contribution is both “original” and “significant.” The respondents identified the fact that an inquiry has a possible “consequence” as the key distinguishing feature of a significant contribution. A scholarly publication has to be of some interest and importance to certain societal actors—societal significance or relevance—and the academic community—scientific or theoretical significance or relevance. “Originality” was defined with respect to the “newness” of a contribution and, more specifically, to the question whether that publication in fact “adds something to the existing body of academic knowledge.”Footnote 57 The respondents emphasized the importance of not leaving the estimation of a publication’s originality and significance to the sole discretion of the reader. Rather, the scholar is well advised to accommodate the reader with a “persuasive justification” thereof. Here, the author may, for instance, argue that her contribution is original in the sense that it identifies or addresses a problem that others have not identified or addressed before, uses a new perspective, theory, or method to readdress a problem others have touched upon before, investigates a known problem by using new and/or more data, and/or proposes and elaborates (a) new argument(s) or solution(s) in relation to a particular problem.Footnote 58 The project’s societal significance can be explained by indicating in what way(s) the knowledge generated can be useful to lawyers, courts, legislators, or litigants, or how it may contribute to solving a societal problem.Footnote 59 The scientific significance, finally, can be justified by substantiating how answering the research question contributes to the development of a particular doctrine, provides insight in the bigger questions the legal discipline faces, or inspires others to tackle other legal problems using a similar approach.Footnote 60 The consequence of these first requirements for the scholar is having to achieve originality and significance is that the scholar is expected to conduct thorough preliminary research with respect to the research topic. After all, in order to be able to indicate the originality and significance of a work, the legal scholar cannot escape from thoroughly investigating and reporting whether other scholars have previously addressed the problem she is interested in, and if so, what perspectives, theoretical approaches, methods, and data were used and/or what solutions were proposed. In other words: The research question must be placed within its disciplinary context.Footnote 61

If the researcher succeeds in establishing an original and significant research question, the next thing she is expected to do is clarify and justify the steps necessary—the research methods and techniques—to come to an answer to that question. Of course, those steps depend largely on the exact research question posed. The respondents emphasized that it is important that the researcher at least explicates and justifies the non-self-evident choices made here. With respect to research projects that contain an evaluative or normative component—popular within traditional legal scholarship—the respondents argued that a scholar by now—as a minimum—is required to establish a research framework in which the criteria on the basis of which an evaluative or normative judgement is made are explicated and operationalized.Footnote 62 An obvious example would be a research project in which a particular legal doctrine—for example, civil liability of a governmental agency for insufficient supervision—is evaluated on its compliance with international human rights. In such a research project, the scholar will have to explain what demands flow from human rights and how they can be measured before she will be able to investigate whether the doctrine under scrutiny actually complies with human rights.Footnote 63 A project where the presence of a framework would already be less self-evident is, for instance, a project that considers whether more intensively regulating the pre-pack would be a good idea. In order to reach a substantiated conclusion to that question, the scholar will, however, also have to develop such a framework. It may, for instance, be constructed from the perspective of the protection of employees, guaranteed by national legislation and/or European law, national and/or international principles of liability or insolvency law, something completely different, or a combination thereof.Footnote 64

II. Requirements related to the composition of a particular line of reasoning between question and answer

1. Accountability

According to the respondents, the traditional legal scholar should be transparent about the complete process that was followed in answering a question. This suggests that the first criterion for excellence in legal scholarship is a principle of accountability. The data revealed that being accountable has a dual connotation: It refers both to the use of references and to the explication and justification of important choices made within a particular research project. What is incontestable is that the accountability criterion implies that the reader must be placed in a position in which she can easily retrieve the materials on which the statements in a publication are based. Therefore, a traditional legal publication should “exhibit a precise and correct body of footnotes.”Footnote 65 This implies that: (1) a reference should actually be included where the scholar draws from the work of others—literature, case law, legislative documents, etc.;Footnote 66 (2) the reference should be precise—indicating the exact page(s) or section were the information can be found; and (3) the reference should be correct in the sense that the research materials referred to in fact contain information that supports the interpretation, argument, or statement of the researcher.Footnote 67

Although important, the fact that a publication contains a precise and correct body of footnotes does not necessarily mean that the researcher sufficiently adhered to the principle of accountability. According to the respondents, the legal scholar is also expected to “justify” the most important,Footnote 68 non-self-evident choices—choices that the majority of readers cannot infer from the references used and that do not flow from common sense or generally shared knowledge—made in composing an answer to a research question.Footnote 69 Justification means: (1) explication that a choice was made; (2) explanation why that choice was made; and (3) clarification of the possible implications the choice may have for the research results.Footnote 70 The last aligns with the thought expressed by nearly all respondents that a researcher cannot and does not have to opt for the theoretically most optimal choice in all instances—consider only time constraints—as long as she honestly reports what the suboptimal choice could mean for the trustworthiness of the findings.Footnote 71 Or, as Epstein and King framed it:

The sign of good research is not that every problem be fixed in every project, since that is impossible and trying to do so would mean we would not accomplish much of anything, but rather, that the problems be noted and the uncertainty in substantive conclusions be responsibly and honestly reported.Footnote 72

An example might be insightful here. Suppose a European constitutional law researcher is interested in a particular legal problem concerning the doctrine of trias politica. Of course, the researcher refers to Montesquieu and Rousseau as the historically most important scholars that have written on the topic, to Conte because he identified the problem the researcher is interested in, and subsequently to two German law professors that have recently addressed the problem from a more modern perspective. Critical questions may probably rise instinctively: Would it not be possible that the researcher reaches different conclusions if she would have consulted a broader scope of research materials? Are there other materials that contradict or bring further nuance to the views represented in the sources used by the researcher; and if so, would it not have been necessary to include these materials in the inquiry? If I understood the respondents correctly, the researcher would have better satisfied the criterion of accountability if she—in addition to precise and correct referencing—had explained why she only used the works of the two German law professors. For instance, by indicating that a systematic search through the international literature revealed that no one else has addressed the problem in the past decade(s) or that those law professors provided a reliable overview of other relevant earlier works, she may assert that a separate study of these materials was regarded unnecessary.

Except for showing that merely referring in a precise and correct way may thus be insufficient to operate in adherence with the principle of accountability, the example shows the close connection between the principle of accountability and the standards I will address hereafter. The way in which the researcher operates in the example above also raises questions that relate to other quality criteria—can we speak of a “balanced” use of research materials? Is it possible to come to “credible” statements by consulting only these two modern scholars? The example finally also clarifies that not all choices made by the researcher require similar elaboration. The choice to use the publications of the two German law professors probably raises more critical questions than the choice to build on the works of Montesquieu, Rousseau, and Conte. Apparently, certain choices are more logical or self-evident than others, and for that reason require less justification.

2. Accuracy

The second criterion suggested by the respondents is that a work of traditional legal scholarship should be “accurate.” In its broadest sense, this criterion means that “what follows after the problem definition must actually contribute to answering it.”Footnote 73 According to the respondents “there must exist a clear fit between the problem definition and conclusions,” substantiated by a “clear line of reasoning” in between. In an outstanding traditional legal publication, “the diverse components of the research project are connected in a seamless way and the conclusions tie the whole thing together.”Footnote 74 “Needless meandering” is to be prevented as much as possible. This does, however, not alter the fact that a publication can be of enormous complexity and may touch upon many different themes and subthemes. If such is the case, however, the discussion of these (sub)themes must both still in some way contribute to answering the research question and be presented in a logical sequence and at a convenient point in the publication.

For the use of research materials, being accurate means that a scholar should be both “exact” and “meticulous.” What that exactly conveys can be clarified by providing an example. Suppose that a researcher aims to answer the following (sub)question: “Under what conditions can a government agency be held liable for damages that were caused by inadequate supervisory activities of that agency?” Let’s assume that there exists an abundance of research materials—policy documents, case law, and commentaries within the literature—that “only” need to be organized and systematized. Suppose the researcher studies dozens of cases that deal with the liability of non-state, private actors for inadequate supervision. The fact that the researcher studies materials that are all about the same topic, means he operates “meticulously.” Here, however, at least at first sight, the researcher does not live up to the principle of “exactness,” as the research materials do not deal with liability of government agencies. The reader might rightly wonder whether the question about liability of governmental agencies can actually be answered by the consultation of materials on liability of private actors. To compensate for this potential accuracy problem, the researcher must persuasively justify the choice to build on these particular cases related to private actors.

A complicating factor here is that the traditional legal scholar often pursues research aimed at positioning a new societal or legal development within the legal system, predicting how a legal (sub)system may evolve, and/or improving a legal (sub)system, instead of clarifying a legal doctrine by merely organizing and systematizing existing research material.Footnote 75 In such cases, there often exist little materials of which the relevance for answering the question posed is beyond doubt. The researcher has to rely heavily on the associations she or others before her were able to draw; associations with legal principles, international law, legal doctrines in other fields of the law—internal comparative law—or other jurisdictions—external comparative law—insights from other academic disciplines, or a combination thereof.Footnote 76 These associations subsequently determine what research materials are consulted. The consequences flowing from the principle of accuracy would then be that the researcher is expected to: (1) be exact and meticulous in the elaboration of the association itself; and (2) explain why the association itself does not erode the accuracy of the inquiry.Footnote 77

3. Balance

Besides the principles of accountability and accuracy, the respondents also considered a principle of “balance” to be applicable to traditional legal scholarship. To be able to reach balance, it is indispensable that “the legal scholar is as complete as possible and necessary in her consultation of research materials that are relevant to the investigation.” If inquiry is pursued on a relatively limited field and based on a sharp, focused, and specific research question, and/or is predominantly oriented on typically national sources, achieving such absolute completeness in the consulted materials is actually practically “possible and therefore also necessary.”Footnote 78 If the research question is, however, broader—which is, for instance, soon the case if it addresses a topic with an international or European dimension—the respondents recognized the practical impossibility, and therefore also unnecessity, of achieving absolute completeness.Footnote 79 If such is the case, that “does not mean that the legal scholar should not strive to be as complete as she can possibly be,” but rather that “making deliberate selections becomes inescapable.” Through selection, the researcher must then make sure that she “consults sources that offer a representative picture of the important different interpretations, arguments, and solutions that have been proposed within the conglomerate of relevant source materials.”Footnote 80 The reader won’t always be able to estimate whether the researcher succeeded in this respect. Therefore, the researcher is well advised to anticipate by explaining why and how she thinks she managed to achieve such a representative image, especially at the points where the completeness of the inquiry might be challenged.Footnote 81

Being as complete as possible is, in itself, still insufficient to achieve a balanced inquiry. The researcher should also make sure that different insights represented in the research materials obtain an “honest” position within the publication. In that sense, completeness and honesty are complementary. “Attaching defendable cogency” to the different insights found in the research materials is no exact science. That does not, however, imply that the traditional legal scholar enjoys unrestricted freedom in this respect. At least, adhering to the principle of honesty means that relevant interpretations and arguments found may not—consciously nor unconsciously—be omitted or undervalued without any form of justification thereof.Footnote 82 In addition, the researcher should consider the extent to which a particular insight is supported by the scholarly community—is it mainstream or a conflicting position?—as well as the quality—does the work live up to academic standards?—and the authority—Supreme Court or lower court?—of the source that presented it. While support, quality, and authority are important, those are not the only decisive indicators for positioning a particular insight. Of course, the relevance for the research question also matters. According to the respondents, it is not problematic if the researcher prioritizes certain arguments above others for that reason, as long as she makes sure that important alternative arguments are presented to the reader as well.Footnote 83

4. Credibility

Even if the researcher adheres to the principles of accountability, accuracy, and balance, the answer to a research question might still be unconvincing. That is because none of the standards discussed thus far tell us something about the truth-value or—as truth is a somewhat contested concept—the “credibility” of the claims the researcher makes within her publication. Each claim, whether descriptive, interpretative, generalist, evaluative, or theoretical, should be credible.Footnote 84 The respondents emphasized that a descriptive inference—in a strict sense, paraphrasing or summarizing a particular source—is only credible if it is correct.Footnote 85 Paraphrases may not contain any errors, summaries may not omit important elements nor wrench the original text from its context, and certainly out of the question is that a concealed interpretation—or even an opinion—is presented as a descriptive inference.Footnote 86

If the researcher moves beyond the descriptive level, what it takes to be credible becomes much less straightforward. Scholars operating in the field of legal theory have dedicated numerous pages on what constitutes a credible interpretation of a legal source. While it is impossible to reproduce the insights represented in that body of literature here, let me at least highlight two important notions. First, legal theorists debate whether it is even possible to come to a more or less “clean” interpretation or whether every interpretation is in fact a concealed normative statement of the interpreter.Footnote 87 Second, legal theorists are stuck in a continuous discussion about the aim that should be pursued by interpretation: Is it about determining what the author(s) of a particular source meant—sometimes referred to as the “originalist approach”—or is it about determining what the source might mean in its contemporary context—sometimes referred to as the “contextualist approach”?Footnote 88 Fortunately, the respondents adhered to a more pragmatic approach. They spoke of the need to achieve a “defensible” rather than a “correct” interpretation. If a consensus exists among the vast majority of the legal scholarly community about the “right” interpretation of a particular source, the researcher is expected to subscribe to that uncontroversial interpretation or persuasively justify why she wishes to deviate. If such a consensus does not exist, for instance, because of the simple fact that the source has not been interpreted before, the researcher seems to enjoy more, though definitely not unrestricted, freedom. According to the respondents, the researchers may, for instance, never reach beyond the intention of the author(s) of the interpreted materialFootnote 89 and must ensure that a reader that deeply compares the interpretation with the interpreted may not get the idea that there exists insufficient congruence between the two.Footnote 90 Researchers should also preferably clarify the technique(s) used for reaching the interpretation and are advised to honestly report on the possible insecurities that were encountered during the interpretation process.Footnote 91 Finally, if there exists no consensus as to the right interpretation of a source that has been interpreted before, the researcher will—apart from the things mentioned before—be additionally required to pay attention to the way in which her particular interpretation relates to, coincides with, or deviates from the existing interpretations.

For statements or claims based on a cluster of research materials rather than individual materials—is the claim about the scope of the duty of care of a house-buyer that was made on the basis of three judicial decisions valid in the sense that the claim holds if other judicial decisions of the Supreme Court are consulted?—the credibility criterion implies the following. Again, if there exists a generally accepted interpretation of a body of research materials that is supported by a large part of the academic community, the researcher should either subscribe to that interpretation or persuasively justify why deviation is preferred. Often, however, that scenario will not be applicable. The researcher then has different options to choose from. She may decide to generalize on the basis of studying all research materials on a particular theme or on the basis of a sample thereof. While the respondents highly preferred the former, they also recognized that the researcher sometimes has no other choice than to opt for the latter. In that case, the credibility of the generalization may be evaluated by a combination of: (1) considering whether the claim made holds for materials that fall outside the selection; and (2) the persuasiveness of the justification of the process that has led to the claim made.Footnote 92

The credibility of evaluative and normative claims—is this particular judicial opinion or regulatory provision “right” or “wrong” or, in a legal context, “legitimate” or “non-legitimate”?—or classifications—is this judicial opinion an illustration of position A or position B?—depends on the consistent employment of an explicated framework that indicates the criteria on which the claim or classification is based—for instance, does it align with prior case law or norms introduced by the European Court of Human Rights? The credibility of evaluative claims is thus not concerned with the rightness of the evaluation criteria formulated—that would be a question of whether the right criteria are formulated on the basis of sufficiently valid theoretical foundations—but rather with the question whether they were applied consistently.Footnote 93

Finally, traditional legal scholars also bring theoretical constructs to a study: The scholar argues that a particular court took a more “conservative” approach, that the introduction of a certain new legal provision is an “act of resistance” towards developments on the commercial market, or that a particular court decision is an illustration of the court making a judgment in order to promote the principle of “legal certainty.”Footnote 94 Here, the labels “conservative,” “act of resistance,” and “legal certainty” are introduced by the researcher. Depending on the importance of such labels for the research question, they either must be explained shortly after the presentation thereofFootnote 95 or close to the place where the construct is introduced. As far as the credibility of the use of such theoretical constructs is concerned, there are at least two things that are important. The first refers to the credibility of the theoretical concepts attached to the description or interpretation of a particular phenomenon itself. For example, is the changed attitude of the Supreme Court indeed a sign of the adoption of a conservative approach—and what is the scholar’s understanding of conservative?—or is it rather something else?Footnote 96 In other words: Is conservative—or perhaps more importantly the operationalization of that concept in the publication of the scholar—the right theoretical concept to address what has happened within the judicial opinion? The second aspect refers to the credibility of the theoretical connections made within a particular publication.Footnote 97 If we want to claim that the fact that the Supreme Court “shifted its perspective”—an interpretation—results in the Supreme Court taking a conservative approach—a theoretical construct—because of the fact that, for instance, the European Court of Human Rights has altered its attitude on that particular aspect of the law, we need to make sure that the conservative approach is really rooted in this fact and not predominantly caused by something else.Footnote 98

III. Readability, persuasiveness, and the identification of presuppositions potentially affecting the inquiry

Apart from the quality standards discussed above, the respondents also mentioned standards that relate to the publication as a whole. The respondents indicated that the quality of a publication also depends on whether it is “linguistically correct” and “easy to read.” That means that “no grammar or spelling mistakes must be made,” that “all sentences and paragraphs in the publication must be fully focused and well formulated,” and that “the writing must be pursued with such clarity that the publication is easy to understand and a pleasure to read.”Footnote 99 The readability of a publication also increases if the author manages to be as brief and concise as she can possibly be to get her message across. The only information that should be brought into the actual publication is the information that is really necessary for understanding the proposed answer to the research question.

A traditional legal publication should also be persuasive. According to the respondents, persuasion can be achieved by obtaining excellent writing and structuring, but more often they associated persuasion with argumentation. According to the respondents, “the argument(s) of the author should be focused, logical, rigorous and sustained” in order to be convincing.Footnote 100 The respondents have emphasized that a publication may possess a strong point of view, as long as that does not go hand-in-hand with subjective inquiry. Persuasiveness is enhanced if the scholar can substantiate her point of view with both arguments that speak for the particular point of view taken and arguments that speak against it. The better the scholar succeeds in countering the contra-arguments, the more persuasive the publication becomes. In this sense, thus, the persuasiveness of a publication is strongly determined by the extent to which other quality standards—especially balance and credibility—are met.

Finally, the respondents stated that the difference between an excellent work of traditional legal scholarship and a work of less quality may also rest in the extent to which the researcher actively identifies, explicates, and estimates the implications of the interest, presuppositions, and assumptions on her behalf that may have consciously or unconsciouslyFootnote 101 influenced any aspect—question, theory, or particular statements—of the inquiry.Footnote 102 As Epstein and King stated before, every research project may be influenced by hundreds of factors.Footnote 103 Of course, this standard is not about mapping all of these factors. That is not even possible, as many of those factors relate to the subconscious. It does, however, mean that the researcher should put sufficient effort in trying to identify those presuppositions or assumptions. As a minimum, then, the researcher is expected to explicate these presuppositions and assumptions. Where possible, the researcher is also well advised to factor in contradictory presuppositions and assumptions.

E. Conclusions

That brings my search of what evidences academic excellence to a temporary ending. In this contribution, I have tried to answer the question: “What standards do legal academics use in evaluating works of traditional legal scholarship?” On the basis of a systematic review of the literature on quality criteria for traditional legal scholarship and an empirical investigation that allowed me to reach deeper insights, I established nine criteria divided over three categories that a researcher—aspirationally—or reader—evaluatively—may employ. I found that a traditional legal publication must, (1) contain a concretely and precisely formulated research question or hypothesis that allows for (2) an original and (3) significant contribution to both society and academia and that (4) is substantiated with adequate methods and research techniques for answering it. Subsequently, in composing the answer to a traditional legal research question, the researcher must (5) use precise and correct references and justify important choices made, (6) be accurate in the sense that the arguments and materials used must actually contribute to answering the research question posed, (7) operate in a balanced way—a complete or representative use of research materials and a fair distribution of weight to the different arguments and insights present within these materials—and (8) achieve a high level of credibility in her descriptions, interpretations, evaluations and use of theoretical constructs. Finally, a traditional publication must also be (9) readable and (10) persuasive, and (11) may not—at least not unaccounted for—be influenced by the presuppositions or assumptions of the author herself.

While this enumeration constitutes the brief answer to the question posed in this Article, a few remarks have to be made. First, the retrieved standards are still formulated in a somewhat abstract way. As was also remarked by some of my respondents, I believe that more insights in the exact manifestation of each of the standards in research and evaluation practice can only be achieved by providing examples of how they play out in particular research scenarios. Also, the enumeration may give the impression of a “checklist.” Such has however never been the aim of this project. Therefore, I hope the reader does not absorb the information provided in such a way. I do not believe it to be a good idea to hold on to the different standards for dear life in future research projects. Rather, it is important to understand the more general, common principles behind those standards and use this understanding to find a balance between being systematic and pragmatic in order to efficiently produce high-quality academic research.

At the start of this contribution I expressed the hope that our discipline engages in a more vivid debate about its quality standards. While this Article provides a first step in this direction, I call upon others to come up with alternatives, modifications, or reinforcements. I address this call to legal scholars—who can contribute by starting to publish more often on the quality of legal research—but also to faculty boards, graduate schools, publishing houses, and journal editors. As the latter actors are specifically involved in the business of evaluating works of traditional legal scholarship, they could certainly do more than they do now to clarify how they do so.

Footnotes

*

Marnix Snel is an associate professor in private law and legal research methods at the University of Curaçao, a research fellow of the Tilburg Institute for Private Law, and a visiting lecturer at Nyenrode Business University. Email: m.v.r.snel@gmail.com.

References

1 See Lee Epstein & Gary King, The Rules of Inference, 69 U. Chi. L. Rev. 1, 18–19 (2002) (“There seems to be a long tradition of legal academics denigrating articles published in their journals. Over the years, they have referred to the content in these journals as ‘junk stream,’ ‘manure,’ ‘[not] readable by humans,’ ‘fuzzy wuzzy,’ ‘junk science,’ ‘spinach,’ ‘boring, too long,’ rife with ‘assertions unconnected to an empirical basis,’ dependent on ‘anecdotes,’ ‘opaque’ and an ‘open scandal.’”).

2 See Mary I. Coombs, Outsider Scholarship: The Law Review Stories, 63 U. Colo. L. Rev. 683, 706 (1992).

3 A search for publications on quality standards or quality evaluation in the social science databases quickly resulted in hundreds of relevant hits, while I only identified about thirty relevant contributions by performing a similar search in the international legal databases. See Alain L. Verbeke, Beyond Quantity—Classifying and Evaluating Legal Research in a Trusting Environment, Leuven L. Res. Classification & Evaluation Model 1, 1 (2013); Theunis Roux, Judging the Quality of Legal Research: A Qualified Response to the Demand for Greater Methodological Rigour, 24 Legal Educ. Rev. 173, 186–87 (2014).

4 Rubin goes as far as to argue that the absence of an evaluative theory has led to a wide range of conceptual and practical difficulties that “have festered to produce confusion and malaise throughout the field.” He argues that many of the most significant debates in legal scholarship—for instance, about the value of critical legal studies and its relationship to traditional scholarship, or the contribution that other fields such as economics, literature, and social science can make to legal thought—involve evaluation, and that many of these are repetitive and unproductive for a lack of an evaluation theory. Edward L. Rubin, On Beyond Truth: A Theory for Evaluating Legal Scholarship, 80 Calif. L. Rev. 889, 889—94 (1992).

5 Information on the methods followed is presented in paragraph D.

6 See generally Philip C. Kissam, The Evaluation of Legal Scholarship, 63 Wash. L. Rev. 221 (1998); Rubin, supra note 4; Rachel J. Anderson, From Imperial Scholar to Imperial Student: Minimizing Bias in Article Evaluation by Law Reviews, 20 Hastings Women’s L.J. 197 (2009); Roux, supra note 3.

7 When conducting socio-legal research, for example, the scholar has access to a body of sociological literature on quality standards that, at least partly, apply.

8 For contributions that address the standards that apply to other types of legal scholarship, see generally Epstein & King, supra note 1 (empirical legal research); William Nelson, Standards of Criticism, 60 Tex. L. Rev. 447 (1982) (legal history); Nancy Levit, Defining Cutting Edge Scholarship: Feminism and Criteria of Rationality, 71 Chicago-Kent L. Rev. 947 (1996) (jurisprudence); Susan Bartie, The Impact of Legal Meta-Scholarship: Love Thy Navel, 18 Griffith L. Rev. 727 (2009) (meta-scholarship); Coombs, supra note 2 (feminism and critical race theory); Gijs van Dijck, Kwaliteit van de Juridische Annotatie: Een Empirische Studie naar Kenmerken en Kwaliteitsindicatoren (2011) (case notes).

9 Of course, there is much more to say about the nature and characteristics of traditional legal scholarship. As the purpose of this Article is to focus on the debate on quality standards, I hope the reader accepts that I subscribe to definitions provided by, for instance, Rubin, supra note 4, at 902–10.

10 A few philosophers of science have even contested the possibility of defining quality standards. See generally Roderick Chisholm, Theory of Knowledge (3d ed. 1989); Thomas F Gieryn, Boundary-Work and the Demarcation of Science from Non-Science: Strains and Interests in the Professional Ideologies of Scientists, 48 Am. Soc. Rev. 781 (1983). I, however, start from the assumption that it is possible to define quality standards, an assumption that is present among nearly all scholars that contributed to the debate on quality evaluation criteria in several disciplines.

11 See Yvonne S. Lincoln & Egon G. Guba, Naturalistic Inquiry 329 (1985); Margaret A. Eisenhart & Kenneth R. Howe, Validity in Educational Research, in The Handbook of Qualitative Research in Education 4 (Margaret D. LeCompte, et al. eds., 1992); Coombs, supra note 2, at 705. Abrams remarked that understandings developed even within a particular subdiscipline—such as traditional legal scholarship—are probably not even unitary nor uncontested. See Kathryn Abrams, Hearing the Call of Stories, 79 Calif. L. Rev. 971, 1018–1019 (1991).

12 Recent empirical research has confirmed that—for one or a combination of the reasons presented below—the majority of the members of the Dutch and Swiss legal scholarly communities by now seem to consider the current situation—in which our quality standards are largely implicit and mysterious—untenable. See Rob A. J. van Gestel, Karin Byland & Adreas Lienhard, Evaluation of Legal Research: Comparison of the Outcomes of a Swiss and Dutch National Survey, 23 Tilburg L. Rev. 3 (2018).

13 See Rubin, supra note 4, at 889; Kissam, supra note 6, at 221–22.

14 See Levit, supra note 8, at 947; Richard A. Posner, The Future of the Student-Edited Law Review, 47 Stan. L. Rev. 1131, 1133 (1995). According to Posner, such criteria may be the reputation of the author, the author’s relation to the editors, the prestige of the author’s law school, the length of the article, and the number and length of the footnotes in it.

A logical exception that might be raised to this point is that the quality of inferences can be judged by knowing the reputation of the investigator…. This kind of appeal-to-authority, however, is wholly irrelevant to valid inference. Even if it were true that famous authors are wrong less often than obscure authors (a hypothetical with little supporting evidence!), they would still sometimes be wrong. Only the evidence—the process by which the data came to be observed, not the investigator’s fame, job, status, or income—is the stuff of serious scientific inference.

Epstein & King, supra note 1, at 34.

15 This argument has been discussed in some depth by Rubin, supra note 4, at 894–902, 963. From a somewhat different perspective:

Because the criteria are under-articulated, the test may in effect become one of familiarity. If we cannot say exactly what good scholarship is, we may use, as an implicit paradigm, something we wrote, or tried to write, or wished we could write. Work that is different than anything we can imagine wanting to write will not qualify.

Coombs, supra note 2, at 707.

16 They are, for instance, used to decide questions of lateral hiring and—more crucially—tenure, when a judgement must be passed on the performance of a faculty, department, research group, program, or institute, or to decide about the distribution of research funds.

17 Carter argues:

At least it is a test that would put all arguments on the same footing; if one wants to say that this article by Scholar Black is better than that article by Scholar White, one at least now has rules within which the battle can be fought, and the possibility of agreed rules is a considerable improvement over the current situation, in which there are as many versions of the rules for evaluation as there are evaluators.

Stephan L. Carter, Academic Tenure and “White Male” Standards: Some Lessons from the Patent Law, 100 Yale L.J. 2065, 2084–85 (1991). I remark that this idea has also been refuted within the literature. See Pierre Schlag, Pre-Figuration and Evaluation, 80 Calif. L. Rev. 965 (1992) (arguing that a theory of evaluation does not free an evaluator from the bias inherent in her own pre-figurations. On the contrary, the belief that better evaluative criteria will advance the cause of fairer evaluation is itself an effect of flawed and unrationalized pre-figurations of conventional legal thought); Richard Delgado, On Telling Stories in School: A Reply to Farber and Sherry, 46 Vand. L. Rev. 665, 765 (1993) (“When standards take hold in the academy, the result may be to legitimate an unfair and biased system.”).

18 Compare Coombs, supra note 2, at 703–04; Jayne W. Barnard, Reflections on Britain’s Research Assessment Exercise, 48 J. Legal Educ. 467, 490 (1998).

19 See also Verbeke, supra note 3, at 1–2 (“Especially our young doctoral and post-doctoral researchers are in demand of more clarity and guidance about what is research and what is not.”); Edward G. White, The Text, Interpretation, and Critical Standards, 60 Tex. L. Rev. 569, 584–85 (1982) (“To characterize the evaluative standards of academics or other professional elites as radically indeterminate is therefore either to abdicate educational responsibility or to advocate an alternative approach to education that is profoundly misguided.”).

20 It should be noted here, though, that scholars have—I believe rightly—warned that when used aspirationally, criteria can only indicate what is minimally required to achieve certain quality levels. What makes a work of scholarship truly outstanding, extraordinary, or even path-breaking, is dependent on many different factors that, moreover, are largely recognizable only in retrospect. See Coombs, supra note 2, at 705; Aviam Soifer, MuSings, 37 J. Legal Educ. 20, 23 (1987).

21 See Barnard, supra note 18, at 491.

22 See Rob A. J. van Gestel & Jan B. M. Vranken, Assessing Legal Research: Sense and Nonsense of Peer Review Versus Bibliometrics and the Need for a European Approach, 12 German L.J. 901, 905 (2011).

23 See Verbeke, supra note 3, at 1–2. (“The outsiders, other sciences, do not take us serious. They wonder whether law is a science and how what is done there could be considered research.”); Roux, supra note 3, at 173; Coombs, supra note 2, at 703; Van Gestel & Vranken, supra note 22, at 906.

24 Robert Cryer et al., Research Methodologies in EU and International Law 10 (2011); Terry Hutchinson & Nigel Duncan, Defining and Describing What We Do: Doctrinal Legal Research, 17 Deakin L. Rev. 83, 84 (2012); Jan M. Smits, What Is Legal Doctrine? On the Aims and Methods of Legal-Dogmatic Research 1, 7 (Maastricht European Private Law Institute, Working Paper No. 2015/06, 2015).

25 Some examples are the reports produced by the German Wissenschaftsrat in Perspektiven Der Rechtswissenschaft in Deutschlands: Situation, Analysen, Empfehlungen (Wissenschaftsrat 2012); reports by the Belgian Vlaamse Interuniversitaire Raad in Model Voor Integralte Kwaliteitsevaluatie Van Het Onderzoek in De Rechtswetenschappen (Vlaamse Interuniversitaire Raad 2004); and reports by the Dutch Evaluatiecommissie Rechtswetenschappelijk Onderzoek, Kwaliteit & Diversiteit: Rechtswetenschappelijk Onderzoek in Nederland (Evaluatiecommissie Rechtswetenschappelijk Onderzoek 2009), but also the reviews resulting from the British Research Assessment Exercise and the Excellence in Research for Australia.

26 See generally Van Gestel & Vranken, supra note 22 (considering protocols developed in the United Kingdom, Belgium, the Netherlands, Australia, and Denmark); Evaluating Academic Legal Research in Europe: The Advantage of Lagging Behind (Rob A. J. van Gestel & Andreas Lienhard eds., forthcoming Apr. 2019) (providing an overview of assessment practices in twelve European countries).

27 See generally Van Gestel & Lienhard, supra note 26 (Europe); Jian Zhang, Evaluating Chinese Legal Scholarship in Journals: Are Journal Rankings and Elite Law Journals Representing the Best Quality, https://ssrn.com/abstract=2706585 (Dec. 15, 2015) (China); Epstein & King, supra note 1, at 45–47 (United States).

28 Van Gestel & Vranken, supra note 22, at 908–11.

29 Although debates within the discipline of law have mainly revolved around the means by which other scholars can determine a scholarly publication’s quality—a large body of literature addresses the “sense and nonsense” of peer review, bibliometrics, journal rankings, and a wide variety of other means that could be helpful in getting some grip on the quality of a legal academic’s contribution—there are some that have tried to define more substantive quality standards for evaluating traditional legal scholarship. Without pretending to be exhaustive, I provide a brief overview of some of the more elaborate and thoughtful accounts I came across.

30 Also remarked by Terrill Pollman & Linda H. Edwards, Scholarship by Legal Writing Professors: New Voices in the Legal Academy, 11 Legal Writing: J. Legal Writing Inst. 3, 46 (2005). See also Erwin Chemerinsky & Catherine Fisk, In Defense of the Big Tent: The Importance of Recognizing the Many Audiences for Legal Scholarship, 34 Tulsa L.J. 667, 676 (1999).

31 Rubin states:

Sometimes, this can be accomplished in a sentence or a paragraph. A constitutional argument might be based on the norm that we should interpret the Constitution as an evolving system of structural relationship; a common law argument might begin by declaring that the law should be predictable by private parties. In other cases, stating one’s normative starting point might require a more elaborate argument.

Rubin, supra note 4, at 916.

32 Rubin argues:

For purposes of this criterion, it is unimportant whether the author chooses to justify his starting premise or merely identify it, whether he engages other scholars who adopt different premises or ignores those scholars and proceeds with his own discussion. These issues will become relevant for other criteria, but with respect to normative clarity, the crucial question is whether the reader can engage the author.

Id.

33 Rubin states:

For example, the scholar might argue that the policy of avoiding accidents can be implemented by holding manufacturers strictly liable for injuries caused by their products, or that the First Amendment should be interpreted in accordance with a principle of personal autonomy. The first argument threats a descriptive statement—accidents can be avoided through strict liability—as problematic and claims validity on the basis of the statement’s ability to implement the stated norm. The second argument presents an expressive claim; it asks the reader to recognize, as a matter of personal experience, that particular way of reading the Constitution is authentic…. Given the pragmatic quality of legal scholarship (not truth, positivism), the author is entitled to assume the premise, as long as it is clearly stated, and proceed to consider means for its implementation. The real test of persuasiveness, then, is whether readers would be convinced if they accepted the initial premise. In other words, if we wanted to avoid accidents, would we impose strict liability on manufactures to implement that policy? If we agree that personal autonomy is a valid norm, can the First Amendment be authentically read to achieve it?

Id. at 920.

34 Id. at 923. According to Rubin, “[t]his judgement is made according to standards that the interpreters would recognize as valid ways of judging what they themselves mean when they write.”

35 Id. at 923–24. As an example, Rubin refers to a study in which the authors tried to demonstrate that a right to privacy is already ensconced in the common law—while that was simply not the case—rather than devoting their argument to arguing for the attractiveness of such a right.

36 Rubin notes that:

As a practical matter, significance may seem to be a difficult criterion to use for evaluative purposes because it requires knowledge of the work’s subsequent history …[b]ut the question of significance is not limited to subsequent effect; it also concerns the work’s relationship to the developing historical context at the time it is written.

Id. at 934.

37 Id. at 937.

38 Kissam, supra note 6, at 228.

39 Id.

40 Id. at 228–29. Thus, similar to Rubin’s criteria of significance and applicability.

41 As clearly shown by Van Gestel & Lienhard, supra note 26.

42 Verbeke, supra note 3.

43 Id. at 5. As he states it, “what is analytical … will also be critical.”

44 Jan B. M. Vranken, Mr. C. Asser’s Handleiding Tot De Beoefening Van Het Nederlands Burgerlijk Recht: Algemeen Deel: Een Synthese (2014). For an English summary, see Jan B. M. Vranken, Methodology in Legal Research and Legal Practice: A Comparison of Judge and Legal Academic, in Recht in Geding II (Marc S. Groenhuijsen, Ewoud Hoondius & Arend Soeteman eds., 2016).

45 Jan B. M. Vranken, Mr. C. Asser’s Handleiding Tot De Beoefening Van Het Nederlands Burgerlijk Recht: Algemeen Deel: Een Synthese (2014) at 50–51. According to Vranken, that newness may be a contribution to theory, or—more modest—adding a new aspect to the discussion in the literature, pointing to the fact that a particular decision of the Supreme Court should be interpreted differently than the scholarly community generally does, or a comparison with lower court decisions or other jurisdictions.

46 Id. at 55–56.

47 Id. at 56–57.

48 Id. at 107–45.

49 Meaning that the data the scholar used is not clandestinely massaged in a particular direction; the composition of the answer must be “imitable,” requiring the scholar to explicate and justify the norms or perspectives that may have consciously or unconsciously influenced the way in which the problem definition is answered.

50 Id. at 145–50. According to Vranken, there may exist conflict between the standards of persuasiveness and readability and the other standards, which means that a balance must be sought between the different criteria.

51 Moreover, they are not without criticism. The standards developed by Rubin have been criticized by Schlag, supra note 17, and Levit, supra note 8, at 953–55. The standards developed by Vranken were critiqued by Willem H. Van Boom, Kritiek En Methode in Juridisch-Dogmatisch Privaatrechtelijk Onderzoek—Een Bespreking Van Asser-Vranken Algemeen Deel****, 32 Nederlands Tijdschrift voor Burgerlijk Recht 123 (2015). What personally struck me is that the different contributions to the debate on quality standards almost appear in isolation. I hardly found any cross-referencing among these and other—less elaborate and therefore undiscussed—contributions.

52 See generally Ginevra Peruginelli, Evaluating Research: The Case of Legal Scholarly Outputs, 15 Legal Info. Mgmt. 50 (2015); Sven E. Hug & Michael Ochsner, How Legal Scholars Could Change the Landscape of Research Evaluation, in L’evaluation De La Recherche En Droit: Enjeux Et Méthodes (Thierry Tanquerel & Alexander Flückiger eds., 2015); Wolfgang Kaltenbrunner & Sarah De Rijcke, Quantifying ‘Output’ for Evaluation: Administrative Knowledge Politics and Changing Epistemic Cultures in Dutch Law Faculties, 44 Sci. & Pub. Pol’y 284 (2016).

53 I chose to focus on law professors as this group of scholars is generally the most experienced, regularly involved in the evaluation of scholarly work—as peer reviewer, editor, or supervisor—and to a certain extent defining the standards of a disciplines because of their function as exemplar. See Thomas Kuhn, The Structure of Scientific Revolutions 187 (3d ed. 1996); Sébastien Pimont, Á Propos De L’activité Doctrinale Civiliste (Quelques Questions Dans L’air Du Temps), Revue Trimestrielle de droit civil 707 (2006); Coombs, supra note 2, at 711. The sample was composed by taking into consideration variations in sociodemographic characteristics such as age, experience, gender, research institutes, and areas of the law—private law, administrative and constitutional law, criminal law, and international law. To assure that the choice to consult Dutch scholars does not result in findings that may only be relevant to Dutch legal scholarship, I made sure that all of the respondents publish(ed) internationally, act(ed) as peer reviewer or editor for international or European journals, and/or were part of dissertation committees in other countries.

54 I decided to use this qualitative approach because prior research conducted within other scientific disciplines revealed that empirical investigations with qualitative approaches towards research habits can yield rich and useful data. See generally Barbara Blummer, Sara L. Watulak & Jeffrey Kenton, The Research Experience for Education Graduate Students: A Phenomenographic Study, 17 Internet Reference Serv. 117 (2012); Rosemary Green & Peter Macauley, Doctorol Students’ Engagement with Information: An American-Australian Perspective, 7 Portal 317 (2007); Vanessa J. Earp, Information Source Preferences of Education Graduate Students, 27 Behav. & Soc. Sci. Libr. 73 (2008); Hug & Ochsner, supra note 52, at 361–62. Some legal scholars have also specifically called for a qualitative approach in order to arrive at a more nuanced and deepened understanding of this particular research activity. See Christopher G. Wren and Jill R. Wren, Reviving Legal Research: A Reply to Berring and Vanden Heuvel, 82 Law Libr. J. 463, 466 (1990); Stephanie Davidson, Way Beyond Legal Research: Understanding the Research Habits of Legal Scholars, 102 Law Libr. J. 561, 570 (2010). Further information on the methods and techniques followed—on sampling, the way in which the semi-structured interviews were setup, the way in which the data was coded, and so on—can be requested from the author.

55 One of the respondents stated:

If I have to evaluate a journal article, the first thing I do is see whether a clear problem definition is provided. Sometimes you might question the content of the problem definition, but the fact that a journal article possesses a problem definition alone is—unfortunately—already quite something within contemporary traditional legal scholarship.

56 Suppose a researcher intends to answer the question: “Is or should obligatory supervision of adolescents that are at risk of derailing because of psychological problems (be) legally legitimized?” Without a description of what the indeterminate concepts of adolescents, derailing, psychological problems, and obligatory supervision mean, this question is not answerable. See Jan B. M. Vranken, Mr. C. Asser’s Handleiding Tot De Beoefening Van Het Nederlands Burgerlijk Recht: Algemeen Deel: Een Synthese (2014) at 56–57.

57 See Carter, supra note 17, at 2080: “The standard that work must be new is simply a matter of what the law reviews sometimes call preemption: Did the writer think of this idea first or is it already in the literature?”

One must further notice a double-edged sword at work, with originality caught between preemption and citation. An article must simultaneously be well supported and not preempted … An insistence on knowing where we have come from will make it more likely that we and our readers will know when we are saying something new.

Note, Originality, 115 Harv. L. Rev. 1989, 2007 (2002). For a somewhat different view:

Our main insight was that doctrinal research does not proceed on the back of a research question in the traditional sense, i.e. a question aimed at filling a gap in a defined body of scholarly knowledge. Instead, doctrinal research is directed at addressing an alleged lack of coherence, disputed issue of application or normative shortcoming in a defined area of law. While there are similarities between the two types of research question, the body of knowledge in the case of doctrinal research is legal doctrine itself.

Roux, supra note 3, at 186.

58 Or, in the words of Epstein and King:

(1) asking a question that the legal community may view as important but that no other scholars has tackled; (2) attempting to settle a question that has evoked conflicting responses; (3) raising an “old” question but addressing it in a unique way; (4) collecting new data on the same observable implications or different implications altogether; or (5) applying better methods to reanalyze existing data.

Epstein & King, supra note 1, at 59. The most elaborate publication I encountered on being original within legal scholarship, although taking a somewhat broader perspective, is the contribution of Siems, proposing several techniques to achieve originality. Mathias M. Siems, Legal Originality, 28 Oxford J. Legal Stud. 147 (2008).

59 Meeting this criterion is usually not problematic within legal scholarship. See Epstein & King, supra note 1, at 60 (“Our survey of law review articles suggest that it is the rare piece of legal scholarship that does not pose a question that has at least a potential implication—normative, policy, or otherwise—for the real word.”).

60 The minimal level of scientific significance will be met when the work belongs to the ongoing development of the field in some fashion. The highest levels of significance are met when a contribution, for instance, helps rethink contract law. In between, we could place a work that, for instance, develops a new theory of damages within contract law. Rubin provides several—quite random—examples of articles that he finds to meet the significance criterion. Rubin, supra note 4, at 930–35.

61 Such a preliminary inquiry is referred to in other academic disciplines as a “systematic literature review.” See generally Helen Aveyard, Doing a Literature Review in Health and Social Care: A Practical Guide (Open University Press 3d. rev. ed. 2010); Chris Hart, Doing a Literature Search: A Comprehensive Guide for the Social Sciences (2001); Arlene Fink, Conducting Research Literature Review: From the Internet to Paper (3d ed. 2010). Epstein and King highlighted another advantage of engaging existing scholarship: It “ensures that someone will be interested in the results,” as “if a body of literature—however slim and underdeveloped it may be—is on hand, it indicates that the question is important to at least some others.” Epstein & King, supra note 1, at 59. For the exact techniques that could be followed, see Marnix V. R. Snel & Janaína G. Garcia de Moraes, Doing a Systematic Literature Review in Legal Scholarship (2018).

62 As has been remarked by many before, these criteria are far from always explicated in legal publications. See Rubin, supra note 4, at 917; Roux, supra note 3. Some respondents even stated that, because of the often-absent frameworks, “the normative part [of] a traditional legal research project is often the least scientific part,” or at least “a vulnerable section.”

63 Evidently, the researcher can also opt for other research frameworks, like a more general framework construed on the basis of the doctrine of tort law, or a framework on the basis of one or more legal principles that dominate the field of civil and/or public law. According to a few respondents, the best of traditional legal scholarship combines all these different frameworks in one single publication, creating a “matrix-like presentation of different insights, recommendations, or solutions.”

64 While some respondents mentioned that, once a scholar has a research question, it may be constructive to begin theorizing—developing a reasoned and precise speculation about the answer to a research question—about possible answers that she can use to generate observable implications—things that we would expect to detect in the real world if our theory is right—others did not in any way refer to the importance hereof. With the few respondents who mentioned such a criterion and Epstein and King, I agree that the importance of theorizing is underestimated within traditional legal scholarship. For more detail on what this criterion may mean, see Epstein & King, supra note 1, at 61–80.

65 That does not, however, mean that every single sentence needs substantiation. See Note, supra note 57, at 2006 (“In other academic journals, by contrast, many facts are considered sufficiently well known to need no citation. But law reviews give citations for everything under the sun, as if every proposition, and some things that are not propositional at all, needed ‘authority.’”).

66 Within the legal literature, three different functions have been assigned to the employment of footnotes: (1) A positioning function—to position the research within its context and to suggest further readings on a particular issue; (2) a legitimizing function—to indicate the authority of the assertion made; and (3) a control function—informing the reader about the information on which assertions are made. See Elizabeth Fajans and Mary R. Falk, Scholarly Writing for Law Students: Seminar Papers, Law Review Notes and Law Review Competition Papers 99 (4th ed. 2011). For publications in other academic disciplines on the functions of footnotes, and their perverse effects, see generally Robert N. Broadus, An Investigation of the Validity of Bibliographic Citations, 34 J. Am. Soc’y Info. Sci. 132 (1983); Terrance A. Brooks, Evidence of Complex Citer Motivations, 37 J. Am. Soc’y Info. Sci. 34 (1986).

67 While the importance of precise and correct referencing might sound obvious, it causes problems in research practice. See Jeffrey L. Harrison & Amy R. Mashburn, Citations, Justifications, and the Troubled State of Legal Scholarship: An Empirical Study (University of Florida Levin College of Law Legal Studies Research Paper Series 2015).

68 The respondents argued that choices made that lie in the peripheral areas of an investigation require less elaborate justification than choices made that relate to the main line of reasoning followed by the author.

69 Epstein and King also write: “In many, perhaps most, instances, legal academics conducting these sorts of investigations rarely provide even a tracing of how they collected the evidence.” Epstein & King, supra note 1, at 42–45.

We rarely learn: (1) How authors canvassed the relevant case law and what precisely was the population from which they sampled; (2) How authors selected their cases and how many they read; (3) How authors distinguished “key” or “a few … exemplary cases” from those that are not central or not typical.

Id.

Law scholars may be fastidious about documenting textual sources of information via the omnipresent footnote, but they are not particularly attentive to the need to document their data procedures, nor, we hasten to note, have they established procedures for ensuring the requisite attention to or repositories (private or public) for their data.

Id.

70 See also Hutchinson & Duncan, supra note 24, at 98–99.

71 In this respect, Emden and Sandelowski argued for “the potential value of including among one’s criteria for goodness a ‘criterion of uncertainty’; that is, an open acknowledgement that claims about our research outcomes are at best tentative and that there may indeed be no way of showing otherwise.” Carolyn Emden & Margarete Sandelowski, The Good, the Bad and the Relative, Part Two: Goodness and the Criterion Problem in Qualitative Research, 5 Int’l J. Nursing Prac. 1, 5 (1999).

72 Epstein & King, supra note 1, at 208.

73 Epstein and King have a somewhat different view:

It may be that, after amassing the evidence for which the design calls, the scholar finds an imperfect fit among it, the main research question, and the theory. Rather than erasing months or even years of work, the investigator certainly should return to the drawing board, design more appropriate procedures, or even recast the original research question.

Epstein & King, supra note 1, at 54.

74 Questions that may be asked in this respect are: Whether the structure of the argumentation is comprehensible, whether the author takes the reader by the hand, or whether the reader is dragged along with the thoughts expressed by the author.

75 For example, new technological developments may bring to light novel questions of liability for harm caused, or the implications of the introduction of a partly new statutory framework for current legal practice.

76 In this context Roux stated:

The best doctrinal researchers are able to draw on a vast field of reference, and are familiar with a lot of positive law (both local and foreign) that potentially has a bearing on the questions they are addressing. In this way, they are able to see system-wide implications of new legal developments that less experienced researchers may miss. They are also able to resolve questions arising in ways that promote greater coherence in the affected body of law.

Roux, supra note 3, at 188.

77 The reader might live under the impression that it should not be too difficult to comply with the principle of accuracy. I found, however, sound evidence—both in the literature and from examples provided by the respondents—that even the most experienced traditional legal scholars make errors in this respect. For an empirical investigation, see generally Rob A. J. van Gestel, et al., Rechtswetenschappelijke Artikelen: Naar Criteria Voor Methodologische Verantwoording, 82 Nederlands Juristenblad 1448 (2007).

78 Within the empirical data, views somewhat differed on what qualifies as a limited or focused enough problem definition to be able to consult all relevant materials.

79 For instance, one of the respondents with an international and European law expertise argued that “any inquiry towards the European law faces the challenge that there are twenty-seven member states where that law applies. That means that there are probably twenty-seven countries in which relevant literature commentaries, case law, and legislation exist.”

80 The consulted material thus needs not only to be relevant to your inquiry, but also “functional.”

81 Achieving completeness and a thorough consideration of alternative interpretations, arguments, and solution is, however—self-evidently—not imperative for every claim made within a doctrinal legal contribution. That is both impossible and would substantively detract from the readability of a publication. In general, “being complete, balanced and honest is more crucial with regard to the main line of reasoning within an inquiry than in relation to its peripheral components.” For instance, that means that, with respect to the main line of reasoning, the easy way out—for example, only providing a footnote that says “for a different view, see author X”—is not considered sufficient by the respondents.

82 A scholar that searches for reasons to extend constitutional rights, to incorporate more safeguards for the victim into criminal procedures, or to intensify regulation to remedy environmental pollution may most certainly find arguments and evidence that such might indeed be necessary. Haack has labelled such research practice as “fake inquiry”; the scholar is not concerned with finding how things really are, but to advance herself by making a case for some proposition to the truth-value of which she is indifferent. Susan Haack, Manifesto of a Passionate Moderate: Unfashionable Essays 8–10 (1999).

83 See also Roux, supra note 3, at 188 (“In pure doctrinal research, however, the style is permissible—indeed it is desirable—provided that opposing arguments are giving their due.”). One of the respondents provided clarification by presenting a hypothetical example:

Suppose there exist two contradictory points of view within the literature, X and Y. The researcher may either chose to aim her research at the identification of different viewpoints regarding a certain topic or to defend a particular position. If the researcher opts for the former, she has to make sure that all viewpoints are discussed on an equal footing. If the researcher opts for the latter, she may attach more value to viewpoint X, as long as she also pays sufficient attention to viewpoint Y, motivating why viewpoint X is to be preferred over viewpoint Y.

84 Often referred to as “validity” in other academic disciplines. In relation to qualitative interpretative research, the validity criterion is often subdivided in “descriptive validity,” “interpretative validity,” “evaluative validity,” “theoretical validity,” and “generalizability.” See generally Joseph A. Maxwell, Understanding and Validity in Qualitative Research, 62 Harv. Educ. Rev. 279 (1992). Comparable concepts were encountered sporadically within the legal literature. See Hutchinson & Duncan, supra note 24; Charles Tremper, Sue Thomas & Alexander C. Wagenaar, Measuring Law for Evaluation Research, 34 Evaluation Rev. 242 (2010).

85 See also Epstein & King, supra note 1, at 3 (“There is a good deal of description in most legal scholarship, and this can be evaluated in terms of truth.”).

86 Compare Bardley Wendel, Explanation in Legal Scholarship: The Inferential Structure of Doctrinal Legal Analysis, 96 Cornell L. Rev. 101, 105 (2011). Because of the textuality of its sources, descriptive validity is less a concern within legal scholarship than it may be in disciplines where sources are non-textual. Where a researcher observing what happens in a classroom needs to make sure that what happened in that classroom is correctly described, the correctness of what happened in, for instance, a judicial opinion is achieved more easily as it is already written down. Still, achieving descriptive validity, even though not hard, remains highly important.

87 For the debate between Mazzarese and Niiniluoto, see Tecla Mazzarese, Norm-Propositions: Epistemic and Semantic Queries, 22 Rechtstheorie 39 (1991); Ilkka Niiniluoto, Norm-Propositions Defended, 4 Ratio Juris 367 (1991).

88 An often-cited publication that addresses these issues is the article of William W. Fisher III, Texts and Contexts: The Application to American Legal History of the Methodologies of Intellectual History, 49 Stan. L. Rev. 1065 (1997).

89 In line with Rubin, supra note 4, at 923–24. The issue here is not the appropriateness of the concept of the account forwarded in the interpretation, but its accuracy as applied to the perspective of the person or institution included in the account, referred to as “authenticity” by Sally Thorne, The Art (and Science) of Critiquing Qualitative Research, in Completing a Qualitative Project: Details and Dialogue (Janice M. Morse ed., 1997); Robin Whittemore, Susan K. Chase & Carol L. Mandle, Validity in Qualitative Research, 11 Qualitative Health Res. 533 (2001); Margarete Sandelowski, The Problem of Rigor in Qualitative Research, 8 Advances in Nursing Sci. 27 (1986).

90 See also Roux, supra note 3, at 188 (“A statement of the law that appears to a person familiar with the area of law concerned to be more persuasive and compelling than the other contending interpretations.”).

91 That does not mean that the researcher always has to add much more information. From the wording of an interpretation, it is often quite clear that the researcher, for instance, applied a teleological or historical approach.

92 If a legal scholar, for example, intends to discover a line in the Supreme Court’s case law with regard to a particular aspect of the duty of care in vendor-buyer relationships and the scholar therefore selects several exemplary or key cases in that area, generalizability would of course also be concerned with the question whether the selected cases were indeed exemplary or key.

93 Maxwell, supra note 84, at 295.

94 Id. at 291–92.

95 See supra note 63.

96 Compare Epstein & King, supra note 1, at 87–88.

97 See also Epstein & King, supra note 1, at 77–80.

98 Such requires that—to a certain extent—alternative explanations for a causal association are systematically tested and dismissed. The greater the number of relevant alternative variables considered and eliminated, the greater the confidence that the relationship is causal. Preferably, the causal relation established is also grounded in existing knowledge. In the legal example above, for instance, the claim made may be substantiated by the fact that judicial opinions of the European Court in Human Rights result in valid high Dutch law and therefore need to be followed by the Dutch Supreme Court. One should, however, be aware that “no matter how perfect the research design, no matter how much data we collect, and no matter how much time, effort, and research resources we expend, we will never be able to make causal inferences with certainty.” Epstein & King, supra note 1, at 37.

99 One of the respondents, for instance, argued that “if you manage to word the matter you are investigating in an exceptional manner, your publication is probably rated better than if you do not.” That such is not always the case is made very clear by Fred Rodell, Goodbye to Law Reviews, 1 J. Contemp. L. 4 (1936).

100 One of the respondents said that “a publication becomes more convincing if it only addresses real arguments and omits the incorporation of non-feasible or unrealistic arguments.” “The fact that argumentation is important in (especially) normative inquiry,” according to one of the respondents, “makes the estimation of a publication’s quality difficult”; “the ordering of arguments in a sense always has a subjective element in it.”

101 For instance, if I intrinsically believe that consumer protection is much more important than complete autonomy of contracting parties, that would—or at least could—presumably influence my research project that addresses the doctrine of contract termination.

102 Also proposed within the literature, see Michael Pendleton, Non-Empirical Discovery in Legal Scholarship—Choosing, Researching and Writing a Traditional Scholarly Article, in Research Methods for Law 164–65 (Mike McConville & Wing H. Chui eds., 2007); Rob A. J. Van Gestel, Hans W. Micklitz & Miguel P. Maduro, Methodology in the New Legal World 1, 23 (European University Institute, Working Paper No. EUI LAW 2012/13, 2012); Roux, supra note 3, at 189–90. Theory of knowledge acknowledges the effect of a researcher’s position and perspectives on any research endeavor, and disputes the belief of neutrality. It is the always limited perspective of the researcher that determines the particularities of a research process. See generally Kirsti Malterud, Qualitative Research: Standards, Challenges, and Guidelines, 358 Lancet 483 (2001); Thomas Nagel, The View from Nowhere (1986); Ludwig Wittgenstein, On Certainty (1969). Absolute objectivity probably cannot be achieved. If I understood the respondents correctly, trying to achieve objectivity can only mean to recognize that knowledge is indeed partial and situated, and to account adequately for the effects of the positioned researcher; bias should thus be accounted for, though not completely eliminated, and subjectivity arises when the effect of the researcher is ignored.

103 Compare Epstein & King, supra note 1, at 42.

Figure 0

Table 1: Overview of Quality Standards