Hostname: page-component-848d4c4894-jbqgn Total loading time: 0 Render date: 2024-06-21T18:56:10.814Z Has data issue: false hasContentIssue false

Assessing Legal Research: Sense and Nonsense of Peer Review versus Bibliometrics and the Need for a European Approach

Published online by Cambridge University Press:  06 March 2019


Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

There is no uniform approach to assessing scholarly publications in legal research. Peer review is still the most accepted form, but the popularity of bibliometrics, such as the impact scores of journals and the citation scores of individual articles is increasing. However, there is no ranking of European law journals and none is likely to materialise any time soon. Comparing the UK, Belgium and the Netherlands, we will demonstrate why both peer review and the use of metrics have serious shortcomings. We believe it is necessary to think about such alternatives as more attention for methodological justification in legal research, more clarity from editorial boards about the quality criteria being used to approve or reject submissions, and more emphasis on standards for different forms of legal scholarship. Last but not least, we call for a Europe-wide debate on the pros and cons of different systems of research assessment, rather than let every country reinvent the wheel.

Copyright © 2011 by German Law Journal GbR 


1 Paraphrased from Oscar Wilde, Lady Windermere's Fan, Act III (1892).Google Scholar

2 See Mark, Pen, Prestatiemeting van Wetenschappelink Onderzoek. Een Empirische Studie Naar De Doorwerking Van Universitaire Prestatiemeetsystemen (2009). (Performance Management of Academic Research: An Empirical Study Concerning the Impact of Academic Performance Assessment Systems. The dissertation is available full text and with an English summary, available at: (last accessed: 8 March 2011).Google Scholar

3 See Dan Jerker B. Svantesson, International Ranking of Law Journals: Can it Be Done and at What Costs? 29 Legal Studies 681 (2009). The author even argues that truly ground-breaking work may be more likely to appear from marginal, dissident or unexpected sources, rather than from well-established and entrenched mainstream publication media.Google Scholar

4 See Wooding, Steven & Grant, John, Assessing Research: The Researchers View (2003).Google Scholar

5 This even applies to high-reputation journals, such as the Modern Law Review, Law Quarterly Review, Oxford Journal of Legal Studies and the Cambridge Law Journal. None of these journals publish which content-based assessment criteria are being used in the peer review process. Nevertheless, it is widely perceived that the reputation of the journal in which research is published affects how that research is assessed in the national research assessment exercise. See Kevin Campbell et al, 26 Journal of Law and Society 470, 349-350 (1999).Google Scholar

6 By transparent we mean, among other things, that choice for a certain form of peer review by law journals is often unclear: Why do some prefer (double) blind reviews, while others favor that both the name of the authors and the reviewer are made public, and still others entrust the editorial board itself with the review of articles?Google Scholar

7 Type refers both to the kind of research, such as doctrinal, empirical or comparative research, and to the style in which the research is presented. Legal research can be formulated as an opinion, an article, an essay, a book, a case note etc.Google Scholar

8 Dissertations, monographs, articles, working papers, case notes, opinions, and such, in print or in an electronic version.Google Scholar

9 Obviously, language is an important factor when deciding for whom one is writing, and for the scope of the scholarly debate.Google Scholar

10 We deliberately confine ourselves to the situation in Europe because a comparison with, for example, the situation in the United States (US) would not only require far more elaborate research, but would also complicate the discussion due to, among other things, the existence of student edited journals, the divide between the faculties of a few top law schools and the rest in terms of the distance between legal scholarship and legal practice, and the far greater dominance of law and economics and empirical legal research if compared to most European countries.Google Scholar

11 Key Perspectives Ltd., A Comparative Review of Research Assessment Regimes in Five Countries and the Role of Libraries in the Research Assessment Process, 8 (2009), available at: (last accessed: 3 January 2011).Google Scholar

12 Osterloh, Margit & Frey, Bruno S., Research Governance in Academia: Are There Alternatives to Academic Rankings? 3 CESIFO Working Paper Series No. 2797 4 (2009), available at: (last accessed: 3 January 2011).Google Scholar

13 For example, even as early as 2001 the UK Research Assessment Exercise assessed over 2,400 submissions and examined over 150,000 publications.Google Scholar

14 In other words: the criteria of originality, thoroughness and profundity apply right across this range, weakly at the bottom, and paradigmatically at the top.Google Scholar

15 Vlaamse Interuniversitaire Raad (Flemish Inter-University Council, VLIR), Model voor Integrale Kwaliteitsevaluatie van het Onderzoek in de Rechtswetenschappen (2004). (The Verbeke Committee Report 2004). Available at: (last accessed: 3 January 2011). The Model is aimed at the assessment of individual scholars, not merely their publications. See under A.I for an explanation of the difference. We concentrate on the criteria to assess individual publications.Google Scholar

16 The Stolker Committee notes that indicators that are extensively used in other disciplines, such as a ranking of journals and citation analysis, are almost completely absent in (Dutch) legal scholarship.Google Scholar

17 Published by the Association of Universities in the Netherlands (VSNU), but no longer available through their website. On file with the authors.Google Scholar

18 Stolker Committee Report, 30 (2005); Smits Committee Report, 12 (2007). No longer available online. On file with the authors.Google Scholar

19 Supra, note 14.Google Scholar

20 An example of the latter is revisiting jurisprudence 5 or 10 years later, to see whether the conclusions of an earlier investigation are still valid. Should this not be the case (wholly), the most interesting theoretical question becomes, ‘Why not?'Google Scholar

21 See Tushnet, Mark, Weak Courts, Strong Rights: Judicial Review and Social Welfare Rights in Comparative Constitutional Law (2008) for a clear example of the former. In this book, Tushnet argues that the US has few constitutional guarantees of social welfare rights such as income, housing, or healthcare. Legal scholars usually believe that the courts cannot possibly enforce such guarantees. By taking a comparative perspective Tushnet turns against this common belief and suggests that such rights can be judicially enforced, not by strengthening the power of the courts (stronger review), but by doing just the opposite, notably creating weaker forms of judicial review where the legislature and judiciary have more room to interact.Google Scholar

22 See, for example, the following special issue: 2 International Journal of Law in Context 217 (2006), and further on law and neuroscience.Google Scholar

23 Hence the classification in the UK RAE and the five points scale in the Netherlands.Google Scholar

24 The Standard Evaluation Protocol (SEP) for Public Research Organizations of VSNU, NWO, and KNAW 9 (2003-2009), for example, describes quality as: ‘international recognition and innovative potential'. See also the above mentioned UK RAE-criteria, and the international benchmark study conducted by Aldo Geuna & Ben R. Martin, University Research Evaluation and Funding: An International Comparison, 41 MINERVA 277 (2003). See also, Expert Advisory Group for an RQF, The Research Quality Framework: Assessing the Quality and Impact of Research in Australia 11 (2005), available at: issuespaper.pdf (last accessed: 3 January 2011).Google Scholar

25 Siems, Mathias M., Legal Originality, 28 Oxford Journal of Legal Studies 174 (2008). He distinguishes between micro legal research, macro legal research, interdisciplinary research, and research on non-legal topics. Interdisciplinary research can in turn be subdivided into many forms. See also Bart van Klink & Sanne Taekema, Dwarsverbanden. Interdisciplinair onderzoek in de rechtswetenschap, 84 Nederlands Juristenblad 2563 (2009).Google Scholar

26 See for example Rhode, Deborah L., Legal Scholarship, 115 Harvard Law Review 1327, 1340 (2002), who writes: ‘Yet too much conventional legal analysis is not done well. It exhaustively exhumes unimportant topics or replicates familiar arguments on important ones. Too little effort is made to connect law to life by assessing the real world consequences of analytic frameworks. Of course, to do so in systematic fashion would require significant time, money, and expertise, which is precisely what most authors of doctrinal works are happy to avoid. The result is that, on many key legal issues, we are glutted with theory and starved for facts.’ See also Gerrit A. de Geest, Hoe maken we van de rechtswetenschap een volwaardige wetenschap? (How to transform the legal discipline into a real science?) 79 Nederlands Juristenblad 58 (2004).Google Scholar

27 See Vick, Douglas W., Interdisciplinarity and the Discipline of Law, 31 Oxford Journal of Legal Studies 163, 164 (2004), who claims that: ‘Many interdisciplinarians perceive doctrinalists to be intellectually rigid, inflexible, and inward-looking; many doctrinalists regard interdisciplinary research as amateurish dabbling with theories and methods the researchers do not fully understand.’ Some scholars argue that the true nature of legal research lies in the normative character, which implies that legal scholarship is all about how the law should read. For a clear example, see Jan M. Smits, Redefining Normative Legal Science: Towards an Argumentative Discipline, in Methods of Human Rights Research 45 (Fons Coomans ed., 2009). Others do not go so far but do criticize much of the interdisciplinary legal scholarship. See Neil Duxbury, A Century of Legal studies, in The Oxford Handbook of Legal Studies 950, 957 (Peter Cane & Mark Tushnet eds., 2003) referring to ‘cross-disciplinary chutzpah.'Google Scholar

28 An extensive description of doctrinal legal research can be found in Michael Salter and Julie Mason, Writing Law Dissertations (2007). See also, Fiona Cownie, Legal Academics: Culture and Identities (2004).Google Scholar

29 See Pen (note 2), 3.Google Scholar

30 For instance, when the research aims at increasing the practical functionality of (a part of) the legal system (making it more effective and efficient), but when its conclusions are based on nothing more than dogmatic interpretation of sources, making it impossible to answer the research question.Google Scholar

31 Smits Committee Report 20, 27 (2007), supra, note 18. The book that the chairman of the committee, Jan Smits, later wrote, Jan M. Smits, Omstreden Rechtswetenschap (Contentious Legal Scolarship) (2009), shows how little he expects from methodological requirements.Google Scholar

32 The Latin root of profundity means depth.Google Scholar

33 Verbeke Committee Report (2004), supra, note 15, cited in the Stolker Committee Report 30 (2005), supra, note 18Google Scholar

34 A good example is the renowned article by Hardin, Garrett, The Tragedy of the Commons, 162 Science 1243, (1968), which comprises a mere 5 pages and 21 footnotes. Regardless, it is one of the most important articles in the field of (environmental) law & economics. Hardin's article serves as inspiration for, among others, the Nobel Prize-winning research by Elinor Ostrom, Governing the Commons: The Evolution of Institutions for Collective Action (1990).Google Scholar

35 Stolker Committee Report 31 (2005), supra, note 18, mentioned 4,000 words, but only as a general and flexible guideline.Google Scholar

36 Inspired by the former British popular science journal of the same name.Google Scholar

37 See Siems (note 25).Google Scholar

38 The quality of research of this kind is the subject of much concern in the US as well. Compare among others, Richard Posner, Legal Scholarship Today, 115 Harvard Law Review 1314, 1314-1326, 1323-1326 (2002), and the harsh criticisms leveled throughout their paper by Lee Epstein & Gary King, The Rules of Inference, 69 Chicago Law Review 1 (2002).Google Scholar

39 Amongst others, the Smits Committee Report 32 (2007), supra, note 18. Google Scholar

40 Bruno de Witte, European Union Law: A Unified ‘Academic Discipline?, 34 EUI Working Papers RSCAS (2008), available at: (last accessed: 3 January 2011).Google Scholar

41 For more extensive information, see Ron Johnston, On Structuring Subjective Judgements: Originality, Significance and Rigour, 62 Higher Education Quarterly 120, 120-147, 132-133 (2008).Google Scholar

42 Society of Legal Scholars, AHRB Journal Reference List (2005), available at: (last accessed: 3 January 2011).Google Scholar

43 A complete reprint can be found online: Lon L. Fuller, The Case of the Speluncean Explorers, 62 Harvard Law Review 4 (1949), available at: (last accessed: 8 March 2011).Google Scholar

44 The latter reproach is aimed at American legal scholars in, for example, Pierre Schlag, Spam Jurisprudence, Air Law, and the Rank Anxiety of Nothing Happening, 97 Georgetown Law Journal, 803, Please insert the specific page no. of this citation, if available (2009). Even in the US, where ‘law &…’ research at top law schools now enjoys a higher status than doctrinal research, fervent supporters of multidisciplinarity such as Richard Posner argue for a revitalization of doctrinal research. See Richard A. Posner, How Judges Think 211 (2008). More on the European context can be found in, for example, Anthony Arnull, The Americanization of EU Law Scholarship in Continuity and Change in EU Law 415 (Anthony Arnull, Piet Eeckhout &Takis Tridimas eds., 2008).Google Scholar

45 See E-Petitions, The Official Site of the Prime Minister's Office, available at: (last accessed: 3 January 2011). The first signs are that the UK government might actually change the societal impact agenda proposed for the Research Excellence Framework. See David Willetts, David Willetts delivers first keynote speech as Minister for Universities and Science, Department for Business Innovation & Skills, available at: (last accessed: 3 January 2011).Google Scholar

46 For a good overview see, Weller, Ann C., Editorial Peer Review: Its strengths and weaknesses (2001).Google Scholar

47 Research by Peter van den Besselaar & Leydesdorff, Loet, Past Performance, Peer Review, and Project Selection: A Case Study in the Social and Behavioral Sciences, 18 Research Evaluation 273, 288 (2009), showed that grants for social and behavioral sciences do not end up with the best researchers in terms of publications in journals from the Web of Science and their citations measured over the recent period.Google Scholar

48 These disadvantages or peer review are thoroughly documented. See, for example, Lamont, Michele, How Professors Think: Inside the curious world of academic judgment (2009). We also refer to the results of the ISI Peer Review Survey, which can be found online: Peer Review Survey 2009: Preliminary Findings, Sense About Science, available at: (last visited 3 January 2011); Parliamentary Office of Science and Technology, Peer Review, Postnote no. 182, 1-4 (2002), available at: (last accessed: 3 January 2011); and Johnston, , (note 41), 120-147.Google Scholar

49 Amstrong, J. Scott, Peer Review of Journals: Evidence on Quality Control, Fairness and Innovation, 3 Science and Engineering Ethics 63, 15 (1997). See also James Bradley, Pernicious Publication Practices, 18 Bulletin of the Psychonomic Society 31, 34 (1981).Google Scholar

50 More background can be found in PEN (note 2), 1-8 among others.Google Scholar

51 This is not without debate, however. See, for example, Bailey, Stephen, Reform of Higher Education Research Assessment and Funding: Response by the Society of Legal Scholars, The Society of Legal Scholars, University of Nottingham School of Law, available at: (last accessed: 8 March 2011).Google Scholar

52 Report on the pilot exercise to develop bibliometric indicators for the REF (Executive Summary, 2009), available at: (last accessed: 3 January 2011).Google Scholar

53 See The Excellence in Research for Australia (ERA) Initiative. Available at: (last accessed: 3 January 2011). The ERA scheme also requires as one of its components international journal ranking. Svantesson (note 3), 680 is very critical about the linkage between assessing research quality and ranking.Google Scholar

54 Forfás/HEA, Research strengths in Ireland: A bibliometric study of the public research base (2009). Available at: (last accessed: 3 January 2011). The report tracked publication and citation levels for Ireland's higher education institutions in the ten-year period to 2007. The analysis was based on a global database of 10,000 of the most highly cited journals in the world. All seven universities were included. During that period the number of peer-reviewed papers published per year more than doubled to approximately 5,000. The study also concluded that the overall citation impact of Irish research is improving.Google Scholar

55 Hansen, Hanne Foss, Research Evaluation: Methods, Practice and Experience, 52-60 Danish Agency for Science, Technology and Innovation (2009).Google Scholar

56 See Key Perspectives Ltd. (note 11), 37. Svantesson (note 3), 690, on the other hand, points out, as one of the specific negative consequences of international journal ranking, that smaller jurisdictions will hardly be able to have national journals classed as top ranked journals, and that this will encourage scholars to publish in foreign journals whose readers are not interested in domestic issues of the author. Moreover, international ranking will favor certain disciplines over others. For example, when many journals of one specialist area are highly ranked, research in that area is automatically higher ranked than in other areas with less top-tier journals.Google Scholar

57 See Verbeke Committee Report (2004), supra, note 15.Google Scholar

58 Gutwirth, Serge, Evaluatie rechtswetenschappelijk onderzoek, 168 Nieuw Juridisch Weekblad 674, 677 (2007).Google Scholar

59 Stolker Committee Report, 26-36 (2005), supra, note 18.Google Scholar

60 See PEN (note 2).Google Scholar

61 In the Netherlands, for example, the faculties of Tilburg University, Rotterdam University, and Amsterdam University.Google Scholar

62 Compare Abraham de Swaan, Oordeel en vooroordeel; over verantwoorde beoordeling van onderzoek (Judgement and Prejudice; Judgement and predjudice; On Responsible Research Assessment) in Macht en Verantwoordelijkheid: Essays Voor Kees Schuyt 267, 271 (Jan Willem Duyvendak et al, eds., 2007).Google Scholar

63 The Social Science Citation Index (ISI) normally uses a period of one or two years. See the KNAW report, Judging research on Its Merits 18 (2005). Available at (last accessed: 3 January 2011). This document points out that where the humanities and social sciences are concerned, this period is usually too short to properly judge the impact of a publication.Google Scholar

64 See KNAW, supra, note 63, at 13, 14.Google Scholar

65 Edwards, Harry T., The Growing Disjunction between Legal Education and the Legal Profession, 91 Michigan Law Review 34 (1992) caused quite a stir in the US.Google Scholar

66 See Pen (note 2), IX, 5-7.Google Scholar

67 Osterloh, Margit & Frey, Bruno S., Evaluitis: de ziekte van de ‘gecontroleerde’ wetenschap, (Evaluitis – The Disease of ‘planned’ science) in If You're So Smart, Why Aren't You Rich? 235, 237 (Chris Lorenz ed., 2008).Google Scholar

68 Compare the interview in NRC Magazine with the Dutch Spinoza Prize winners, who gave an acute description of how they manipulated the NWO grant system, for example by applying for grants for research proposals which had already been completed - which allowed them to give very precise descriptions of the research method and expected outcomes, and using the resulting money to finance new research. NRC Wetenschap June 4 2009.Google Scholar

69 Sastry, Tom & Bekhradnia, Bahram, Using metrics to allocate research funds: A short evaluation of alternatives to the Research, 18 (2006), available at: (last accessed: 3 January 2011).Google Scholar

70 See Pen (note 2), 165-168.Google Scholar

71 According to the KNAW Report, Kwaliteitszorg in de wetenschap 6 (2008), existing quality assurance systems are seen as too much of a burden.Google Scholar

72 Can and will the editorial board overrule peer reviewers in case of contradicting or poorly motivated reports? What happens with peer reviewers who are very critical, while afterwards the article may be published (elsewhere) with great success? Do journals work with a ‘letter to the editor’ procedure to allow for correction of peer review failures after publication?Google Scholar

73 See De Swaan (note 62), 267.Google Scholar

74 A famous example is Ludwig Wittgenstein, who was offered a position at Cambridge in 1930 even though he had not published since the twenties and failed to do so until he left again in 1947. It was discovered after his death he wrote 26 pages (on average) on average each day for of those 17 years, but those were for his Philosophical Investigations. This was, in turn, published posthumously in 1953 and is regarded as one of the seminal works of Western philosophy. Wittgenstein would probably have been refused a professorship at Cambridge under the purview of the RAE, agrees Donald Gillies in, Lessons from the History and Philosophy of Science regarding the Research Assessment Exercise, in Philosophy of Science 37 (Anthony O'Hear ed., 2007).Google Scholar

75 Boomkens, Rene, Topkitsch en slow science. Kritiek van de academische rede (Top Kitsch and Slow Science: A Critique on Academic Reason) 87 (2009).Google Scholar

76 Arguing that this is a real issue is Frankenberg, G., Stranger than Paradise: Identity & Politics in Comparative Law, 259 Utah Law Review 259 (1997).Google Scholar

77 See SMITS (note 31), 116. For the opinions of Antonin Scalia, compare Melissa A. Waters, Justice Scalia on the Use of Foreign Law in Constitutional Interpretation: Unidirectional Monologue or Co-constitutive Dialogue, 12 Tulsa Journal Comp & Int'l Law 149 (2004).Google Scholar

78 Compare, , Cass R. Sunstein & Adrian Vermeule, Is Capital Punishment Morally Required? The Relevance of Life-Life Trade-offs, U Chicago Law & Econ, OLIN Working Paper No. 239 (2005); Donohue, John & Wolfers, Justin J., The Death Penalty: No Evidence for Deterrence, The Economists’ Voice 3.5 (2006).Google Scholar

79 Among others Jan Vranken, Als een arrest niet bevalt, laten wij het gewoon weg (We leave out judicial decisions we don't like), in Ex Libris Hans Nieuwenhuis 77 (Alex Geert Castermans et al, eds., 2009).Google Scholar

80 Hesselink, Martijn, A European Legal Method? On European Private Law and Scientific Method, 15 European Law Journal 20, 22(2009).Google Scholar

81 Compare Carel J. J. Stolker, Legal Journals: In Pursuit of a More Scientific Approach, 2 EUROPEAN JOURNAL OF LEGAL EDUCATION 87, 89 (2005).Google Scholar

82 A scholarship header, such as the one in the Nederlands Juristenblad (Dutch Law Journal), would in principle need to be open for good foreign contributions. We would even go one step further by asking if every serious law journal that does not restrict itself to an audience of (national) legal practitioners should open up to contributions from foreign authors? If so, does this not imply that journals with academic ambitions should think about how language barriers can be overcome instead of leaving this entirely to the responsibility of individual authors?Google Scholar

83 Gestel, Rob van & Vranken, Jan, Legal Scholarly Papers. Naar criteria voor methodologische verantwoording (Three Methodological Requirements for Legal Research Papers), 82 Nederlands Juristenblad 1448 (2007). Available in English on file with the authors.Google Scholar

84 Mike McConville & Wing Hong Chui (eds.). Research Methods for Law (2007).Google Scholar

85 Pushing is not the same as steering. The latter means influencing the substance. Pushing is used in the sense of motivating. On the same subject and on the various functions of a proper research question, see Herve Tijssen, De Juridische Dissertatie Onder De Loep. De Verantwoording Van Methodologische Keuzes in Juridische Dissertatie (How to Justify Methodological Choices in Legal Ph.D. Dissertations) 68, 120, 143 (2009).Google Scholar

86 Lovitts, Barbara E., Making the Implicit Explicit: Creating Performance Expectations for the Dissertation (2007).Google Scholar

87 See Pen (note 2) 151, 153 among others.Google Scholar

88 We cannot go further into this here but Osterloh and Frey, for instance, argue that a stricter selection and socialization of academics who want to become legal scholars, for example by using ‘tenure track’ systems, might seriously downplay some of the negative side effects of quantitative research assessments. Margit Osterloh & Bruno S. Frey, Research Governance in Academia: are there alternatives to academic rankings? CESIFO Working Paper No. 2797 26 (2009).Google Scholar

89 Genn, Dame Hazel, Martin Partington & Sally Wheeler, Law in the Real World: Improving Our Understanding of How Law Works, Final Report and Recommendations (2006).Google Scholar

90 Tracey E.George, An Empirical Study of Empirical Legal Scholarship: The Top Law Schools, 81 Indiana Law Journal 141 (2006).Google Scholar

91 See Epstein and King (note 38).Google Scholar

92 Vranken, Jan, Exploring the Jurist's Frame of Mind (2006).Google Scholar

93 Posner, Richard A., The State of Legal Scholarship Today: A Comment on Schlag, 97 Georgetown Law Journal 854 (2009).Google Scholar

94 Posner, Richard A., How Judges Think 211 (2008).Google Scholar

95 Namely to supplement the Research Excellence Framework's (REF) process of peer review. See, Research Excellence Framework (REF), Higher Education Funding Council for England (HEFCE), available at: (last accessed: 3 January 2011).Google Scholar

96 Legal Scholarship Network, Social Science Research Network (SSRN), available at: (last accessed:3 January 2011).Google Scholar

97 Juraf, , available at: (last accessed: 8 March 2011).Google Scholar