Hostname: page-component-76fb5796d-vvkck Total loading time: 0 Render date: 2024-04-26T03:48:24.442Z Has data issue: false hasContentIssue false
  • English
  • Français

Citation Metrics for Legal Information Retrieval: Scholars and Practitioners Intertwined?

Published online by Cambridge University Press:  15 August 2022

Gineke Wiggers ,
Suzan Verberne  and
Gerrit-Jan Zwenne
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper is written by Gineke Wiggers, Suzan Verberne and Gerrit-Jan Zwenne and examines citations in legal documents in the context of bibliometric-enhanced legal information retrieval. It is suggested that users of legal information retrieval systems wish to see both scholarly and non-scholarly information, and legal information retrieval systems are developed to be used by both scholarly and non-scholarly users. Since the use of citations in building arguments plays an important role in the legal domain, bibliometric information (such as citations) is an instrument to enhance legal information retrieval systems. This paper examines, through literature and data analysis, whether a bibliometric-enhanced ranking for legal information retrieval should consider both scholarly and nonscholarly publications, and whether this ranking could serve both user groups, or whether a distinction needs to be made. Their literature analysis suggests that for legal documents, there is no strict separation between scholarly and non-scholarly documents. There is no clear mark by which the two groups can be separated, and in as far as a distinction can be made, literature shows that both scholars and practitioners (non-scholars) use both types. They perform a data analysis to analyze this finding for legal information retrieval in practice, using citation and usage data from a legal search engine in the Netherlands. They first create a method to classify legal documents as either scholarly or non-scholarly based on criteria found in the literature. We then semi- automatically analyze a set of seed documents and register by what (type of) documents they are cited. This resulted in a set of 52 cited (seed) documents and 3086 citing documents. Based on the affiliation of users of the search engine, we analyzed the relation between user group and document type. The authors’ data analysis confirms the literature analysis and shows much crosscitations between scholarly and non-scholarly documents. In addition, we find that scholarly users often open non-scholarly documents and vice versa. Our results suggest that for use in legal information retrieval systems citations in legal documents measure part of a broad scope of impact, or relevance, on the entire legal field. This means that for bibliometric-enhanced ranking in legal information retrieval, both scholarly and non-scholarly documents should be considered. The disregard by both scholarly and non-scholarly users of the distinction between scholarly and non-scholarly publications also suggests that the affiliation of the user is not likely a suitable factor to differentiate rankings on. The data in combination with literature suggests that a differentiation on user intent might be more suitable.

Keywords

bibliometricscitationslegal information retrievalNetherlands
Type
Legal Informatics
Information
Legal Information Management , Volume 22 , Issue 2 , June 2022 , pp. 88 - 103
Copyright
Copyright © The Author(s), 2022. Published by British and Irish Association of Law Librarians

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Gineke Wiggers is affiliated with Legal Intelligence as business analyst. An earlier version of this paper has been presented at the 8th International Workshop on Bibliometric-enhanced Information Retrieval (BIR 2019) at ECIR 2019 as G. Wiggers and S. Verberne (2019) Citation Metrics for Legal Information Retrieval Systems, available at http://ceur-ws.org/Vol-2345/paper4.pdf. This work expands on the previous work by an increased sample size, more thorough methodological description, analysis of the inter-rater agreement and statistical analysis of the results, a much expanded literature section, and a more substantial discussion.

References

Abendroth, RJ, ‘Rangwisseling Pandrecht Door Eigenlijke Achterstelling. Weekblad voor Privaatrecht’ (2014) Notariaat en Registratie 7029, 756–762.Google Scholar
Beel, J & Gipp, B, ‘Google Scholar's Ranking Algorithm: the Impact of Citation Counts (an Empirical Study)’ (2009) in Third International Conference on Research Challenges in Information Science (RCIS), IEEE, 439–446.Google Scholar
De Bellis, N., ‘History and Evolution of (Biblio)Metrics’ in Cronin, B & Sugimoto, CR (eds.) (2014), Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact, 23–44.Google Scholar
Bonaccorsi, et al. ‘Do Social Sciences and Humanities Behave Like Life and Hard Sciences?’ (2017) Scientometrics 112, 607–653.10.1007/s11192-017-2384-0CrossRefGoogle Scholar
Boom, WH van, Door meten tot weten’. Over rechtswetenschap als kruispunt. (Boom Juridische uitgevers, Den Haag 2015).Google Scholar
Bornmann, L, Bowman, BF, Bauer, J, Marx, W, Schier, H, Palzenberg, M, ‘Bibliometric Standards for Evaluating Research Institutes in the Natural Sciences’, in Cronin, B and Sugimoto, CR (eds), Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact, 201–223 (The MIT Press, 2014).Google Scholar
Boyack, KW, ‘Mapping Knowledge Domains: Characterizing PNAS’ (2004) Proceedings of the National Academy of Sciences of the United States of America 101, 5192–5199.10.1073/pnas.0307509100CrossRefGoogle ScholarPubMed
Brody, T, Harnad, S, ‘Earlier Web Usage Statistics as Predictors of Later Citation Impact’ (2006) Journal of the Association for Information Science and Technology 57(8), 1060–1072.Google Scholar
Cohen, J, ‘A Coefficient of Agreement for Nominal Scales’. (1960) Educational and psychological measurement 20(1), 37–46.10.1177/001316446002000104CrossRefGoogle Scholar
Cronin, B, ‘Scholars and Scripts, Spoors and Scores’ in Cronin, B, Sugimoto, CR (eds.), Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact 3–21 (The MIT Press, 2014).10.7551/mitpress/9445.001.0001CrossRefGoogle Scholar
Damen, LJA, ‘De Invloed van Bestuursrechtelijke Annotaties op de Rechtspraak’ (2016) Nederlands Tijdschrift voor Bestuursrecht 35, 280–281.Google Scholar
Garfield, E, Citation Indexing: its Theory and Application in Science, Technology, and Humanities. (New York: John Wiley & Sons, Inc., 1979).Google Scholar
Garfield, E, ‘Citation Analysis as a Tool in Journal Evaluation’ (1972) Science 178(4060), 471–479.10.1126/science.178.4060.471CrossRefGoogle ScholarPubMed
Van Gestel, R, Vranken, J, ‘Assessing Legal Research: Sense and Nonsense of Peer versus Bibliometrics and the Need for a European Approach’ (2011) German Law Journal 12(3), 901–929.10.1017/S2071832200017144CrossRefGoogle Scholar
Giménez-Toledo, E, Mañana-Rodríguez, J, Engels, TC, Ingwersen, P, Pölönen, J, Sivertsen, G, … & Zuccala, AA, ‘Taking Scholarly Books into Account: Current Developments in Five European Countries’. (2016) Scientometrics, 107(2), 685–699.10.1007/s11192-016-1886-5CrossRefGoogle Scholar
Hicks, D, ‘The Four Literatures of Social science’. in Handbook of Quantitative Science and Technology Research (Dordrecht: Springer 2004) pp. 473–496.Google Scholar
Jackson, P, Al-Kofahi, K, ‘Human Expertise and Artificial Intelligence in Legal Search’ in: Geist, A, Brunschwig, CR, Lachmayer, F, Schefbeck, G (eds.), Strukturierung der Juristischen Semantik. (Bern: Weblaw, 2011).Google Scholar
Kaltenbrunner, W, de Rijcke, S., ‘Quantifying ‘Output’ for Evaluation: Administrative Knowledge Politics and Changing Epistemic Cultures in Dutch Law Faculties’ (2017) Science and Public Policy 44(2), 284–293.Google Scholar
Kooijmans, T, ‘De roekeloze automobilist’ (2014) Ars Aequi Maandblad 118, 118–124.Google Scholar
Krans, B, ‘Vorm of Inhoud: de annotatie als wetenschappelijke publicatie?’ (2017) Ars Aequi Maandblad 237, 237–239.Google Scholar
Kurtz, MJ, Henneken, EA, ‘Measuring Metrics – a 40-Year Longitudinal Cross-Validation of Citations, Downloads, and Peer Review in Astrophysics’ (2017) Journal of the Association for Information Science and Technology, 68, 695–708.10.1002/asi.23689CrossRefGoogle Scholar
Merton, RK, ‘The Matthew Effect in Science, II. Cumulative Advantage and the Symbolism of Intellectual Property’ (1988) Isis 79, 606–623.10.1086/354848CrossRefGoogle Scholar
Merton, RK, ‘The Normative Structure of Science’ in Merton, RK (ed.) The Sociology of Science: Theoretical and Empirical Investigations (University of Chicago Press, 1973), 267–278.Google Scholar
Van Opijnen, M, Op en in het web. Hoe de toegankelijkheid van rechterlijke uitspraken kan worden verbeterd (PhD University of Amsterdam 2014).Google Scholar
Van Opijnen, M, Santos, C, ‘On the Concept of Relevance in Legal Information Retrieval’ (2017) Artificial Intelligence and Law 25, 65–87.10.1007/s10506-017-9195-8CrossRefGoogle Scholar
Pedregosa, et al. ‘Scikit-learn: Machine Learning in Python’ (2011) JMLR 12 , 2825–2830.Google Scholar
Perneger, TV, ‘Relation Between Online “Hit Counts” and Subsequent Citations: Prospective Study of Research Papers in the BMJ’ (2004) BMJ 329(7465), 546–547.10.1136/bmj.329.7465.546CrossRefGoogle ScholarPubMed
Saracevic, T, ‘Relevance: A Review of and a Framework for the Thinking on the Notion in Information Science’. (1975) Journal of the American Society for information science 26(6), 321–343.10.1002/asi.4630260604CrossRefGoogle Scholar
Snel, MVR, ‘Hoera, een Lijstje! Over Bronvermelden’ (2018) Ars Aequi 3, 254–260.Google Scholar
Snel, MVR, Meester(s) over bronnen: een empirische studie naar kwaliteitseisen, gevaren en onderzoekstechnieken die betrekking hebben op het brongebruik in academisch juridischdogmatisch onderzoek. (Den Haag: Boom Juridische Uitgevers, 2016).Google Scholar
Soetenhorst, WJ, ‘Een Juridische Citatie-Index: Het Proof of Concept is Voorhanden’ (2017) Nederlands Juristenblad 17(915), 1184–1186.Google Scholar
Stolker, C, Rethinking the Law School: Education, research, outreach and governance (Cambridge University Press 2015).Google Scholar
Stolker, C, ‘Een Discipline in Transitie: Rechtswetenschappelijk Onderzoek na de Commissie Koers’. (2011) Recht en Methode in onderzoek en onderwijs 1 ( 3), 13–43.Google Scholar
Royal Netherlands Academy of Arts and Sciences, Judging Research on its Merits – An Advisory Report by the Council for the Humanities and the Social Sciences Council (Amsterdam: Royal Netherlands Academy of Arts and Sciences 2005).Google Scholar
Teufel, S, Argumentative Zoning for Improved Citation Indexing, Shanahan, JG, Qu, Y and Wiebe, J, Computing Attitude and Affect in Text: Theory and Applications. (Dordrecht: Springer 2006) 159–169.Google Scholar
West, JD, Vilhena, DA, ‘A Network Approach to Scholarly Evaluation’, in Cronin, B, Sugimoto, CR (eds.), Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact. (The MIT Press 2014) 151–165.Google Scholar
Wiggers, G, Verberne, S, Zwenne, G-J, ‘Exploration of Intrinsic Relevance Judgments by Legal Professionals in Information Retrieval Systems’ in (2018) Proceedings of the 17th Dutch Belgian Information Retrieval workshop, 5–8.Google Scholar
Wiggers, G, Lamers, WS, ‘Shepard's Citations Revisited - Citation Metrics for Dutch Legal Information Retrieval’ in (2019) Proceedings of 17th International Conference of Scientometrics and Informetrics.Google Scholar
Winkels, R, Ruyter, J, & Kroese, H, ‘Determining Authority of Dutch Case Law’ (2011) Legal Knowledge and Information Systems, 235, 103–112.Google Scholar
Winkels, R, de Ruyter, J, ‘Survival of the Fittest: Network Analysis of Dutch Supreme Court Cases’ in International Workshop on AI Approaches to the Complexity of Legal Systems. (Berlin: Springer 2011) 106–115.Google Scholar
Winkels, R, Boer, A, Plantevin, I, ‘Creating Context Networks in Dutch Legislation’ in (2013) Proceedings of JURIX, 155–164.Google Scholar
Winkels, R, Boer, A, Vredebregt, B, van Someren, A, ‘Towards a Legal Recommender System’ In (2014) Proceedings of JURIX 271, 169–178.Google Scholar
Zuccala, A, and Cornacchia, R, ‘Data Matching, Integration, and Interoperability for a Metric Assessment of Monographs’ (2016) Scientometrics, 108(1), 465-484.10.1007/s11192-016-1911-8CrossRefGoogle Scholar