2.1 NLP applications in legal texts

The history of automated approaches to the legal domain dates back to the 1970s (Buchanan and Headrick Reference Buchanan and Headrick1970). In the late 1980s, practical applications of artificial intelligence-based approaches to law have started to appear (Francesconi et al. 2010; Bench-Capon et al. Reference Bench-Capon, Araszkiewicz, Ashley, Atkinson, Bex, Borges, Bourcier, Bourgine, Conrad, Francesconi, Gordon, Governatori, Leidner, Lewis, Loui, McCarty, Prakken, Schilder, Schweighofer, Thompson, Tyrrell, Verheij, Walton and Wyner2012; Casanovas et al. Reference Casanovas, Pagallo, Palmirani and Sartor2014).

Among the variety of NLP tasks that can be applied to legal documents, studies are commonly concentrated on predicting case outcomes (Ruger et al. Reference Ruger, Kim, Martin and Quinn2004; Martin et al. Reference Martin, Quinn, Ruger and Kim2004; Aletras et al. Reference Aletras, Tsarapatsanis, PreoŢiuc-Pietro and Lampos2016; Şulea et al. Reference Şulea, Zampieri, Vela and van Genabith2017b; Katz et al. Reference Katz, Bommarito and Blackman2017; Branting et al. Reference Branting, Yeh, Weiss, Merkhofer and Brown2018; Long et al. Reference Long, Tu, Liu and Sun2019). The European Court of Human Rights (ECHR) decisions and the French Supreme Court cases are used in several works for case outcome prediction (Aletras et al. Reference Aletras, Tsarapatsanis, PreoŢiuc-Pietro and Lampos2016; O’Sullivan and Beel Reference O’Sullivan and Beel2019; Medvedeva et al. Reference Medvedeva, Vols and Wieling2020). Studies for case outcome prediction also include Virtucio et al. (Reference Virtucio, Aborot, Abonita, Aviñante, Copino, Neverida, Osiana, Peramo, Syjuco and Tan2018), Kowsrihawat et al. (Reference Kowsrihawat, Vateekul and Boonkwan2018), Mumcuoğlu et al. (Reference Mumcuoğlu, Öztürk, Ozaktas and Koç2021), and Long et al. (Reference Long, Tu, Liu and Sun2019) for the Higher Courts of the Phillipines, Thailand, Turkey, and China, respectively.

Other core tasks are automatic summarization (Galgani et al. Reference Galgani, Compton and Hoffmann2012), text classification (Şulea et al. Reference Şulea, Zampieri, Malmasi, Vela, Dinu and van Genabith2017a), and QA (Kim et al. Reference Kim, Xu and Goebel2017; Morimoto et al. Reference Morimoto, Kubo, Sato, Shindo and Matsumoto2017). A comprehensive survey on text summarization in the legal domain has been compiled by Kanapala, Pal, and Pamula (Reference Kanapala, Pal and Pamula2019). To study the large-scale and extreme multi-label text classification problem, Chalkidis et al. (Reference Chalkidis, Fergadiotis, Malakasiotis, Aletras and Androutsopoulos2019a,b) introduced a corpus for the legal domain (called EURLEX57k) that contains 57,000 case documents from EU’s legislation database. Information retrieval applications on legal documents such as learning logical structures, labeling sentences, and finding legal facts by using machine learning methods are also increasingly grabbing attention (Ashley and Brüninghaus Reference Ashley and Brüninghaus2009; Bach et al. Reference Bach, Minh, Oanh and Shimazu2013; Sangeetha et al. Reference Sangeetha, Kavyashri, Swetha and Vignesh2017; Shulayeva, Siddharthan, and Wyner Reference Shulayeva, Siddharthan and Wyner2017). Another interesting task is capturing the relations between parts of legal documents (Nanda et al. Reference Nanda, Adebayo, Di Caro, Boella and Robaldo2017; Nguyen et al. Reference Nguyen, Nguyen, Tojo, Satoh and Shimazu2018).

Since the language of law is very different than daily language, domain-specific word embeddings for the legal domain are required to deploy state-of-the-art deep learning models (Chalkidis et al. Reference Chalkidis, Jana, Hartung, Bommarito, Androutsopoulos, Katz and Aletras2021). Chalkidis and Kampas (Reference Chalkidis and Kampas2019) presented the Law2Vec embeddings, which accelerate the developments in NLP for law. An important work to consolidate and accelerate the NLP applications in the legal domain is recently presented by Chalkidis et al. (Reference Chalkidis, Jana, Hartung, Bommarito, Androutsopoulos, Katz and Aletras2021). Their work is a consolidating collection of corpora and several standardized benchmarks for different NLP tasks. Recently, unwanted bias issues such as gender bias are also studied for legal word embeddings (Sevim, Şahinuç, and Koç Reference Sevim, Şahinuç and Koç2022).

2.2 General domain NER models

Early versions of NER models mostly consist of handcrafted rules and heavily rely on feature engineering-based machine learning models such as hidden Markov models (Leek Reference Leek1997; Freitag and McCallum Reference Freitag and McCallum1999) and maximum entropy Markov models (MEMMs) (Borthwick and Grishman Reference Borthwick and Grishman1999; McCallum, Freitag, and Pereira Reference McCallum, Freitag and Pereira2000). Having replaced these models in terms of popularity, CRFs became the contemporary sequence tagging models considered for NER tasks (Lafferty, McCallum, and Pereira Reference Lafferty, McCallum and Pereira2001). Still, these models poorly express nonlocal dependencies within sentences. The study of Finkel et al. (Reference Finkel, Grenager and Manning2005) tackled this problem with Gibbs sampling and reached 86.86% F1 score on CoNLL2003 corpus, which is a standardized corpus for a NER task (Sang and De Meulder Reference Sang and De Meulder2003). Later, the study of Krishnan and Manning (Reference Krishnan and Manning2006) proposed a double-layer CRF model to overcome the same problem, and they reported an F1 score of 87.24% for all entities in the same dataset.

Finkel and Manning (Reference Finkel and Manning2009) propose a joint learning of parsing and NER. They experimented on OntoNotes Release 2.0, which consists of six different news corpora. Their results vary from 66.49% to 88.18% F1 scores. Among these six corpora, Tkachenko and Simanovsky (Reference Tkachenko and Simanovsky2012) improved two of them in OntoNotes (ABC and CNN) by using further feature engineering. They also reached a 91.02% F1 score in the CoNLL2003 corpus with the same model.

The first attempt to utilize LSTM units for a NER task was made in the work of Hammerton (Reference Hammerton2003). Limitations on computational power and word embedding techniques of the time prevented this study to stand out, resulting with a 60% F1 score in English. In later years, artificial neural network-based models with their language-independent structures and feature engineering boosted the efficiency of NER applications (Yadav and Bethard Reference Yadav and Bethard2019). Collobert et al. (Reference Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa2011) proposed a CRF layer on top of CNN and reached a 89.59% F1 score.

Huang et al. (Reference Huang, Xu and Yu2015) experimented with four neural network architectures: LSTM, LSTM with a CRF layer, BiLSTM, and BiLSTM with a CRF layer. Models with BiLSTMs are effectively able to express and combine the forward and backward dependencies within a sentence, thus resulting in higher F1 scores. Among the reported F1 scores in the work of Huang et al. (Reference Huang, Xu and Yu2015), the most successful was BiLSTM with CRF model supported using “gazetteers,” which is a type of external resource that contains a list of NEs. Concurrently, Chiu and Nichols (Reference Chiu and Nichols2016) proposed a BiLSTM model without a CRF layer while strengthening their embeddings with character-level word embeddings extracted by a separate CNN. The final results are the concatenated versions of word2vec word embeddings (Mikolov et al. Reference Mikolov, Chen, Corrado and Dean2013) and extracted character-level features. Best reported results were 90.91% (not supported by lexicons) and 91.62% (supported by lexicons) F1 scores.

Lample et al. (2016) proposed two models, which they refer as BiLSTM/CRF and Stack LSTM (S-LSTM). Unlike Chiu and Nichols (Reference Chiu and Nichols2016), they constructed character-level features with another BiLSTM rather than CNN. Their BiLSTM/CRF model with character-level features reached state-of-the-art results for four languages without any language-specific resources (English 90.94%, German 78.76%, Dutch 81.74%, Spanish 85.75%). Similar to the work of Lample et al. (2016), Ma and Hovy (Reference Ma and Hovy2016) used CNN for character-level features as Chiu and Nichols (Reference Chiu and Nichols2016). This study presented the current state-of-the-art results in English (91.21% F1 score on CoNLL dataset) among the models that do not use any external resources. Peters et al. (Reference Peters, Neumann, Iyyer, Gardner, Clark, Lee and Zettlemoyer2018) improve NER in the general domain by using ELMo-based contextualized embeddings as word representations and achieve 92.22 F1 score. Straková et al. (Reference Straková, Straka and Hajič2019) extend the main task by investigating the nested NER structures. In their work, they used several architectures based on sequence-to-sequence encoders. They also enhanced their models with ELMo, BERT, and Flair, which are pretrained contextualized word embeddings. They have a reported 93.38% F1 score.

2.3 NER models developed for Turkish

Turkish is a morphologically rich language with its agglutinative structure. This property of Turkish language causes lower accuracy in purely statistical approaches in NLP tasks due to sparsity problem. Initially proposed by Oflazer (Reference Oflazer1994), morphological analysis was then applied to other Turkish information extraction models to increase performances by Tür et al. (Reference Tür, Hakkani-Tür and Oflazer2003). The model proposed in Tür et al. (Reference Tür, Hakkani-Tür and Oflazer2003) reached a $92.73 \%$ F1 score in NER task on their own corpus, which is widely used to evaluate Turkish NER models. Scores reported in this subsection are all performances obtained on this corpus. There exists some Turkish NER models that do not take morphological analysis into account such as Küçük and Yazıcı (Reference Küçük and Yazıcı2009); Küçük and Yazıcı (Reference Küçük and Yazıcı2012). These models are experimented on several domains such as news, history, child stories, and so on.

After CRF proved to be a successful tool for NER, work of Şeker and Eryiğit (Reference Şeker and Eryiğit2012) applied it to Turkish. Their model uses a morphological analyzer and two gazetteers, reaching a 91.94% F1 score. The work of Şeker and Eryiğit (Reference Şeker and Eryiğit2012) was also used as a part of the pipeline called ITU Turkish NLP Web Service as given in Eryiğit (Reference Eryiğit2014), where an input text is thoroughly processed for several tasks including NER.

First deep learning-based approach to Turkish NER is the work of Kuru, Can, and Yuret (Reference Kuru, Can and Yuret2016), where character-level representations are deployed instead of more popular word-level embeddings. Their model consists of a BiLSTM with five layers and a Viterbi decoder that is used to convert tag probabilities to tag sequences. They reached an F1 score of 91.30% without using any external resources. Akkaya and Can (Reference Akkaya and Can2021) tackled Turkish NER problem with transfer learning and additional CRF layers reaching an entity level F1 score of 67.39% for a non-domain-specific noisy corpus.

In NER models, the usual way of constructing word representations is by concatenating word embeddings with character-level features, that is constructed using BiLSTM or CNN. Güngör et al. (Reference Güngör, Güngör and Üsküdarli2019) further advanced this approach by adding an additional component, “morphological embedding.” They formed the morphological embeddings by feeding a separate BiLSTM by the morphological tags. This paper currently holds the state-of-the-art results with 92.93% F1 score. Moreover, this work also encapsulates the most recent NER results for four different morphologically rich languages, which are Czech (81.05% F1), Hungarian (96.11% F1), Finnish (84.34% F1), and Spanish (86.95% F1). Concurrently, Akdemir (Reference Akdemir2018) combined the same input configuration with joint learning of dependency parsing and NER, which was initially described by Finkel and Manning (Reference Finkel and Manning2009). This model reached up to a 90.9% F1 score.

2.4 NER models applied to legal documents

There are few previous works for NER applied to legal documents. Successful handcrafted NER models and legal metadata extractors were presented due to the highly uniform structures of corpora from the legal domain (Dozier et al. Reference Dozier, Kondadadi, Light, Vachher, Veeramachaneni and Wudali2010; Chalkidis et al. Reference Chalkidis, Androutsopoulos and Michos2017; Cardellino et al. Reference Cardellino, Teruel, Alemany and Villata2017; Sleimi et al. Reference Sleimi, Sannier, Sabetzadeh, Briand and Dann2018). Work that built the foundations of a NER idea in the legal domain was conducted by Dozier et al. (Reference Dozier, Kondadadi, Light, Vachher, Veeramachaneni and Wudali2010) on the US legal system. Cardellino et al. (Reference Cardellino, Teruel, Alemany and Villata2017) used Yet Another Great Ontology (YAGO) and Legal Knowledge Interchange Format (LKIF) corpora to implement their models for NER on judgments of the ECHR. On the other hand, even though they do not presented their work as a NER model in the legal domain, Chalkidis et al. (Reference Chalkidis, Androutsopoulos and Michos2017) and Sleimi et al. (Reference Sleimi, Sannier, Sabetzadeh, Briand and Dann2018) covered NER task while performing information extraction from legal texts. These works focus on English legal texts by utilizing available external resources or handcrafted rules.

NER models in the legal domain that do not deploy any external resource nor handcrafted rules have also started to emerge (Luz de Araujo et al. Reference Luz de Araujo, de Campos, de Oliveira, Stauffer, Couto and Bermejo2018; Leitner et al. Reference Leitner, Rehm and Moreno-Schneider2019; Vardhan et al. Reference Vardhan, Surana and Tripathy2020). Luz de Araujo et al. (Reference Luz de Araujo, de Campos, de Oliveira, Stauffer, Couto and Bermejo2018) introduced a NER corpus in the legal domain in Brazilian Portuguese and reached up to a 97.04% F1 score with a BiLSTM/CRF model in some NE categories that they defined. Similarly, Leitner et al. (Reference Leitner, Rehm and Moreno-Schneider2019) presented several deep learning models based on combinations of LSTMs and CRFs on German legal cases. Their paper is constructed upon two experiments, which can be titled as “fine-grained” (19 entity categories) and “coarse-grained” (7 entity categories). They reached 95.46% and 95.95% F1 scores for fine-grained and coarse-grained NER, respectively. For the case of English, in their proof-of-concept work, Vardhan et al. (Reference Vardhan, Surana and Tripathy2020) only analyzed the feasibility of NER task in the legal domain by modifying a ready-to-use NLP toolkit and reported 59.31% F1 score.

3.1 NE categories in the legal domain

When NER has been first introduced, it has been constructed with three main NE types: PER (Person), LOC (Location), and ORG (Organization). However, for domain-specific applications, these types can be expanded to any desired number. Leitner et al. (Reference Leitner, Rehm and Moreno-Schneider2019) used NRM (Legal Norm), REG (Case-by-case regulation), RS (Court Decision), and LIT (Legal Literature) in their work on German NER in the legal domain. However, due to the disparity of legal languages across nations, we did not migrate their classes directly but created our own classes. Still, some of our classes have an irrefutable correspondence with the classes they have. In total, we have decided on eight distinct NE tags, of which four are NEs in the legal domain. These are given in Table 1, extending common NEs (PER-LOC-ORG-DAT) with the task-specific ones (LEG-COU-REF-OFF).

Table 1. Named-entity tags in the legal domain

LEG is the abbreviation for the legislation relevant to the verdict and intuitively is the most crucial information of an arbitrary court case. COU is the abbreviation for court type. Court types are very important in law since each court has its specific authorities, areas of expertise, and features. Therefore, court types are highly relevant for information extraction from legal texts. REF is the short for reference if there is referral to a previous litigation. Extraction of references are significant in NLP applications in the legal domain, because previous litigation and court verdicts are extremely important in law. By using this information, one can find related cases easily and their verdicts, which are likely to apply to the current case at hand as well. Finally, OFF is the short for the official gazette, and it is also critical to track the effects of new or altered legislation on cases.

3.2 Tagging scheme

There are two common tagging schemes for NER tasks, which are Inside-Outside-Beginning (IOB) (Sang and De Meulder Reference Sang and De Meulder2003) and Inside-Outside-Beginning-End-Singleton (IOBES), adding “Singleton” and “End” to the previous one (Akdemir Reference Akdemir2018). In our paper, we use IOB Scheme. In IOB scheme, Out (O) is used for non-NE, Begin (B) for the initialization of a NE, and Inside (I) for an internal word of a NE. An example for this scheme is given in Table 2.

Table 2. Example sentence for IOB scheme

3.3 Labeling guidelines

3.4 Corpus statistics

We have manually labeled 350 cases from the Court of Cassation. This corresponds to 2198 sentences, 5311 NEs, and 123,173 tokens in total. We have randomly divided this corpus into training and test sets with 1758 and 439 sentences, respectively. Tag statistics for each set are given in Table 3.

Table 3. Named-entity distribution for the training and test sets

There are four resulting documents, which are sentence–tag pairs of the training set and the test set. Total size of these documents is $1.24$ MB.

4.1 NER model components in the legal domain

4.1.1 BiLSTM layer

Recurrent neural networks (RNNs) are a type of neural network which are widely used to process sequential data such as time series. Even though RNNs are useful for sequential data, they suffer from the vanishing gradient problem when processing long sequences of data. Thus, these networks are unable to fully capture the long-term dependencies within a sequential input. Processing natural languages requires handling of long-term dependencies within sentences for coherent extraction of information. LSTM has been introduced to overcome this weakness (Hochreiter and Schmidhuber Reference Hochreiter and Schmidhuber1997).

Figure 1. A single LSTM unit.

LSTM structure captures long-term dependencies by additional gates, such as forget gate, which determines the proportion of the information that should be remembered through the process. The structure of a single LSTM unit is given in Figure 1, and the following equations reflect the mathematical background of LSTM architecture (Huang et al. Reference Huang, Xu and Yu2015):

(1) \begin{align} i_t &= \sigma\!(W_{xi}x_{t} + W_{yi}y_{t-1} + W_{ci}c_{t-1} + b_i) \nonumber \\[3pt] f_t &= \sigma(W_{xf}x_{t} + W_{yf}y_{t-1} + W_{cf}c_{t-1} + b_f) \nonumber\\[3pt] c_t &= f_t\,{\odot}\,c_{t-1} + i_t \odot \tanh\!(W_{xc}x_{t} + W_{yc}y_{t-1} + b_c) \nonumber \\[3pt] o_t &= \sigma(W_{xo}x_{t} + W_{yo}y_{t-1} + W_{co}c_{t} + b_o)\nonumber \\[3pt] y_t &= o_t \odot \tanh\!(c_t),\end{align}

where i, f, and o are input, forget, and output gates, respectively. x, y, and c are input, hidden, and cell vectors, respectively. $\sigma$ is the logistic sigmoid function. Finally, W’s are weight matrices between realizations and units denoted by the two letters in the subscripts. For example, $W_{xi}$ is the weight matrix between the realization x and unit i.

In our work, we used the BiLSTM architecture given in Graves and Schmidhuber (Reference Graves and Schmidhuber2005), which examines a sentence both in forward and reverse orders to catch and relate forward and backward dependencies. This is achieved by feeding a separate LSTM with reversed sequence of words. The final output is reached by concatenating forward and backward LSTM outputs.

4.1.2 CRF layer

Sequence tagging tasks are widely necessary in several applications such as part-of-speech (POS) tagging, syntactic disambiguation, and NER. Hidden Markov models (HMMs) are the earliest models that strongly rely on the conditional independency. Due to the high dependency in NER tasks (i.e., I-PER tag can only come after B-PER tag), HMMs are not the superior sequence tagging model in the said models (Lample et al. 2016). Non-generative models, such as MEMMs, are successful for dependent environments. However, they suffer from being biased to the most frequently detected transitions during the training (Lafferty et al. Reference Lafferty, McCallum and Pereira2001).

On the other hand, since they do not require conditional independency between features to tackle the biasing problem, CRFs are proven to have a better performance in sequence tagging tasks. The main difference of CRF networks is that they have a single exponential model for the joint probability of the entire sequence of labels given the observation sequence (Lafferty et al. Reference Lafferty, McCallum and Pereira2001). CRFs are used to extract the most probable output per element in a model via defining conditional probabilities observed on training data. Mainly, given a graph $\mathcal{G} = (\mathcal{V},\mathcal{E})$ with $\mathcal{V}$ is the set whose elements are vertices and the set $\mathcal{E}$ holds the paired vertices, called edges. In our case, this is a linear chain of tags and corresponding tokens. Then, observation variable $\textbf{X}$ is defined as the sequence of tokens, and a random variable $\textbf{Y} = \textbf{Y}_{v\in V}$ can be defined as a possible tag from the vertices $\mathcal{V}$ of $\mathcal{G}$ . Resulting $(\textbf{X}, \textbf{Y}_{\textbf{v}})$ is a CRF that random variables $\textbf{Y}$ are conditioned on $\textbf{X}$ , obeying the Markov property on $\mathcal{G}$ :

(2) \begin{equation}P(Y|X) = \frac{exp\!\left(\sum_j{ w_j F_j(X, Y)}\right)}{\sum_{Y^{'}}{ exp\!\left( \sum_j {w_j F_j(X, Y^{'})}\right)}},\end{equation}

where $F_j(X,Y)$ denotes the feature function and $w_j$ denotes corresponding weights. Both $F_j(X,Y)$ and $w_j$ are trained with back-propagation. Equation (2) can be interpreted as the inner summation of feature functions on a set of vertices normalized with summation over all features on $\mathcal{G}$ . Tuning this, the output is given as the tag that has the maximum likelihood, defined on $\textbf{Y}$ for each observation variable (token in our corpus) $\textbf{X}$ .

4.2 Word representations

As our word representations, we used six different combinations of trained GloVe embeddings, trained Morph2Vec embeddings, and extracted character representations in order to feed our models. We present these combinations of embeddings in Figure 2. Both GloVe and Morph2Vec are used separately as well as concatenated to each other as shown in Figure 2a and c. The model that uses them in the concatenated mode is called as hybrid model. These three options are considered as base word embeddings. Building on them, additional character-level features (CFs) are included as supplementary models as given in Figure 2b and d. CF is appended to base models, denoted as (GloVe/Morph2Vec + CF) and (GloVe + Morph2Vec + CF), respectively. In the following sections, we will refer (GloVe + Morph2Vec) by Hybrid. In what follows, we provide details on these word representation alternatives.

Figure 2. Word representation models.

4.2.3 Character-level features

Usage of character-level features is another way to improve model performance, while working on morphologically rich languages on NLP tasks. Instead of using sub-word information, character patterns in a word can be examined with this technique.

Character-level features (CFs) are extracted with two different neural architectures, which are BiLSTM and CNN. Considering that each word is a sequence of letters, processing these sequences by using a BiLSTM architecture allows us to capture each character embedding. Similar to BiLSTMs, CNNs are again very powerful in capturing complex relations. In both techniques, each column of the final weight matrix of the trained neural architecture corresponds to a character embedding. CFs are just the concatenated versions of these character embeddings. Applications of these extracted features to substantiate NER tasks are presented by Lample et al. (2016) and Chiu and Nichols (Reference Chiu and Nichols2016), respectively, for BiLSTM and CNN. Extracted CFs are concatenated with GloVe, Morph2Vec, and Hybrid representations in separate experiments, where the dimension for extracted CFs is 100. We have experimented with both extraction techniques and compared the results.

To sum up, we have six different word representation alternatives as given above with three of them containing CF representations. By using two different techniques to obtain CFs, we end up with nine different configurations. We explicitly tabulate each one of them with their dimensions in Table 4, where concatenation is denoted with $+$ .

Table 4. Word representation combinations and their dimensions used in experiments

4.3 NER models in the legal domain

NER models we constructed to address domain-specific NER for Turkish legal documents rely on three different architectures and three different base word embeddings, which are GloVe, Morph2Vec, and Hybrid. Our first and base architecture is the LSTM-CRF (LC for short) model (Figure 3), where character-level features are not considered explicitly (#1, #4, and #7 in Table 4). Turkish NER in the legal domain is therefore characterized in the embedding part of this network. This model is the most basic approach and generalizable for NER in other languages. In this base architecture and in the following architectures stemmed from LSTM-CRF, 1-layer BiLSTM is used.

Figure 3. LSTM-CRF model. CS stands for a single unit of CRF.

Since in Turkish, sequential structure of suffixes in a word create diversified lemma, another LSTM unit for extracting the features within words are proposed. With this regard, we used LC model with character-level features extracted by another BiLSTM as illustrated in Figure 4, and we refer this model as LSTM-LSTM-CRF (LLC for short) model (#2, #5, and #8 in Table 4). With this model, we aim to enhance upon the performance of LC models in the domain of Turkish legal texts.

Figure 4. LLC model. CS stands for a single unit of CRF.

The last architecture is similar to the LSTM-LSTM-CRF model. However, in this architecture, we extracted the character-level features with CNN as demonstrated in Figure 5, and thus we refer it as CNN-LSTM-CRF (CLC for short) model (#3, #6, and #9 in Table 4). While character-level features are in a sequential structure for agglutinative languages, CNN is also a promising approach in the sense of capturing not only all suffices but also the most defining ones in terms of changes in meaning. Legal texts are in a form that is exceedingly formalized and well defined in order to prevent misinterpretations. With this structure of the input, effect of individual words and their different morphological structures are designed to be captured well enough without the need of any sequential extraction.

Figure 5. CLC model. CS stands for a single unit of CRF.

5.1 Quantitative results

We fed our NER models (LC, LLC, and CLC) with three different base word representations, which are GloVe, Morph2Vec, and Hybrid embeddings. Training and test procedures were done using the manually tagged corpus presented in Section 3.4. The main focus of different combinations of embeddings and models is capturing the effect of character- or morpheme-level extraction done at different levels. Extra CNN and LSTM layers enable additional character features concatenated to the initial word embedding. In addition, base word embedding may also contain morpheme-level information as it is the case for Hybrid and Morph2Vec. Therefore, we mainly structured our experiments to highlight the differences between the effects of including multiple representations. Furthermore, effects of obtaining them with CNN or LSTM in between the structures of our proposed models are investigated with respect to their class based and overall scores.

Results are clustered and presented in a similar manner, where each table refers to the results of a respective base word representation for proposed models. They are evaluated with F1 score metric due to the highly unbalanced distribution of tags (e.g., O class: 24,124; LOC class: 5). This is also the reason why we do not consider individual entity-based accuracy scores but evaluate individual F1 scores. Tables 5, 6, and 7 present results in terms of Precision, Recall, and F1 score for each base word embedding GloVe, Morph2Vec, and Hybrid, respectively. The best results for F1 score within each word representation (for all Tables 5, 6, and 7) are given underlined, separately for each NE as well as total score. The best results for a class among all the inter-model results are also given in bold across Tables 5, 6, and 7.

Table 5. GloVe as the selected word embedding (all metrics are given in percentages.)

Table 6. Morph2Vec as the selected word embedding (all metrics are given in percentages.)

Table 7. Hybrid word embeddings as the selected word embeddings (all metrics are given in percentages.)

To summarize and compare all models with their best-performing word representation variant, Table 8 reiterates the best results of each model and embedding with respect to their overall F1 performance as well as accuracy. The main interest is F1 score, since we deal with highly unbalanced sets.

Table 8. Overall best results for each model and word embedding. All metrics are given in percentages

Along with the metric-wise results presented in Tables 5, 6, 7, and 8, we also provide count statistics for each model in Table 9. First two columns are the properties of our corpus, so they are equal for all models. This table mainly extricates the number of recognized and correctly recognized entities in the test set. Comparing Tables 8 and 9, we verify that selection of F1 score as the main metric is validated in the sense that F1 results are more consistent.

Table 9. General entity count statistics of the test set

To substantiate our results, we extend our experiments using k-fold cross-validation with all models. To obtain a controlled comparison environment, the same random order is used for all models. We used cross-validation to evaluate the performance of our models with respect to randomized partitions. The results are given in Figure 6, where the quantile distributions, maximum and minimum points, the median, and outlying points (if exist) of the test runs are plotted. Due to the limited size of our corpus, we started with only five partitions as given in Figure 6a to prevent insufficient test sizes. We encoded the model names on the lateral axis using their initials, where one can observe LC models perform poorly compared to the others. Among LLC and CLC results, a comparable performance is observed where Morph2Vec approach performs slightly worse. Extending our validation to 10 partitions on Figure 6a, we observe similar behaviors to support our comments. Overall, we report an average F1 score of 80% with five points of standard deviation with cross-validation on our better performing models. Note that these F1 values are approximately 10 points less than our reported test results. This is due to test sizes getting one-fifth of the original size with fivefold validation and even less with 10-fold case. Considering that O entities are much more frequent than the others (Table 3), having more O tag ratio on a partition by chance dramatically reflects on the performance since sample sizes of the other entity types shrink substantially to compensate. Still, the cross-validation results support our claims about the relationship between the proposed models.

Figure 6. K-fold cross validation results.

Statistical significance analyses are carried out for our experimental results. It is once again important to stress that our aim is to present a baseline for Turkish Legal NER domain rather than introducing state-of-the-art architectures. Therefore, we present these metrics to lay a better baseline. Replicating the experiments 10 times per model, we conducted analysis of variance (ANOVA) tests on the results. Overall, our total of nine models represent the variance in data quite well with $p < 0.05$ value. Combining them in groups of three as LSTM-CRF, LSTM-LSTM-CRF, and CNN-LSTM-CRF-based architectures, we once again obtain statistically significant ANOVA results for respective embeddings within these groups. To compare the models one by one with respect to nine-factor experiments, confidence intervals of LC-based models are negative and contain zeros. It follows that we can conclude LSTM-CRF models do not contribute much to the goodness-of-fit to the data at hand. We further conducted t-tests to the performances (as given in Table 8) of dual combinations of our models. Whether the dual combinations are statistically significant or not are presented in Table 10. Out of 36 possible combinations, we find 28 statistically significant results, whereas only 8 insignificant changes on the outcomes are detected, most of which occurring on cross-architectural comparisons. One common pattern is between GloVe and Hybrid embeddings for each architecture, where we find $p > 0.05$ for all of them. Therefore, we will qualitatively analyze some features for which Hybrid representations capture better than GloVe ones in the next subsection.

Table 10. Statistical significance of the experimental results presented in Table 8. “+” for p-value $< 0.05$ , “–” otherwise

5.2 Qualitative results

In order to provide qualitative analysis on our results, we present sample test sentences. These sentences are representative examples of the enhancing effect of Hybrid representations over GloVe only ones. The first sentence is tested on LSTM-LSTM-CRF and the second one is tested on CNN-LSTM-CRF models. These sentences are provided in Table 11. Their correct tagging and corresponding predictions of GloVe and Hybrid embedded models are given in Tables 12 and 13, respectively. We selected these empirically better models, which have poor significance test results, for qualitative comparison to keep this section succinct.

Table 11. Example test sentences

Table 12. Sentence 1 prediction comparison

Table 13. Sentence 2 prediction comparison

In Table 12, LSTM-LSTM-CRF-Hybrid is able to classify both forms of Hazine (Treasure), but LSTM-LSTM-CRF GloVe can only classify the orientation form of the word. This is a result of the Morph2Vec portion of the Hybrid representation that learns the suffix structures.

5.3 Comparison with related works on NER tasks in general domain

In this subsection, we evaluate our architectures on two different corpora and compare our results with previously published results. First of them is the CONLL2003 corpus in English, which contains around 300K tokens. The other is the Turkish corpus presented in Tür et al. (Reference Tür, Hakkani-Tür and Oflazer2003), which consists around 350K tokens. Detailed properties of both corpora are given in Table 14. Even though succeeding works alter them in minor ways, both corpora are most widely used ones in their respective languages.

Table 14. Properties of corpora used in comparisons

The results of our models and the performances of several baseline methods on CONLL2003 and Tür et al. (Reference Tür, Hakkani-Tür and Oflazer2003) corpora are presented in Table 15. These results should be investigated in two dimensions, English and Turkish. In the case of English, we obtain highly comparable results. Despite the fact that English is not a morphologically rich language, even the architectures including pure Morph2Vec embeddings have acceptable results. Results on Turkish NER, on the other hand, are slightly trickier to investigate. The Turkish corpus had several modifications since its first publication in 2003. We had access to the version of Güngör et al. (Reference Güngör, Güngör and Üsküdarli2019) which includes morphological descriptions of each word in the corpus, where their notation was introduced by Oflazer (Reference Oflazer1994). Their morphological analyzer pipeline with handcrafted linguistic rules utilize this additional information and substantially increase the performance of the architecture. We had to exclude these part of the corpus to make it compatible with our architectures. Morph2Vec, which was trained in an unsupervised manner, is not able to match this morphological analyzer. That being said, the scope is not to improve the state-of-the-results neither in the Turkish nor in English “general” NER but to set the baseline in Turkish legal domain NER. Nevertheless, our results on these domains are also quite comparable with the state-of-the-art results reported in the literature. As a final note, one of the possible future research directions would be to enhance our legal NER corpus with the morphological descriptions as Güngör et al. (Reference Güngör, Güngör and Üsküdarli2019) did for the general Turkish NER and then combine Güngör et al. (Reference Güngör, Güngör and Üsküdarli2019)’s approach with ours to obtain better performing methods.

Table 15. Comparison of the performance of our approach with other NER corpora