The promise of scaling lower court decisions
How similar is the legal reasoning in comparable court decisions? The answer to this question is of key concern to everyone with an interest in law – scholars or practitioner alike. Some might want to quickly identify particularly controversial decisions. Others seek to get a more comprehensive overview over a large set of decisions, for example, to understand patterns across different courts or the development of a legal doctrine over time. This requires legal methods and a close reading of each decision, which is realistic for a limited set of decisions. Existing approaches for comparing decisions at large scale mostly rely on data on the voting behavior of the respective judges in a court (e.g., Clark and Lauderdale Reference Clark and Lauderdale2010). It turns out, however, that in cross-national comparison judges do not always take a vote – and if they do, courts only rarely publish the individual votes (Kelemen Reference Kelemen2013; Raffaelli Reference Raffaelli2012). Existing methods only allow comparing courts’ decisions in some countries (e.g., Martin and Quinn Reference Martin and Quinn2002; Hanretty Reference Hanretty2012a,Reference Hanretty b).
We introduce a scaling model that estimates the location of court decisions in a common case-space. Instead of published votes, the model relies on citations.Footnote 1 How often does a decision refer to a particular legal source? Similar to Clark and Lauderdale (Reference Clark and Lauderdale2010), we assume that the closer the decision to a legal source in a common case-space—and hence the more amenable a source to the legal reasoning in the decision—, the more likely is a “panel of judges” (or simply a “court”) to refer to this legal source. The main advantage of our approach is that citations are commonly available in every legal system at all levels because judges need these references to justify their reasoning.
For our model, we first preselect decisions that actually can be compared on legal grounds. While curating a concise set of legal documents can of course be done manually, we show how to help scale the human effort with algorithms from information retrieval. We assess data from the legal database Juris,Footnote 2 and investigate systematic tendencies in judicial decision making in an unlikely case. Germany has a civil law system that is renowned for a particularly impartial and objective way of creating legal decisions (Langbein Reference Langbein1985). However, a recent debate on ‘forum selling’ and ‘forum shopping’ at Germany’s Landgerichte (district courts) suggests that there are areas where courts are systematically biased (Klerman and Reilly Reference Klerman and Reilly2016; Bechtold, Frankenreiter and Klerman Reference Bechtold, Frankenreiter and Klerman2019). Studying cases from press law and antitrust, we indeed corroborate these findings at Germany’s lower courts in Cologne (Köln) and Hamburg for press law, as well as Cologne, Dortmund and Mannheim for antitrust.
We contribute to the literature in three ways. First, we introduce a new model that allows scholars and legal practitioners to systematically compare decisions that share a common case-space on the basis of their references—which is data that is widely available. Second, we showcase how to scale measurement efforts when analyzing decisions from a large legal corpus of lower court decisions. Tools from information retrieval allow to identify decisions that lie in the same common case-space and that can thus be compared to one another. Third, the substantive application of our new scaling model informs recent debates on “forum selling” and “forum shopping” because we show that some courts are systematically more plaintiff friendly than other courts. Our model helps investigate systematic biases in court decisions comparatively and at large scale.
The paper proceeds as follows. We begin with surveying existing approaches that are capable of mapping court decisions in a common case-space. The subsequent section explains why citations leave a trace about a court’s tendency in rulemaking. We then translate this reasoning into a statistical model and finally study forum selling in Germany’s lower courts in press law and antitrust. A final section concludes.
Measuring the content of court decisions
Measuring the relative legal positions of court decisions implicitly accepts the notion that the decisions reside on a continuum: Cases that are clearly within the bounds of the law on one end of a spectrum, cases that clearly violate it on the other end and all others somewhere in between. This spatial notion has been formalized into an analytical framework originally developed to study the influence of politics in high courts such as the U.S. Supreme Court (for an overview, e.g., Clark and Lauderdale Reference Clark and Lauderdale2010; Lax Reference Lax2011). But while the resulting case-space model (Kornhauser Reference Kornhauser1992a,Reference Kornhauser b; Landa and Lax Reference Landa and Lax2008; Lax Reference Lax2011, Reference Lax2012) offers a rigorous analytical framework, one of its most significant limitations is an empirical one. So far, there are no tools to easily and reliably map a large number of real-world cases into such a common case-space.
Existing empirical strategies offer room for improvement. Broad ideological categories (e.g., liberal-conservative) are of limited use since they fail to honor the respective legal context. Nevertheless, some even equate the latent political – not the legal – position of a politically nominated (median) judge with a court’s resulting decisions (e.g., Brouard and Hönnige Reference Brouard and Hönnige2017; Carrubba et al. Reference Carrubba, Friedman, Martin and Vanberg2012; Hönnige Reference Hönnige2009; Sternberg et al. Reference Sternberg, Gschwend, Wittig and Engst2015). The judge’s displayed political position that is taken as a proxy for her decisions might not even be her own but can be inherited from other actors who nominated the (median) judge such as parties, senators, or presidents (e.g., Epstein et al. Reference Epstein, Martin, Segal and Westerland2007; Hönnige Reference Hönnige2009). An alternative approach is to closely analyze the decisions’ content. Law scholars embraced content analysis methods a while ago (Hall and Wright Reference Hall and Wright2008). However, when hand coding potentially a large number of written decisions, reliability can indeed be an issue. In addition, since the task is labor intensive, it is hard to scale – particularly so when time and resources are scarce.
There are various ways in which scholars can make the most of computers when they want to locate decisions in a common case-space at large scale. Of course, the words of decisions reflect their meaning. Existing approaches pay close attention to studying the language of court decisions and have been exploring supervised and unsupervised text scaling (e.g., Dyevre Reference Dyevre2019; Evans et al. Reference Evans, McIntosh, Lin and Cates2007; Jakab, Dyevre and Itzcovich Reference Jakab, Dyevre and Itzcovich2017; McGuire and Vanberg Reference McGuire and Vanberg2005) or dictionary methods (e.g., Owens and Wedeking Reference Owens and Wedeking2011, Reference Owens and Wedeking2012). In the light of the specific requirements of legal terminology, further research in this direction promises important progress. Another option is to rely on the legal sources courts cite in their decisions (Whalen Reference Whalen2016). The resulting citation networks allow, for example, to uncover the relevance of a court decision (Coupette and Fleckner Reference Coupette and Fleckner2018; Fowler et al. Reference Fowler, Johnson, Spriggs, Jeon and Wahlbeck2007; Petersen and Towfigh Reference Petersen and Towfigh2017; Winkels, de Ruyter and Kroese Reference Winkels, de Ruyter and Kroese2011). Citation patterns have also been used to show that judges chose legal sources neither randomly nor independently of their preferences. Frankenreiter (Reference Frankenreiter2017) exploits the institutional setting at the ECJ where – unlike in other courts – two opinions are drafted, one by the Advocate General and another one by the Judge Rapporteur. He finds that judges tend to cite decisions of judges appointed by Member State governments that have similar preferences regarding European integration. Finally, we are not the first to use citation patterns for explicitly mapping decisions into a common case-space. Clark and Lauderdale (Reference Clark and Lauderdale2010) analyze search and seizure cases and freedom of religion opinions before the U.S. Supreme Court between 1953 and 2006. But while Clark and Lauderdale have to consider the judges’ voting behavior to estimate valid positions of opinions, we show how to do so without such data. Given that, in cross-national comparison the highest courts rarely, or never, publish judges’ votes (Kelemen Reference Kelemen2013; Raffaelli Reference Raffaelli2012), we believe that our approach offers a welcome contribution to the existing toolkit for locating decisions in a common case-space.
In short, current approaches that locate decisions in a common case-space too often work with crude proxies. There are unexplored potentials in going beyond the human effort of reading and coding each single decision. While first work exists that makes use of the information in citation patterns, there is yet no model that allows mapping a court decision in a common case-space in a convenient way.
Locating decisions using citation data
The text of a written decision serves the same purpose in any legal system. It provides an argument why a certain case is decided in the particular way it is and not differently. Judges refer to legal sources to bolster their argument. The sources a court is citing in a decision are not only a technical requirement; citations may also reveal legal preferences. Ultimately, the paper seeks to provide nuanced measurements that allow scaling court decisions – which could, for example, be used to tell a “hard” from a “soft” verdict. To explain how to quantify this statement, we first lay a conceptual foundation and take a closer look at the case-space framework. We will then understand why different legal preferences express themselves in the citations of a court decision and finally consider the consequences arising from this observation.
The common case-space
The case-space model comes with a number of core concepts (Cameron and Kornhauser Reference Cameron and Kornhauser2017a,Reference Cameron and Kornhauser b). A legal case can be defined as a “concrete, fact-ridden dispute between two (or perhaps more) parties (Cameron and Kornhauser Reference Cameron and Kornhauser2017a, 2).” The court has to resolve the dispute – and it does so by applying the law as a rule to the legal facts of the case based on the available evidence. All possible cases on the same subject matter reside on a one-dimensional common case-space. For example, if the police catches car drivers at a certain speed, it is possible to map all cases into this common case-space on the basis of their speed. For the court to be able to take a decision, it requires a rule that tells the court when the defendant has to be convicted. In the example, there needs to be a law that defines the maximum speed. A driver below that threshold will not be convicted; a driver above that speed will be.
Turning facts into a disposition on the basis of a rule seems straightforward in the simplified speeding example. And if adjudicating were a simple matter of applying a rule to clear facts, an algorithm would be qualified enough to sit on the bench. The legal reality, however, is far more complex. There are two processes that describe how a case based on available evidence can be mapped into a case-space to arrive at a disposition. They are not strictly deterministic and require human judgment (Cameron and Kornhauser Reference Cameron and Kornhauser2017a). First, judges have to translate the available evidence into legal facts. We expect that courts do not differ systematically in their translation from evidence into legal facts. Second, they need to select the appropriate rule and apply it to the identified legal facts. Typically, a court has to evaluate legal facts with a rule on more than one single issue to arrive at a verdict, which is why judges ultimately need to weigh and aggregate all issues when they finally speak the law.
When courts evaluate facts on the basis of a rule – be it laws, existing precedent, previous opinions on similar cases, or legal scholarship – they have to provide reasons to legitimize their judgment (Charlotin Reference Charlotin2017) and signal consistency of their reasoning (Lupu and Voeten Reference Lupu and Voeten2012) by citing particular legal sources. The set of citations in a decision is indicative of the court’s opinion (Clark and Lauderdale Reference Clark and Lauderdale2010; Choi and Gulati Reference Choi and Gulati2008). Two courts who apply the same set of rules to assess the same legal facts will come to the same conclusion citing similar legal sources. If the judges decide a case differently, they are likely to apply different rules to the legal facts and hence will refer to different legal sources.Footnote 3 Therefore, the legal sources in a set of decisions of the same case-space hold the key for locating the decisions in that common case-space.
How do courts cite?
The legal sources a court refers to signal important information about the kind of legal argument and the framing a court is advancing in a decision (Alschner and Charlotin Reference Alschner and Charlotin2018; Clark and Lauderdale Reference Clark and Lauderdale2012; Charlotin Reference Charlotin2017; Choi and Gulati Reference Choi and Gulati2008). How do judges pick these legal sources? We argue that there are three different behavioral mechanisms that drive what and how often a source is referred to in a court decision. First, studying citation practices of federal appellate court judges in the United States, Choi and Gulati (Reference Choi and Gulati2008, p. 91) argue that the substance matter dictates an authoritative core set of legal sources any judge would mention, for instance, binding rules and norms or previous decisions on similar cases. This argument travels across different legal traditions and also holds for civil law systems (e.g., Choi and Gulati Reference Choi and Gulati2008; Clark and Lauderdale Reference Clark and Lauderdale2010). Those core legal sources define the legal substance of what is at stake. All court decisions in a common case-space refer to these core legal sources with a similar frequency. Second, there are legal sources we call idiosyncratic legal sources because they appear in only one written decision. A court’s decision may refer to idiosyncratic legal sources to account for the characteristics of a particular case. Citations of such idiosyncratic legal sources cannot be informative to determine the relative locations among similar decisions within a common case-space. Finally, there are informative legal sources that help us estimate the relative locations of decisions within a common case-space. Decisions refer to these sources with a different frequency that reflects their location in the common case-space. In short, we distinguish between three different types of what Posner (Reference Posner2000, p. 384) calls “informational” citations. Among these three different citation mechanisms, only the latter one using informative legal sources allows identifying the relative location of a decision in the common case-space.
Many studies of citation patterns find that a decision is more likely to refer to legal sources that are in line with the court’s reasoning (e.g., Choi and Gulati Reference Choi and Gulati2008; Clark and Lauderdale Reference Clark and Lauderdale2010). Sources that reflect the spirit of the decision help make a legally sound argument (Charlotin Reference Charlotin2017, p. 282), which is why the court is likely to quote these legal sources. A court tends to refrain from citing legal sources that are not supportive of the advanced legal argument because they generate cognitive dissonance and, eventually, cause extra effort when justifying the court’s decision (Charlotin Reference Charlotin2017; Posner Reference Posner2000). The court refers to dissenting legal sources – if at all – only in passing and will mention them strategically (Alschner and Charlotin Reference Alschner and Charlotin2018; Lupu and Voeten Reference Lupu and Voeten2012; Lupu and Fowler Reference Lupu and Fowler2013). Courts may distinguish their legal argument from existing ones to show why a certain legal belief is not binding or valid by entertaining a potential counterargument or introducing a more nuanced view. Similar to the analysis of precedent in decisions of the ECJ (Jacob Reference Jacob2014), these negative citations are used to bolster the credibility of the decision, demonstrating the argument’s coherence or fending off potential revisions from higher courts. In addition, judges try to minimize their chances of reversal by a higher court and to foster their own reputation (Choi and Gulati Reference Choi and Gulati2008; Jacob Reference Jacob2014).
In result, given limited time and resources, courts are more likely to refer to a legal source in line with the legal reasoning of the decision than a legal source that runs against it. Thus, courts are not only more likely to refer to supportive legal sources in their decisions, they will also refer to them more frequently.Footnote 4
Implications and features
To illustrate the implications of the model, we introduce a simple visualization in Figure 1. We begin with the left panel where two decisions
$ {D}_1 $
and
$ {D}_2 $
in a common case-space refer to the informative legal sources
$ {S}_1 $
and
$ {S}_2 $
. The thickness of the lines corresponds to how often the two decisions are referring to the sources. The decision
$ {D}_1 $
is referring to the source
$ {S}_1 $
quite heavily, which is why their locations are close to one another. In contrast, the decision
$ {D}_2 $
is referring to the two sources
$ {S}_1 $
and
$ {S}_2 $
to a similar degree and its location is therefore more central. Note that
$ {D}_2 $
is not completely in the middle between the two sources because of the way in which
$ {D}_1 $
is referring to the two sources. The location of the decisions
$ {D}_1 $
and
$ {D}_2 $
changes when we introduce a third decision
$ {D}_3 $
that is referring to
$ {S}_2 $
and
$ {S}_1 $
to the same degree as
$ {D}_1 $
is referring to
$ {S}_1 $
and
$ {S}_2 $
. The symmetry of citations now enforces the symmetry of the locations. Both of the previous decisions
$ {D}_1 $
and even more so
$ {D}_2 $
change their location with regard to
$ {S}_1 $
and
$ {S}_2 $
when adding
$ {D}_3 $
.Footnote 5
Figure 1. Locations of decisions
$ {D}_1 $
,
$ {D}_2 $
and
$ {D}_3 $
in the same case-space given how often they refer to sources
$ {S}_1 $
and
$ {S}_2 $
.
Notes: The thickness of the lines corresponds to the number of citations. A third decision in the sample changes the absolute and relative locations of
$ {D}_1 $
and
$ {D}_2 $
.
In addition, adding
$ {D}_3 $
also changes the relative locations in the common case-space. Now,
$ {D}_1 $
and
$ {D}_2 $
do not represent the minimum and maximum of the underlying common case-space any longer. Instead, the extreme locations of the new space are defined by
$ {D}_1 $
and
$ {D}_3 $
. If we define the decisions’ common case-space to have a standardized metric – for example, z-scores – then the numerical value from
$ {D}_2 $
would change to a much more central location once the decision
$ {D}_3 $
is included.
In contrast to previous research (e.g., Clark and Lauderdale Reference Clark and Lauderdale2010), we consider all legal sources and do not exclude any sources, such as procedural legal sources. Also, with our proposed approach it is not necessary to ex ante distinguish between positive and negative citations and between different types of legal sources to estimate the location of lower court decisions in an appropriate common case-space. Positive and negative citations are already accounted for with different expected frequencies in the data-generating process since negative citations occur less often than positive citations. Furthermore, irrespective of the type of the legal sources, the frequency with which lower courts refer to certain legal sources – substantive or procedural – will always be informative because it reflects a choice of the author of a decision.
A case-space estimator for decisions and cited legal sources
Now that we developed an understanding for how courts refer to legal sources when writing decisions, we translate these insights into an appropriate measurement model. The data we observe consist of a
$ n\times m $
decision-source matrix
$ \mathbf{Y} $
of citation counts, that is,
$ {y}_{ij} $
represents how often a court’s decision
$ i\hskip0.35em \in \hskip0.35em \left\{1,\dots, n\right\} $
is referring to legal source
$ j\hskip0.35em \in \hskip0.35em \left\{1,\dots, m\right\} $
. We use a Poisson distribution as a typical probability model for such citation count data. The systematic component of our model rests on three assumptions. (1) Each decision has a fixed location along a unidimensional case-space. (2) A written decision will refer to a legal source more often the closer their locations in this common case-space.Footnote 6 (3) Each decision has a positive probability to refer to any legal source. We express the probability of observing any particular distribution of legal source citations as
$$ {y}_{ij}\hskip0.35em =\hskip0.35em \mathrm{Poisson}\left({\lambda}_{ij}\right) $$
$$ {\lambda}_{ij}\hskip0.35em =\hskip0.35em \exp \left({\alpha}_j+{\beta}_i-\gamma \parallel {\theta}_i-{\phi}_j{\parallel}^2\right). $$
The distance between a decision
$ i $
and a legal source
$ j $
is expressed as
$ \parallel {\theta}_i-{\phi}_j{\parallel}^2 $
, where
$ {\theta}_i\hskip0.35em \in \hskip0.35em \mathrm{\mathbb{R}} $
is the location of decision
$ i $
and
$ {\phi}_j\hskip0.35em \in \hskip0.35em \mathrm{\mathbb{R}} $
is the location of legal source
$ j $
.Footnote 7 The parameter
$ \gamma $
captures the overall sensitivity of this difference in the respective common case-space. The parameters
$ {\alpha}_j $
and
$ {\beta}_i $
explicitly capture the idiosyncrasies of citation counts. The parameter
$ {\alpha}_j $
expresses the authority of a legal source
$ j $
. Some legal sources are by default cited more often than others simply because they are, on average, more relevant. Similarly,
$ {\beta}_i $
captures decision-specific differences. Some decisions refer to, on average, more legal sources than others, for example, because they are longer.
Identification is a core concern when estimating latent variable models. These models typically have more parameters than observations, which results in infinitely many “correct” solutions. Shifting, rotating or scaling one parameter can be easily offset by inversely shifting, rotating or scaling another related parameter. A unique solution needs to address all of these concerns. First, the model has to be anchored in the parameter space so that it does not float around. Second, identification also requires a well-defined scale and the model must not be allowed to arbitrarily stretch. But even if local identification is guaranteed, it would still be possible to, third, symmetrically rotate the model in the parameter space by inverting all parameters.
Bayesian estimators address identification challenges by specifying prior distributions (Gelman and Hill Reference Gelman and Hill2007; Jackman Reference Jackman2009). Following Barberá (Reference Barberá2015), we solve local identification of our model with a standard normal distribution for
$ {\theta}_i $
and with a normal distribution with a mean at 0 and standard deviation
$ {\sigma}_{\alpha } $
for
$ {\alpha}_j $
.
$$ {\alpha}_j\sim \hskip0.6em N\left(0,{\sigma}_{\alpha}\right) $$
$$ {\beta}_i\sim \hskip0.6em N\left({\mu}_{\beta },{\sigma}_{\beta}\right) $$
$$ {\phi}_j\sim \hskip0.48em N\left({\mu}_{\phi },{\sigma}_{\phi}\right) $$
$$ {\theta}_i\sim \hskip0.48em N\left(0,1\right). $$
Global identification is more challenging. Political scientists who estimate locations of political actors and legislative proposals in a common space face a similar problem when they repurpose item response theory (e.g., Clinton, Jackman and Rivers Reference Clinton, Jackman and Rivers2004; Poole and Rosenthal Reference Poole and Rosenthal2007). They address rotational invariance in at least two ways. Authors globally identify their model. Highly informative priors on a well-known political actor clearly determine who belongs to “the right” or “the left” (Clinton, Jackman and Rivers Reference Clinton, Jackman and Rivers2004; Martin and Quinn Reference Martin and Quinn2002). In a similar vein, in the case of multidimensional scaling where prior intuitions about political actors might not be as well defined, Jackman (Reference Jackman2001) proposes setting priors on well-understood legislative proposals instead. However, fixing certain decisions or legal sources is not possible in our context. Ex ante, an analyst is neither certain about a clear location of certain decisions nor about the location of the cited legal sources. Avoiding any unjustified bias from priors, the model should identify the parameters on the basis of the citation data only.
The other option to address rotational invariance is to not identify a model globally at all. Since the model can flip, the respective posterior distribution can have two modes. While in theory the sampler could visit both sides, in practice this concern often turns out not to be an issue as long as there is a reasonably large number of latent positions whose locations can be reasonably well distinguished (Jackman Reference Jackman2001, Reference Jackman2004). It is then sufficient to initialize the sampler around an educated guess. Even with symmetric – and thus for rotational invariance uninformative – priors the algorithm quickly converges to the correct posterior (Jackman Reference Jackman2009). Barberá (Reference Barberá2015) follows this latter approach in his original implementation of the model we are building on here. We find, however, that in our application a sampler can indeed flip sides, which is why we have to resort to a different strategy.
We use STAN to run an initial model that identifies the two most extreme decisions as anchors for the second run. For this first run, we use one single chain that is not identified with regard to rotational invariance and calculate the median of all parameters,
$ \theta $
. For the second run – the “proper” estimation – we use these two anchors to identify rotational invariance. If in a draw the right anchor ends up to the left of the left anchor, we have reason to believe that the model flipped and therefore multiply all
$ \theta $
parameters with
$ -1 $
. In this second run, we draw overall
$ {8}^{\prime }000 $
times from the posterior across multiple chains to effectively explore its central tendency and variation.Footnote 8
Forum shopping and forum selling in Germany’s lower courts
Are some lower courts systematically more plaintiff friendly than other courts? We now study decisions by German Landgerichte (district courts) and Oberlandesgerichte (regional courts of appeal) with our model to see whether we can uncover systematic evidence for forum shopping and forum selling. While it has been documented that plaintiffs engage in forum shopping – and judges in forum selling – if respective institutional incentives are in place (e.g., Bechtold, Frankenreiter and Klerman Reference Bechtold, Frankenreiter and Klerman2019), Germany is a particularly unlikely case for this kind of behavior: Its civil law system is known to generate decisions in a bureaucratic way and most importantly, without recording the individual votes of the participating judges who can be seen as apolitical career civil servants. As a matter of fact, Germany is often cited as the exemplary case of an impartial and objective civil law system (Langbein Reference Langbein1985). Extending research beyond the well-known U.S. context, Bechtold, Frankenreiter and Klerman (Reference Bechtold, Frankenreiter and Klerman2019) interview attorneys, judges and court officials and document mechanisms through which forum selling in fact also occurs in Germany. In areas like press law and antitrust where German citizens can choose the court they want to file their case with, plaintiffs indeed strategically “shop” for the forums that suit their purpose. In return, courts who wish to establish themselves as a go-to-place in a certain legal area rely on different means to be particularly appealing. Earning a reputation for a tendency in rulemaking is a successful strategy when “selling” their own forum – anecdotal evidence also echoed in recent press reports (Dahlkamp and Schmid Reference Dahlkamp and Barbara2014; van Lijnden Reference van Lijnden2016). Following the lead of Bechtold, Frankenreiter and Klerman (Reference Bechtold, Frankenreiter and Klerman2019), we therefore decide to study case-spaces in press law and antitrust. Our quantitative analysis will test expectations generated by their case studies on a large sample of decisions.
Collecting data for a common case-space
How does one identify an appropriate set of court decisions that can be compared to one another? And how does one turn citation patterns into a document-source matrix that can be fed into the measurement model? Typically, an analyst would first have to curate cases that relate to the exact same legal topic and the same regulatory context – after all, each decision needs in theory to be able to refer to each legal source. She then identifies and counts all legal sources the judge was citing and turns the result into a table: Each row corresponds to a decision and each column carries the count for the respective citations of a legal source. To save the effort of relying on manual labor for this task, we show how to employ technology and, at the same time, keep this process transparent and replicable. We use methods from information retrieval for collecting a well-defined set of decisions, then identify the citations, and finally generate the decision-source matrix.
Our data are from the legal database Juris,Footnote 9 which comprises a comprehensive digital collection of all available German court decisions. Each of Juris’ records contains not only the complete text but also further metadata such as titles, dates or the respective court. The data provide information on the further fate of a legal proceeding within the judicial hierarchy, all citations a court decision makes (backward citation), as well as other court decisions that refer to a certain decision (forward citation). The annotation of the corpus of decisions with metadata makes the further processing particularly easy.
Granted access to the backend of Juris, we worked with a database (MongoDB) that was indexed with a Lucene-based search engine (ElasticSearch). This infrastructure allowed us to make full use of search engine functionalities and query this corpus as we saw fit. While we collected the data for cases on press law with this infrastructure, we also used a second approach in the antitrust case for the sake of reproducibility. There, we accessed the data through the front end, working with Juris’ search functionality of their homepage.
First, we need to define the set of decisions that belong to the common case-space. For the application to press law, we begin with selecting key words that describe the two areas we are interested in: decisions regarding claims for compensation and decisions concerned with claims for injunction.Footnote 10 Acknowledging human limitations in devising dictionaries for direct document selection (e.g., Beauchamp Reference Beauchamp2017; King, Pan and Roberts Reference King, Pan and Roberts2013; Puglisi and Snyder Jr Reference Puglisi and Snyder2011), we strive to mitigate this bias and cast a fairly wide set of terms to retrieve a large collection of 100 documents. A trained human coder then uses reasoning to identify those cases that truly belong to a common case-space. In result, we select nine cases on privacy infringements through the publication of photos in print media and a respective claim for compensation (dataset
$ {d}_{1A} $
). We also collect six decisions on privacy infringement through criminal act allegation in the media and the corresponding claim for injunction (dataset
$ {d}_{2A} $
).Footnote 11
In the light of these fairly small sets, we seek to reliably enlarge our samples. We take the titles of the decisions from the two already identified sets
$ {d}_{1A} $
and
$ {d}_{2A} $
as query terms.Footnote 12 The search engine identifies relevant decisions on the basis of the cosine similarity between the titles of the decisions in the seed sets and the full text of the decisions in the database.Footnote 13 We find that the top 25 most similar results reliably belong to a common case-space and use those decisions as the enlarged set for claims for compensation (dataset
$ {d}_{1B} $
) and enlarged set of decisions on privacy infringement through criminal act allegation in the media and the corresponding claim for injunction (dataset
$ {d}_{2B} $
).Footnote 14
For antitrust law, we proceed in a similar way – this time, however, we access the database through the Juris online portal. We search with key words on antitrustFootnote 15 for decisions from German district courts (Landgerichte). Again, a trained coder reviewed all resulting 64 decisions and clustered them into reasonable topics. We identified 20 decisions in the common case-space regarding damage claims following from a cartel’s action (dataset
$ {d}_3 $
). Given the size of this set of decisions, we decide to not further enlarge our sample.
Once the decisions in each set are defined, identifying the references is straightforward: Juris already enriched all decisions with metadata. When accessing the original Juris data for press law with our own search engine, all sources of a decision are available as a list. In turn, when querying the data via Juris’ homepage, downloading each decision’s HTML page allows extracting citations on the basis of embedded hyperlinks.Footnote 16
As a third and final step, we count the sources in all court decisions. We define a legal source by its section (Paragraph) and paragraph (Absatz). The same law and section, but with a different paragraph counts as another legal source. Legal sources can be references to German civil code (e.g., a particular section of the BGB), to criminal law (e.g., a section in the StGB), to code of civil procedure (e.g., a section in the ZPO) or to a section in the German constitution (the Grundgesetz, GG), and also previous decisions of both lower courts (e.g., a decision written at the OLG Hamburg) and higher courts such as the Federal Court of Justice (BGH), or the German Federal Constitutional Court (BVerfG) as a specialized court of higher order. We also consider mentions of academic articles. Using the metadata, we then construct the respective decision-source matrix
$ {Y}_{ij} $
that identifies all referenced legal sources (
$ j $
) for each preselected decision (
$ i $
). When counting the sources, we leave out all citations to idiosyncratic legal sources, meaning any legal source that is mentioned by only one decision.Footnote 17
Application to press law
Let us now analyze the court decisions in press law. Jürgens (Reference Jürgens2014) gathers data on press law caseload at Germany’s regional courts. He finds that, between 2010 and 2012, Berlin (28.67%), Hamburg (22.32%) and Cologne (Köln) (11.94%) adopt the bulk of all court decisions, with the rest of the caseload shared among all other courts. Authors disagree on how to interpret these data. Some argue that there are systematic tendencies in rulemaking favoring plaintiffs who defend themselves against the press at these three places (Dahlkamp and Schmid Reference Dahlkamp and Barbara2014; Höch Reference Höch2018; Jürgens Reference Jürgens2014, Reference Jürgens2016; Kompa Reference Kompa2012; Sajuntz Reference Sajuntz2014; van Lijnden Reference van Lijnden2016). The high caseload is a result of plaintiffs who – aware of these tendencies – select their courts strategically. Adding to the data from caseloads, authors substantiate their claims with anecdotal evidence from either the court in Cologne (Dahlkamp and Schmid Reference Dahlkamp and Barbara2014; Jürgens Reference Jürgens2014, Reference Jürgens2016) or Hamburg (Höch Reference Höch2018; Jürgens Reference Jürgens2014, Reference Jürgens2016; Kompa Reference Kompa2012). Others contend that this interpretation is wrong. The high number of cases in Berlin, Hamburg or Cologne is rather the result of a concentration of media companies – and that includes an ecosystem of specialized press lawyers who can serve them (Dölling Reference Dölling2015; Höcker and Brost Reference Höcker and Brost2015). Anecdotal evidence suggests that this is the case in particular for Berlin (Dölling Reference Dölling2015). Answering the call for an analysis that “meets the standards of an empirical science” (Dölling Reference Dölling2015, 130, translation), we investigate these claims with our measurement model. Are the courts in Hamburg and Cologne really more likely to adopt decisions that are friendly to plaintiffs and hostile toward the press?
Estimates of decision locations
Based on these 25 most (cosine) similar decisions each, we generate the decision-source matrices for
$ {d}_{1B} $
and
$ {d}_{2B} $
, omitting idiosyncratic references. Figure 2 displays the median estimates with a circle, and the bars indicate the core 90% credible interval.Footnote 18 In the first set of court decisions – the collection on compensation – the courts in Cologne and Hamburg cluster on one end of the spectrum; all other courts can be found on the other end. Decisions on injunctions show more mixed positions. On the top of each figure, we present the mean difference between the positions of decisions from Cologne and Hamburg, in contrast to all others. As the credible intervals indicate, there is evidence in favor of a difference between the two groups.
Figure 2. Estimated locations of written decisions (
$ {\hat{\theta}}_i $
).
Notes: Set of decisions using a “more-like-this" query. On the top: mean difference between the decisions from courts in Cologne and Hamburg and all others. Points indicate the median of the posterior draws. The bars represent the central 90% credible interval.
In sum, we find evidence that lends itself to the experts’ and journalists’ expectations. In the context of the analyzed decisions in press law, lower courts throughout Germany show systematic differences in their judgments on comparable cases when it comes to compensation or injunction in privacy infringement cases.
A qualitative case study to assess the model validity
We now qualitatively assess the validity of our estimates and study three decisions from the extended set of cases on privacy infringement and the claim for compensation (
$ {d}_{1B} $
): Figure 3 charts two decisions that are scaled at similar positions on the latent dimension (LG Hamburg, 324 O 161/15 and LG Cologne, 28 O 466/14) and one decision that is scaled at the opposite end (LG Munich, 9 O 23075/07).Footnote 19 Our model predicts a high probability that the position of LG Munich, 9 O 23075/07 is different from LG Hamburg, 324 O 161/15 and LG Cologne, 28 O 466/14. In return, the probability that LG Hamburg, 324 O 161/15 and LG Cologne, 28 O 466/14 are different from one another is relatively low. If our approach is valid, then similar court decisions will derive their legal argument using similar legal sources and the court decision that is distinct should rely on different legal sources.
Figure 3. Estimated differences in the location (
$ {\hat{\theta}}_i $
) for Hamburg Decision 324 O 161/15, Cologne Decision 28 O 466/14 and Munich Decision 9 O 23075/07.
Notes: Points indicate the median of the difference of the respective posterior draws. The bars represent the central 95% credible interval of the difference of the respective posterior draws.
In the appendix, we report a full legal interpretation of the arguments and sources the three courts are using. Figure 4 summarizes our findings and illustrates that our scaling approach is indeed capable of differentiating between different nuances in the main body of the decisions, where courts argue the cases. The figure is read from top to bottom. Following the solid line (Hamburg decision), dashed line (Cologne decision) and dotted line (Munich decision) leads to the legal norms and the case law that are used to develop the judicial arguments over the course of the different decisions. In the initial parts of the decisions, there is overlap between the legal norms on which a case is based on. This is not surprising as these norms are presented by the plaintif to initiate the cases based on similar scenarios. However, the Hamburg and Cologne decisions argue based on similar legal norms and case law while the Munich decision refers to different sources than the other two decisions. Ultimately, referring to different norms and case law leads to different outcomes favoring either the plaintif or the defendant. This speaks to the validity of our scaling approach: The median scores estimated for the Hamburg and Cologne decision are located at similar ends of our common case-space while the Munich decision is placed at the opposite end (Figure 2).
Figure 4. Summary of legal argumentation in three decisions from the common case-space on privacy infringement and claims for compensation.
Assessment of model fit using posterior predictive checks
Finally, we also want to assess whether the model actually fits the data-generating process that we posit using posterior predictive checks. We therefore predict the outcome variable on the basis of the last 50 parameter draws and plot the predictions against the original data. Figure 5 displays the results for all models. Each figure represents the predicted citation counts
$ {\hat{y}}_{ij} $
for all decision-source pairs as a histogram. The figures chart the respective counts horizontally and show the square of their occurrence vertically.Footnote 20 Credible intervals are at the core 80%, 60%, 40% and 20% of the distribution. The solid lighter line depicts the median prediction across all models. The darker line plots the observed data. Our predictions mostly match the observed data, and we conclude that the models do a good job in predicting the observed counts. We are thus confident that our systematic and our stochastic component correctly model the true underlying data-generating citation count process.
Figure 5. Posterior predictive checks.
Notes: The figures visualize the citation counts of all possible decision source pairs in the form of a histogram. Counts are displayed horizontally and their respective frequencies are shown vertically. Predictions in red are based on the last 50 parameter draws of the sampling chain with credible intervals at 80%, 60%, 40% and 20%. Median predicted counts are represented with a red line, the observed data with a blue line.
Application to antitrust law
Antitrust is yet another area in German law that offers incentives for plaintiffs or courts to behave strategically. Plaintiffs can file their case either at the seat of a cartel member or at the place where mischief occurred. Often, cartels operate nationally – which means that the plaintiff can chose from the overall 24 regional courts when filing the complaint. Courts themselves are interested in attracting cases for their highly visible and exciting nature.
Bechtold, Frankenreiter and Klerman (Reference Bechtold, Frankenreiter and Klerman2019) indeed find some evidence in favor of forum selling for the regional court in Mannheim. However, they are more cautious in their conclusions with regard to other cases. According to their interviews, the three regional courts in Cologne, Dortmund and Mannheim are particularly plaintiff friendly. In contrast, there are other courts such as Kiel, Leipzig, Düsseldorf or Munich that are less so.
Figure 6 plots the result for our 20 decisions. The courts in Mannheim and Dortmund tend to adopt decisions that are on one side of the spectrum. Decisions from Cologne seem to have a more central position in our sample. Other courts either occupy the center or display estimated locations of decisions on the center right. The small plot on the top charts the difference between the decisions from courts in Cologne, Dortmund and Mannheim versus all other courts. The results support the existing qualitative evidence. Our results show that the mean differences between decisions of those three courts and the other courts for the common case-space in antitrust is systematic.
Figure 6. Estimated locations (
$ {\hat{\theta}}_i $
) of written decisions on antitrust.
Notes: On the top: mean difference between the decisions from courts in Cologne, Dortmund and Mannheim and all others. Points indicate the median of the posterior draws. The bars represent the central 90% credible interval.
Conclusion
Legal scholars and political scientists are not only interested in understanding the decisions judges make but also seek to analyze the policy implications of written decisions. Previous strategies usually rely on voting data of the involved judges (e.g., Clark and Lauderdale Reference Clark and Lauderdale2010). But in cases where no such records exists, empirical strategies to scale decisions in a common space are still missing. We show how to estimate and consequently compare positions of various actors in a common case-space of court decisions using citations networks. Our model rests on the notion that “similar” decisions, that is, decisions with similar locations in a common space, express their similar legal reasoning through similar citation patterns.Footnote 21
To showcase the abilities of our model, we study a particularly challenging case: the German legal system. Qualitative evidence indicates that German lower courts have a systematic bias in some legal areas, engaging in forum selling in press law and antitrust (Klerman and Reilly Reference Klerman and Reilly2016; Bechtold, Frankenreiter and Klerman Reference Bechtold, Frankenreiter and Klerman2019). We corroborate these findings. Indeed, there is evidence that some lower courts – in press law Cologne and Hamburg and for antitrust Cologne, Dortmund and Mannheim – do make systematically biased decisions. We can also replicate our findings for different sets of decisions. In addition, we show how to find appropriate decisions in a large legal corpus that all belong to a common case-space, thus leading the way to analyzing larger sets of court decisions without human intervention.
Of course, there are limits to what our model can do. Just like any other item-response theory approach – for example, when locating parliamentarians on a left-right spectrum – all units that are being scaled need to reside in the same latent space. For our legal application, this means that all decisions have to concern the same legal matter, and this also includes a constant regulatory context: Any major legislative change that affects the legal sources available to the judge is likely to introduce bias. Future research might build on our model to automate the process of identifying decisions that belong to a common case-space even further. Moreover, although our model makes moderate assumptions about how courts refer to legal sources to justify their legal arguments, the evidence we provide comes only from one country, Germany, which is a typical civil law country. Future research might be able to provide more evidence that this model is useful to locate and compare court decisions within an appropriate case-space in other contexts as well.
Our model enriches the toolbox for an interdisciplinary group of scholars who study judges and their decisions quantitatively. Legal scholars might be interested in unveiling the development of doctrine across time and space. Political scientists might want to understand the nexus between institutions and power. Others, like economists or sociologists, might rather be concerned with the effects of court decisions on business or societies at large. Common space positions of court decisions are the foundation for the development and testing of powerful analytical models and may find their application in any of these fields.
Lastly, our insights have the potential to be useful beyond academia. Locating written lower court decisions in a common space has also important practical implications for law firms. Suppose a lawyer who is in favor of an extreme position in an ongoing case is trying to build an argument. Using our scaling model, she can obtain a quick overview over the tendency in each ruling she selects. This not only helps her identify the locations of courts who are in favor of her own opinion, but she could also study the arguments of the opposite side to be able to anticipate and preempt them. Thus, locating actual decisions in a common case-space is very helpful for all who want to quickly spot similar or opposing decisions. With our model, they can retrieve an easy roadmap capable of guiding them in their professional effort.
Supplementary Materials
To view supplementary material for this article, please visit https://doi.org/10.1086/717420
Acknowledgments
We thank Pablo Barberá, Jens Frankenreiter, J. Andrew Harris, Kevin Quinn, Nils Schaks, Philip Schroeder and Alexander Tischbirek for helpful comments. We also thank Felicia Riethmüller for her insightful and instrumental research assistance. Arnold is grateful for support through Cardiff’s “Darlithwyr Disglair Development Scheme.” Engst and Gschwend acknowledge financial support for SFB 884 (project C4) at the University of Mannheim from the German Research Foundation (DFG). All data and code are available at the Journal’s Dataverse https://dataverse.harvard.edu/dataverse/jlc.
Open access
