2.1 New Technologies and the Rise of the Algorithmic Society

New technologies offer human agents entirely new ways of doing things.Footnote ¹ However, as history shows, ‘practical’ innovations always bring with them more significant changes. Each new option introduced by technological evolution allowing new forms affects the substance, eventually changing the way humans think and relate to each other.Footnote ² The transformation is especially true when we consider information and communication technologies (so-called ICT); as indicated by Marshall McLuhan, ‘the media is the message’.Footnote ³ Furthermore, this scenario has been accelerated by the appearance of artificial intelligence systems (AIS), based on the application of machine learning (ML).

These new technologies not only allow people to find information at an incredible speed; they also recast decision-making processes once in the exclusive remit of human beings.Footnote ⁴ By learning from vast amounts of data – the so-called Big Data – AIS offer predictions, evaluations, and hypotheses that go beyond the mere application of pre-existing rules or programs. They instead ‘induce’ their own rules of action from data analysis; in a word, they make autonomous decisions.Footnote ⁵

We have entered a new era, where big multinational firms (called ‘platforms’) use algorithms and artificial intelligence to govern vast communities of people.Footnote ⁶ Conversely, data generated by those platforms fuel the engine of the ‘Algorithmic Society’.Footnote ⁷

From this point of view, the Algorithmic Society is a distinctive evolution of the ‘Information Society’,Footnote ⁸ where a new kind of ‘mass-surveillance’ becomes possible.Footnote ⁹

This progress generates a mixture of excitement and anxiety.Footnote ¹⁰ The development of algorithms and artificial intelligence technologies is becoming ubiquitous, omnipresent, and seemingly omnipotent. They promise to eliminate our errors and make our decisions better suited for any purpose.Footnote ¹¹

In this perspective, a relatively old prophecy, predicted by Herbert Marcuse in one of the ‘red books’ of that massive socio-political movement usually known as ‘1968’, The One-Dimensional Man, becomes reality. Marcuse starts the first page of that seminal book as follows:

If technology replaces all ‘socially necessary but painful performances’ – work included – personal freedom reaches its final fulfilment (that is, its very end). In Marcuse’s eyes, this is how technological power will take over our freedom and political system: not through a bloody ‘coup’ but by inducing people – practically and happily – to give up all their responsibilities.

However, this dystopic perspective – a future of ‘digital slavery’, where men and women will lose their liberty and quietly reject all democratic principlesFootnote ¹³ – produces a reaction. It is not by chance that the European Commission’s strategically endorsing the transformation of the EU into an AI-led economy, at the same time, requires great attention to people’s trust and a high level of fundamental rights protection.Footnote ¹⁴

One of the most common areas where we experience the rise of these concerns is public and private security.Footnote ¹⁵ For a large part of the 2010s onward, technological innovations have focused on safety and control; the consequence has been an alarming increase in public and private surveillance, coupled with growing threats to political and civil liberties.Footnote ¹⁶ In addition to this, the global ‘COVID-19’ pandemic has doubtlessly boosted the already fast-growing ‘surveillance capitalism’.Footnote ¹⁷

While at the beginning of the twenty-first century, there was an increasing awareness of the risks of the new pervasive surveillance technologies, today, hit by the pandemic and searching for practical tools to enforce social distancing or controlling policies, the general institutional and academic debate seems to be less worried by liberty-killing effects and more allured by health-preserving results.Footnote ¹⁸

Regardless, the most worrying challenges stem from the increasing power of algorithms, created through Big Data analytics such as machine learning and used to automate decision-making processes.Footnote ¹⁹ Their explicability,Footnote ²⁰ liability, and culpability are still far from being clearly defined.Footnote ²¹ As a consequence, several scholars and policymakers are arguing, on the one hand, to aggressively regulate tech firmsFootnote ²² (since classic antitrust law is unfit for this purpose) or, on the other, to require procedural safeguards, allowing people to challenge the decisions of algorithms which can have significant consequences on their lives (such as credit score systems).Footnote ²³

2.2 The Impact of the Algorithmic Society on Constitutional Law

As we know, at its very origin, constitutional theory wrestles with the problem of power control.Footnote ²⁴ Scholars commonly consider constitutional law that part of the legal system whose function is to legallyFootnote ²⁵ delimit power.Footnote ²⁶ In the ‘modern sense’,Footnote ²⁷ this discipline establishes rules or builds institutions capable of shielding personal freedoms from external constraints.Footnote ²⁸ According to this idea, constitutionalism historically always ‘adapted’ itself to power’s features; that is to say, the protection of freedoms in constitutions has been shaped following the evolving character of the threats to those same freedoms.Footnote ²⁹

At the beginning of the modern era, the power to be feared was the king’s private force.Footnote ³⁰ The idea of ‘sovereignty’, which appeared at the end of the Middle Ages, had its roots in the physical and military strength of the very person of the ‘Sovereign’.Footnote ³¹ Sovereignty evoked an ‘external power’Footnote ³² grounded on the monopoly (actual or potential) of the physical ‘force’Footnote ³³ used against individuals or communities (e.g., ‘military force’ or the ‘force of law’).Footnote ³⁴ Consequently, liberties were those dimensions of human life not subjected to that power (e.g., habeas corpus). As the offspring of the French and American Revolutions, the ‘rule of law’ doctrine was the main legal tool ‘invented’ by constitutional theory to delimit the king’s power and protect personal freedom and rights. To be ‘legitimate’, any power has to be subjected to the rule of law.

The other decisive turning point in the history of constitutionalism was World War II and the end of twentieth-century European totalitarian regimes. It may sound like a paradox, but those regimes showed that the ‘legislative state’, built on the supremacy of law and therefore exercising a ‘legitimate power’, can become another terrible threat to human freedom and dignity.

If the law itself has no limits, whenever it ‘gives’ a right, it can ‘withdraw’ it. This practice is the inhuman history of some European twentieth-century states that cancelled human dignity ‘through the law’.

With the end of World War II, a demolition process of those regimes began, and learning from the American constitutional experience, Europe transformed ‘flexible’ constitutions – until then, mere ordinary laws – into ‘rigid’ constitutions,Footnote ³⁵ which are effectively the ‘supreme law’ of the land.Footnote ³⁶

In this new scenario, the power that instils fear is no longer the king’s private prerogative; the new limitless force is the public power of state laws, and the constitutional tool intended to effectively regulate that power is vested in the new ‘rigid’ constitution: a superior law, ‘stronger’ than ordinary statutes and thus truly able to protect freedoms, at least apparently, even against legal acts.

With the turn of the twenty-first century, we witness the rise of a new kind of power. The advent of new digital technologies, as discussed previously, provides an unprecedented means of limiting and directing human freedom that has appeared on the global stage; a way based on not an ‘external’ force (as in the two previous constitutional scenarios, the private force of the king or the public ‘force of the law’) but rather an ‘internal’ force, able to affect and eventually substitute our self-determination ‘from inside’.Footnote ³⁷

This technological power is at the origin of ‘platform capitalism’,Footnote ³⁸ which is a vast economic transformation induced by the exponentially fast-growing markets of Internet-related goods and services – for example, smart devices (Apple, Samsung, Huawei, Xiaomi), web-search engines (Google), social media corporations (Facebook, Instagram, Twitter), cloud service providers (Amazon, Microsoft, Google), e-commerce companies (Amazon, Netflix), and social platforms (Zoom, Cisco Webex).

Consider that today,Footnote ³⁹ the combined value of the S&P 500’s five most prominent companiesFootnote ⁴⁰ now stands at more than $7 trillion, accounting for almost 25 per cent of the market capitalization of the index, drawing a picture of what a recent doctrine accurately defined as a ‘moligopoly’.Footnote ⁴¹

These ‘moligopolists’Footnote ⁴² are not only creating communities and benefitting from network effects generated by users’ transactions, but they also develop a de facto political authority and influence once reserved for legal and political institutions. More importantly, they are taking on configurations that are increasingly similar to the state and other public authorities.Footnote ⁴³ Their structure reflects a fundamental shift in the political and legal systems of Western democracies – what has been called a new type of ‘functional sovereignty’.Footnote ⁴⁴ Elsewhere we used the term ‘cybernetic power’,Footnote ⁴⁵ which perhaps sounds like an old-fashioned expression. Still, it is more accurate in its etymology (‘cyber’, from its original ancient Greek meaning,Footnote ⁴⁶ shares the same linguistic root as ‘govern’ and ‘governance’) to identify how automation and ICT have radically transformed our lives.

As algorithms begin to play a dominant role in the contemporary exercise of power,Footnote ⁴⁷ it becomes increasingly important to examine the ‘phenomenology’ of this new sovereign power and its unique challenges to constitutional freedoms.

2.3 The ‘Algorithmic State’ versus Fundamental Rights: Some Critical Issues

As already stated, the main force of algorithms is their practical convenience, so their interference with our freedom is not perceived as an ‘external’ constraint or a disturbing power. Instead, it is felt as evidence-based support for our decisions, capturing our autonomy by lifting our deliberation burden.

Who would like to switch back to searching for information in volumes of an encyclopaedia? Who would want to filter their email for spam manually anymore? Who would like to use manual calculators instead of a spreadsheet when doing complex calculations? We are not just living in an increasingly automated world; we are increasingly enjoying the many advantages that come with it. Public administrations are using more and more algorithms to help public-sector functions, such as welfare, the labour market, tax administration, justice, crime prevention, and more. The use of algorithms in decision-making and adjudications promises more objectivity and fewer costs.

However, as we said, algorithms have a darker side, and the following chapters of this section of the book illustrate some of the facets of the Algorithmic State phenomenology.

The fast-growing use of algorithms in the fields of justice, policing, public welfare, and the like could end in biased and erroneous decisions, boosting inequality, discrimination, unfair consequences, and undermining constitutional rights, such as privacy, freedom of expression, and equality.Footnote ⁴⁸

And these uses raise considerable concerns not only for the specific policy area in which they are operated but also for our society as a whole.Footnote ⁴⁹ There is an increasing perception that humans do not have complete control over Algorithmic State decision-making processes.Footnote ⁵⁰ Despite their predictive outperformance over analogue tools, algorithmic decisions are difficult to understand and explain (the so-called black box effect).Footnote ⁵¹ While producing highly effective practical outcomes, algorithmic decisions could undermine procedural and substantive guarantees related to democracy and the rule of law.

Issues related to the use of algorithms as part of the decision-making process are numerous and complex, but at the same time, the debate is at an early stage. However, efforts towards a deeper understanding of how algorithms work when applied to legally tricky decisions will be addressed soon.

In this section, we will examine four profiles of the use of algorithmic decisions: the relation between automation and due process, the so-called ‘emotional’ AI, the algorithmic bureaucracy, and predictive policing.

2.4 The Effects of the ‘Algorithmic State’ on the Practice of Liberty

Trying to synthetize some of the most critical issues brought about by the advent of what we call the Algorithmic State on the practice of constitutional liberties, there appear to be two main sensitive areas: surveillance and freedom.

3.1 Introduction

Automation is influencing ever more fields of law. The dream of disruption has permeated the US and British legal academies and is making inroads in Australia and Canada, as well as in civil law jurisdictions. The ideal here is law as a product, simultaneously mass producible and customizable, accessible to all and personalized, openly deprofessionalized.Footnote ¹ This is the language of idealism, so common in discussions of legal technology – the Dr. Jekyll of legal automation.

But the shadow side of legal tech also lurks behind many initiatives. Legal disruption’s Mr. Hyde advances the cold economic imperative to shrink the state and its aid to the vulnerable. In Australia, the Robodebt system of automated benefit overpayment adjudication clawed back funds from beneficiaries on the basis of flawed data, false factual assumptions, and misguided assumptions about the law. In Michigan, in the United States, a similar program (aptly named “MIDAS,” for Michigan Integrated Data Automated System) “charged more than 40,000 people, billing them about five times the original benefits” – and it was later discovered that 93 percent of the charges were erroneous.Footnote ² Meanwhile, global corporations are finding the automation of dispute settlement a convenient way to cut labor costs. This strategy is particularly tempting on platforms, which may facilitate millions of transactions each day.

When long-standing appeals to austerity and business necessity are behind “access to justice” initiatives to promote online dispute resolution, some skepticism is in order. At the limit, jurisdictions may be able to sell off their downtown real estate, setting up trusts to support a rump judicial system.Footnote ³ To be sure, even online courts require some staffing. But perhaps an avant-garde of legal cost cutters will find some inspiration from US corporations, which routinely decide buyer versus seller disputes in entirely opaque fashion.Footnote ⁴ In China, a large platform has charged “citizen juries” (who do not even earn money for their labor but, rather, reputation points) to decide such disputes. Build up a large enough catalog of such encounters, and a machine learning system may even be entrusted with deciding disputes based on past markers of success.Footnote ⁵ A complainant may lose credibility points for nervous behavior, for example, or gain points on the basis of long-standing status as someone who buys a great deal of merchandise or pays a taxes in a timely manner.

As these informal mechanisms become more common, they will test the limits of due process law. As anyone familiar with the diversity of administrative processes will realize, there is an enormous variation at present in how much opportunity a person is entitled to state their case, to demand a written explanation for a final (or intermediate) result, and to appeal. A black lung benefits case differs from a traffic violation, which in term differs from an immigration case. Courts permit agencies a fair amount of flexibility to structure their own affairs. Agencies will, in all likelihood, continue to pursue an agenda of what Julie Cohen has called “neoliberal managerialism” as they reorder their processes of investigation, case development, and decision-making.Footnote ⁶ That will, in turn, bring in more automated and “streamlined” processes, which courts will be called upon to accommodate.

While judicial accommodations of new agency forms are common, they are not automatic. At some point, agencies will adopt automated processes that courts can only recognize as simulacra of justice. Think, for instance, of an anti-trespassing robot equipped with facial recognition, which could instantly identify and “adjudicate” a person overstepping a boundary and text that person a notice of a fine. Or a rail ticket monitoring system that would instantly convert notice of a judgment against a person into a yearlong ban on the person buying train tickets. Other examples might be less dramatic but also worrisome. For example, consider the possibility of “mass claims rejection” for private health care providers seeking government payment for services rendered to persons with government-sponsored health insurance. Such claims processing programs may simply compare a set of claims to a corpus of past denied claims, sort new claimants’ documents into categories, and then reject them without human review.

In past work, I have explained why legislators and courts should reject most of these systems, and should always be wary of claims that justice can be automated.Footnote ⁷ And some initial jurisprudential stirrings are confirming that normative recommendation. For example, there has been a backlash against red-light cameras, which automatically cite drivers for failing to obey traffic laws. And even some of those who have developed natural language processing for legal settings have cautioned that they are not to be used in anything like a trial setting. These concessions are encouraging.

And yet there is another danger lurking on the horizon. Imagine a disability payment scheme that offered something like the following “contractual addendum” to beneficiaries immediately before they began receiving benefits:

There are two diametrically opposed ways of parsing such a contract. For many libertarians, the right to give up one’s rights (here, to a certain level of privacy and appeals) is effectively the most important right, since it enables contracting parties to eliminate certain forms of interference from their relationship. By contrast, for those who value legal regularity and due process, this “addendum” is anathema. Even if it is possible for the claimant to re-apply after a machine learning system has stripped her of benefits, the process offends the dignity of the claimant. A person must pass on whether such a grave step is to be taken.

These divergent approaches are mirrored in two lines of US Supreme Court jurisprudence. On the libertarian side, the Court has handed down a number of rulings affirming the “right” of workers to sign away certain rights at work, or at least the ability to contest their denial in court.Footnote ⁹ Partisans of “disruptive innovation” may argue that startups need to be able to impose one-sided terms of service on customers, so that investors will not be deterred from financing them. Exculpatory clauses have spread like kudzu, beckoning employers with the jurisprudential equivalent of a neutron bomb: the ability to leave laws and regulations standing, without any person capable of enforcing them.

On the other side, the Supreme Court has also made clear that the state must be limited in the degree to which it can structure entitlements when it is seeking to avoid due process obligations. A state cannot simply define an entitlement to, say, disability benefits, by folding into the entitlement itself an understanding that it can be revoked for any reason, or no reason at all. On this dignity-centered approach, the “contractual addendum” posited above is not merely one innocuous add-on, a bit of a risk the claimant must endure in order to engage in an arms’ length exchange for $20. Rather, it undoes the basic structure of the entitlement, which included the ability to make one’s case to another person and to appeal an adverse decision.

If states begin to impose such contractual bargains for automated administrative determinations, the “immoveable object” of inalienable due process rights will clash with the “irresistible force” of legal automation and libertarian conceptions of contractual “freedom.” This chapter explains why legal values must cabin (and often trump) efforts to “fast track” cases via statistical methods, machine learning (ML), or artificial intelligence. Section 3.2 explains how due process rights, while flexible, should include four core features in all but the most trivial or routine cases: the ability to explain one’s case, a judgment by a human decision maker, an explanation for that judgment, and the ability to appeal. Section 3.3 demonstrates why legal automation often threatens those rights. Section 3.4 critiques potential bargains for legal automation and concludes that the courts should not accept them. Vulnerable and marginalized persons should not be induced to give up basic human rights, even if some capacious and abstract versions of utilitarianism project they would be “better off” by doing so.

3.2 Four Core Features of Due Process

Like the rule of law, “due process” is a multifaceted, complex, and perhaps even essentially contested concept.Footnote ¹⁰ As J. Roland Pennock has observed, the “roots of due process grow out of a blend of history and philosophy.”Footnote ¹¹ While the term itself is a cornerstone of the US and UK legal systems, it has analogs in both public law and civil law systems around the world.

While many rights and immunities have been evoked as part of due process, it is important to identify a “core” conception of it that should be inalienable in all significant disputes between persons and governments. We can see this grasping for a “core” of due process in some US cases, where the interest at stake was relatively insignificant but the court still decided that the person affected by government action had to have some opportunity to explain him or herself and the contest the imposition of a punishment. For example, in Goss v. Lopez, students who were accused of misbehavior were suspended from school for ten days. The students claimed they were due some kind of hearing before suspension, and the Supreme Court agreed:

This is a fair encapsulation of some core practices of due process, which may (as the stakes rise) become supplemented by all manner of additional procedures.Footnote ¹³

One of the great questions raised by the current age of artificial intelligence (AI) is whether the notice and explanation of the charges (as well as the opportunity to be heard) must be discharged by a human being. So far as I can discern, no ultimate judicial authority has addressed this particular issue in the due process context. However, given that the entire line of case law arises in the context of humans confronting other humans, it does not take a stretch of the imagination to imagine such a requirement immanent in the enterprise of due process.

Moreover, legal scholars Kiel Brennan-Marquez and Henderson argue that “in a liberal democracy, there must be an aspect of ‘role-reversibility’ to judgment. Those who exercise judgment should be vulnerable, reciprocally, to its processes and effects.”Footnote ¹⁴ The problem with robot or AI judges is that they cannot experience punishment the way that a human being would. Role-reversibility is necessary for “decision-makers to take the process seriously, respecting the gravity of decision-making from the perspective of affected parties.” Brennan-Marquez and Henderson derive this principle from basic principles of self-governance:

Thus, for Brennan-Marquez and Henderson, “even assuming role-reversibility will not improve the accuracy of decision-making; it still has intrinsic value.”

Brennan-Marquez and Henderson are building on a long tradition of scholarship that focuses on the intrinsic value of legal and deliberative processes, rather than their instrumental value. For example, applications of the US Supreme Court’s famous Mathews v. Eldridge calculus have frequently failed to take into account the effects of abbreviated procedures on claimants’ dignity.Footnote ¹⁵ Bureaucracies, including the judiciary, have enormous power. They owe litigants a chance to plead their case to someone who can understand and experience, on a visceral level, the boredom and violence portended by a prison stay, the “brutal need” resulting from the loss of benefits (as put in Goldberg v. Kelly), the sense of shame that liability for drunk driving or pollution can give rise to. And as the classic Morgan v. United States held, even in complex administrative processes, the one who hears must be the one who decides. It is not adequate for persons to play mere functionary roles in an automated judiciary, gathering data for more authoritative machines. Rather, humans must take responsibility for critical decisions made by the legal system.

This argument is consistent with other important research on the dangers of giving robots legal powers and responsibilities. For example, Joanna Bryson, Mihailis Diamantis, and Thomas D. Grant have warned that granting robots legal personality raises the disturbing possibility of corporations deploying “robots as liability shields.”Footnote ¹⁶ A “responsible robot” may deflect blame or liability from the business that set it into the world. This is dangerous because the robot cannot truly be punished: it lacks human sensations of regret or dismay at loss of liberty or assets. It may be programmed to look as if it is remorseful upon being hauled into jail, or to frown when any assets under its control are seized. But these are simulations of human emotion, not the thing itself. Emotional response is one of many fundamental aspects of human experience that is embodied. And what is true of the robot as an object of legal judgment is also true of robots or AI as potential producers of such judgments.

3.3 How Legal Automation and Contractual Surrender of Rights Threaten Core Due Process Values

There is increasing evidence that many functions of the legal system, as it exists now, are very difficult to automate.Footnote ¹⁷ However, as Cashwell and I warned in 2015, the legal system is far from a stable and defined set of tasks to complete. As various interest groups jostle to “reform” legal systems the range of procedures needed to finalize legal determinations may shrink or expand.Footnote ¹⁸ There are many ways to limit existing legal processes, or simplify them, in order to make it easier for computation to replace or simulate them. The clauses mentioned previously – forswearing appeals of judgments generated or informed by machine learning or AI – would make non-explainable AI far easier to implement in legal systems.

This type of “moving the goalposts” may be accelerated by extant trends toward neoliberal managerialism in public administration.Footnote ¹⁹ This approach to public administration is focused on throughput, speed, case management, and efficiency. Neoliberal managerialists urge the public sector to learn from the successes of the private sector in limiting spending on disputes. One potential here is simply to outsource determinations to private actors – a move widely criticized elsewhere.Footnote ²⁰ I am more concerned here with a contractual option: to offer to beneficiaries of government programs an opportunity for more or quicker benefits, in exchange for an agreement not to pursue appeals of termination decisions, or to thereby accepting their automated resolution.

I focus on the inducement of quicker or more benefits, because it appears to be settled law (at least in the US) that such restrictions of due process cannot be embedded into benefits themselves. A failed line of US Supreme Court decisions once attempted to restrict claimants’ due process rights by insisting that the government can create property entitlements with no due process rights attached. On this reasoning, a county might grant someone benefits with the explicit understanding that they could be terminated at any time without explanation: the “sweet” of the benefits could include the “bitter” of sudden, unreasoned denial of them. In Cleveland Board of Education v. Loudermill (1985), the Court finally discarded this line of reasoning, forcing some modicum of reasoned explanation and process for termination of property rights.

What is less clear now is whether side deals might undermine the delicate balance of rights struck by Loudermill. In the private sector, companies have successfully routed disputes with employees out of process-rich Article III courts, and into stripped-down arbitral forums, where one might even be skeptical of the impartiality of decision-makers.Footnote ²¹ Will the public sector follow suit? Given some current trends in the foreshortening of procedure and judgment occasioned by public sector automation, the temptation will be great.

These concerns are a logical outgrowth of a venerable literature critiquing rushed, shoddy, and otherwise improper automation of legal decision-making. In 2008, Danielle Keats Citron warned that states were cutting corners by deciding certain benefits (and other) claims automatically, on the basis of computer code that did not adequately reflect the complexity of the legal code it claimed to have reduced to computation.Footnote ²² Virginia Eubanks’s Automating Inequality has identified profound problems in governmental use of algorithmic sorting systems. Eubanks tells the stories of individuals who lose benefits, opportunities, and even custody of their children, thanks to algorithmic assessments that are inaccurate or biased. Eubanks argues that complex benefits determinations are not something well-meaning tech experts can “fix.” Instead, the system itself is deeply problematic, constantly shifting the goal line (in all too many states) to throw up barriers to access to care.

A growing movement for algorithmic accountability is both exposing and responding to these problems. For example, Citron and I coauthored work setting forth some basic procedural protections for those affected by governmental scoring systems.Footnote ²³ The AI Now Institute has analyzed cases of improper algorithmic determinations of rights and opportunities.Footnote ²⁴ And there is a growing body of scholarship internationally exploring the ramifications of computational dispute resolution.Footnote ²⁵ As this work influences more agencies around the world, it is increasingly likely that responsible leadership will ensure that a certain baseline of due process values applies to automated decision-making.

Though they are generally optimistic about the role of automation and algorithms in agency decision-making, Coglianese and Lehr concede that one “due process question presented by automated adjudication stems from how such a system would affect an aggrieved party’s right to cross-examination. … Probably the only meaningful way to identify errors would be to conduct a proceeding in which an algorithm and its data are fully explored.”Footnote ²⁶ This type of examination is at the core of Keats Citron’s concept of technological due process. It would require something like a right to an explanation of the automated profiling at the core of decision.Footnote ²⁷

3.4 Due Process, Deals, and Unraveling

However, all such protections could be undone. The ability to explain oneself, and to hear reasoned explanations in turn, is often framed as being needlessly expensive. This expense of legal process (or administrative determinations) has helped fuel a turn to quantification, scoring, and algorithmic decision procedures.Footnote ²⁸ A written evaluation of a person (or comprehensive analysis of future scenarios) often requires subtle judgment, exactitude in wording, and ongoing revision in response to challenges and evolving situations. A pre-set formula based on limited, easily observable variables, is far easier to calculate.Footnote ²⁹ Moreover, even if individuals are due certain explanations and hearings as part of law, they may forego them in some contexts.

This type of rights waiver has already been deployed in some contexts. Several states in the United States allow unions to waive the due process rights of public employees.Footnote ³⁰ We can also interpret some Employee Retirement Income Security Act (ERISA) jurisprudence as an endorsement and approval of a relatively common situation in the United States: employees effectively signing away a right to a more substantive and searching review of adverse benefit scope and insurance coverage determinations via an agreement to participate in an employer-sponsored benefit plan. The US Supreme Court has gradually interpreted ERISA to require federal courts to defer to plan administrators, echoing the deference due to agency administrators, and sometimes going beyond it.Footnote ³¹

True, Loudermill casts doubt on arrangements for government benefits premised on the beneficiary’s sacrificing due process protections. However, a particularly innovative and disruptive state may decide that the opinion is silent as to the baseline of what constitutes the benefit in question, and leverage that ambiguity. Consider a state that guaranteed health care to a certain category of individuals, as a “health care benefit.” Enlightened legislators further propose that the disabled, or those without robust transport options, should also receive assistance with respect to transportation to care. Austerity-minded legislators counter with a proviso: to receive transport assistance in addition to health assistance, beneficiaries need to agree to automatic adjudication of a broad class of disputes that might arise out of their beneficiary status.

The automation “deal” may also arise out of long-standing delays in receiving benefits. For example, in the United States, there have been many complaints by disability rights groups about the delays encountered by applicants for Social Security Disability Benefits, even when they are clearly entitled to them. On the other side of the political spectrum, some complain that persons who are adjudicated as disabled, and then regain capacities to work, are able to keep benefits for too long after they regain the capacity to work. This concern (and perhaps some mix of cruelty and indifference) motivated British policy makers who promoted “fit for work” reviews by private contractors.Footnote ³²

It is not hard to see how the “baseline” of benefits might be defined narrowly, and all future benefits would be conditioned in this way. Nor are procedures the only constitution-level interest that may be “traded away” for faster access to more benefits. Privacy rights may be on the chopping block as well. In the United States, the Trump administration proposed reviews of the social media of persons receiving benefits.Footnote ³³ The presumption of such review is that a picture of, say, a self-proclaimed depressed person smiling, or a self-proclaimed wheelchair-bound person walking, could alert authorities to potential benefits fraud. And such invasive surveillance could again feed into automated review, which could be flagged by such “suspicious activity” in a way similar to the activation of investigation at US fusion centers by “suspicious activity reports.”

What is even more troubling about these dynamics is the way in which “preferences” to avoid surveillance or preserve procedural rights might themselves become new data points for suspicion or investigation. A policymaker may wonder about the persons who refuse to accept the new due-process-lite “deal” offered by the state: What have they got to hide? Why are they so eager to preserve access to a judge and the lengthy process that may entail? Do they know some discrediting fact about their own status that we do not, and are they acting accordingly? Reflected in the economics of information as an “adverse selection problem,” this kind of speculative suspicion may become widespread. It may also arise as a byproduct of machine learning: those who refuse to relinquish privacy or procedural rights may, empirically, turn out to be more likely to pose problems for the system, or non-renewal of benefits, than those who trade away those rights. Black-boxed flagging systems may silently incorporate such data points into their own calculations.

The “what have you got to hide” rationale leads to a phenomenon deemed “unraveling” by economists of information. This dynamic has been extensively analyzed by the legal scholar Scott Peppet. The bottom line of Peppet’s analysis is that every individual decision to reveal something about himself or herself may also create social circumstances that pressure others to also disclose. For example, if only a few persons tout their grade point average (GPA) on their resumes, that disclosure may merely be an advantage for them in the job-seeking process. However, once 30 percent, 40 percent, 50 percent, or more of job-seekers include their GPAs, human resources personnel reviewing the applications may wonder about the motives of those who do not. If they assume the worst about non-revealers, it becomes a rationale for all but the very lowest GPA holders to reveal their GPA. Those at, say, the thirtieth percentile, reveal their GPA to avoid being confused with those in the twentieth or tenth percentile, and so on.

This model of unraveling parallels similar theorizing in feminist theorizing. For example, Catharine Mackinnon insisted that the “personal is political,” in part because any particular family’s division of labor helped either reinforce or challenge dominant patterns.Footnote ³⁴ A mother may choose to quit work and stay home to raise her children, while her husband works fifty hours a week, and that may be an entirely ethical choice for her family. However, it also helps reinforce patterns of caregiving and expectations in that society which track women into unpaid work and men into paid work. It is not merely accommodating but also promoting gendered patterns of labor.Footnote ³⁵ Like a path through a forest trod ever clearer of debris, it becomes the natural default.

This inevitably social dimension of personal choice also highlights the limits of liberalism in addressing due process trade-offs. Civil libertarians may fight the direct imposition of limitations of procedural or privacy rights by the state. However, “freedom of contract” may itself be framed as a civil liberties issue. If a person in great need wants immediate access to benefits, in exchange for letting the state monitor his social network feed (and automatically terminate benefits if suspect pictures are posted), the bare rhetoric of “freedom” also pulls in favor of permitting this deal. We need a more robust and durable theory of constitutionalism to preempt the problems that may arise here.

3.5 Backstopping the Slippery Slope toward Automated Justice

As the spread of plea bargaining in the United States shows, there is a clear and present danger of the state using its power to make an end-run around protections established in the constitution and guarded by courts. When a prosecutor threatens a defendant with a potential hundred-year sentence in a trial, or a plea for five to eight years, the coercion is obvious. By comparison, given the sclerotic slowness of much of the US administrative state, giving up rights in order to accelerate receipt of benefits is likely to seem to many liberals a humane (if tough) compromise.

Nevertheless, scholars should resist this “deal” by further developing and expanding the “unconstitutional conditions” doctrine. Daniel Farber deftly explicates the basis and purpose of the doctrine:

Farber gives several concrete examples of the types of waivers that have been allowed over time. “[I]n return for government funding, family planning clinics may lose their right to engage in abortion referrals”; a criminal defendant can trade away the right to a jury trial for a lighter sentence. Farber is generally open to the exercise of this right to trade one’s rights away.Footnote ³⁷ However, even he acknowledges that courts need to block particularly oppressive or manipulative exchanges of rights for other benefits. He offers several rationales for such blockages, including one internal to contract theory and another based on public law grounds.Footnote ³⁸ Each is applicable to many instances of “automated justice.”

Farber’s first normative ground for unconstitutional conditions challenges to waivers of constitutional rights is the classic behavioral economics concern about situations “where asymmetrical information, imperfect rationality, or other flaws make it likely that the bargain will not be in the interests of both parties.”Footnote ³⁹ This rationale applies particularly well to scenarios where black-box algorithms (or secret data) are used.Footnote ⁴⁰ No one should be permitted to accede to an abbreviated process when the foundations of its decision-making are not available for inspection. The problem of hyperbolic discounting also looms large. A benefits applicant in brutal need of help may not be capable of fully thinking through the implications of trading away due process rights. Bare concern for survival occludes such calculations.

The second normative foundation concerns the larger social impact of the rights-waiver bargain. For example, Farber observes, “when the agreement would adversely affect the interests of third parties in some tangible way,” courts should be wary of it. The unraveling dynamic described above offers one example of this type of adverse impact on third parties from rights sacrifices. Though it may not be immediately “tangible,” it has happened in so many other scenarios that it is critical for courts to consider whether particular bargains may pave the way to a future where the “choice” to trade away a right is effectively no choice at all, because the cost of retaining it is a high level of suspicion generated by exercising (or merely retaining the right to exercise) the right.

Under this second ground, Farber also mentions that we may “block exchanges that adversely affect the social meaning of constitutional rights, degrading society’s sense of its connection with personhood.” Here again, a drift toward automated determination of legal rights and duties seems particularly apt for targeting. The right of due process at its core means something more than a bare redetermination by automated systems. Rather, it requires some ability to identify a true human face of the state, as Henderson and Brennan-Marquez’s work (discussed previously) suggests. Soldiers at war may hide their faces, but police do not. We are not at war with the state; rather, it is supposed to be serving us in a humanly recognizable way. The same is true a fortiori of agencies dispending benefits and other forms of support.

3.6 Conclusion: Writing, Thinking, and Automation in Administrative Processes

Claimants worried about the pressure to sign away rights to due process may have an ally within the administrative state: persons who now hear and decide cases. AI and ML may ease their workload, but could also be a prelude to full automation. Two contrasting cases help illuminate this possibility. In Albathani v. INS (2003), the First Circuit affirmed the Board of Immigration Appeals’ policy of “affirmance without opinion” (AWO) of certain rulings by immigration judges.Footnote ⁴¹ Though “the record of the hearing itself could not be reviewed” in the ten minutes which the Board member, on average, took to review each of more than fifty cases on the day in question, the court found it imperative to recognize “workload management devices that acknowledge the reality of high caseloads.” However, in a similar Australian administrative context, a judge ruled against a Minister in part due to the rapid disposition of two cases involving more than seven hundred pages of material. According to the judge, “43 minutes represents an insufficient time for the Minister to have engaged in the active intellectual process which the law required of him.”Footnote ⁴²

In the short run, decision-makers at an agency may prefer the Albathani approach. As Chad Oldfather has observed in his article “Writing, Cognition, and the Nature of the Judicial Function,” unwritten, and even visceral, snap decisions have a place in our legal system.Footnote ⁴³ They are far less tiring to generate than a written record and reasoned elaboration of how the decision-maker applied the law to the facts. However, in the long run, when the reduction of thought and responsibility for review reduces to a certain vanishing point, it is difficult for decision-makers to justify their own interposition in the legal process. A “cyberdelegation” to cheaper software may be proper then.Footnote ⁴⁴

We must connect current debates on the proper role of automation in agencies to requirements for reasoned decision-making. It is probably in administrators’ best interests for courts to actively ensure thoughtful decisions by responsible persons. Otherwise, administrators may ultimately be replaced by the types of software and AI now poised to take over so many other roles now performed by humans. The temptation to accelerate, abbreviate, and automate human processes is, all too often, a prelude to destroying them.Footnote ⁴⁵

4.1 Introduction

Is a future in which our emotions are being detected in real time and tracked, both in private and public spaces, dawning? Looking at recent technological developments, studies, patents, and ongoing experimentations, this may well be the case.Footnote ¹ In its Declaration on the manipulative capabilities of algorithmic processes of February 2019, the Council of Europe’s Committee of Ministers alerts us for the growing capacity of contemporary machine learning tools not only to predict choices but also to influence emotions, thoughts, and even actions, sometimes subliminally.Footnote ² This certainly adds a new dimension to existing computational means, which increasingly make it possible to infer intimate and detailed information about individuals from readily available data, facilitating the micro-targeting of individuals based on profiles in a way that may profoundly affect their lives.Footnote ³ Emotional artificial intelligence (further ‘emotional AI’) and empathic media are new buzzwords used to refer to the affective computing sub-discipline and, specifically, to the technologies that are claimed to be capable of detecting, classifying, and responding appropriately to users’ emotional lives, thereby appearing to understand their audience.Footnote ⁴ These technologies rely on a variety of methods, including the analysis of facial expressions, physiological measuring, analyzing voice, monitoring body movements, and eye tracking.Footnote ⁵

Although there have been important debates as to their accuracy, the adoption of emotional AI technologies is increasingly widespread, in many areas and for various purposes, both in the public and private sectors.Footnote ⁶ It is well-known that advertising and marketing go hand in hand with an attempt to exploit emotions for commercial gain.Footnote ⁷ Emotional AI facilitates the systematic gathering of insightsFootnote ⁸ and allows for the further personalization of commercial communications and the optimization of marketing campaigns in real time.Footnote ⁹ Quantifying, tracking, and manipulating emotions is a growing part of the social media business model.Footnote ¹⁰ For example, Facebook is now infamous in this regard due to its emotional contagionFootnote ¹¹ experiment where users’ newsfeeds were manipulated to assess changes in emotion (to assess whether Facebook posts with emotional content were more engaging).Footnote ¹² A similar trend has been witnessed in the political sphere – think of the Cambridge Analytica scandalFootnote ¹³ (where data analytics was used to gauge the personalities of potential Trump voters).Footnote ¹⁴ The aforementioned Declaration of the Council of Europe, among others, points to the dangers for democratic societies that emanate from the possibility to employ algorithmic tools capable of manipulating and controlling not only economic choices but also social and political behaviours.Footnote ¹⁵

Do we need new (constitutional) rights, as suggested by some, in light of growing practices of manipulation by algorithms, in general, and the emergence of emotional AI, in particular? Or, is the current law capable of accommodating such developments adequately? This is undoubtedly one of the most fascinating debates for legal scholars in the coming years. It is also on the radar of CAHAI, the Council of Europe’s Ad Hoc Committee on Artificial Intelligence, set up on 11 September 2019, with the mission to examine the feasibility and potential elements of a legal framework for the development, design, and application of AI, based on the Council of Europe’s standards on human rights, democracy, and the rule of law.Footnote ¹⁶

In the light of these ongoing policy discussions, the ambition of this chapter is twofold. First, it will discuss certain legal-ethical challenges posed by the emergence of emotional AI and its manipulative capabilities. Second, it will present a number of responses, specifically those suggesting the introduction of new (constitutional) rights to mitigate the potential negative effects of such developments. Given the limited scope of the chapter, it does not seek to evaluate the appropriateness of the identified suggestions, but rather to provide the foundation for a future research agenda in that direction. The focus of the chapter lies on the European legal framework and on the use of emotions for commercial business-to-consumer purposes, although some observations are also valid in the context of other highly relevant uses of emotional AI,Footnote ¹⁷ such as implementations by the public sector, or for the purpose of political micro-targeting, or fake news. The chapter is based on a literature review, including recent academic scholarship and grey literature. Its methodology relies on a legal analysis of how the emergence of emotional AI raises concerns and challenges for ‘constitutional’ rights and values through the lens of its use in the business to consumer context. With constitutional rights, we do not refer to national constitutions, but given the chapter’s focus on the European level, to the fundamental rights and values as enshrined in the European Convention for the Protection of Human Rights and Fundamental Freedoms (‘ECHR’), on the one hand, and the EU Charter of Fundamental Rights (‘CFREU’) and Article 2 of the Treaty on European Union (‘TEU’), on the other.

4.2 Challenges to Constitutional Rights and Underlying Values

4.3 Suggestions to Introduce New (Constitutional) Rights

In the light of the previously noted factors, it comes as no surprise that some authors have discussed or suggested the introduction of some novel rights, in order to reinforce the existing legal arsenal.Footnote ¹¹⁰ Although both autonomy and dignity as relevant underlying values and some relevant rights such as right to privacy, freedom of thought, and freedom of expression are protected by the ECHR, some scholars argue that the ECHR does not offer sufficient protection in the light of the manipulative capabilities of emotional AI.Footnote ¹¹¹ The subsequent paragraphs portray, in a non-exhaustive manner, such responses that concern the introduction of some new (constitutional) rights.

A first notable (American) scholar is Shoshana Zuboff, who has argued (in a broader context of surveillance capitalism)Footnote ¹¹² for the ‘right to the future tense’. As noted by Zuboff, ‘we now face the moment in history when the elemental right to future tense is endangered’ by digital architecture of behavioural modification owned and operated by ‘surveillance capital’.Footnote ¹¹³ According to Zuboff, current legal frameworks as mostly centred on privacy and antitrust have not been sufficient to prevent undesirable practices,Footnote ¹¹⁴ including the exploitation of technologies for manipulative purposes. The author argues for the laws that reject the fundamental legitimacy of certain practices,

While arguing about the rationale of the so-called right to the future tense, the author relies on the importance of free will (i.e., Zuboff argues that in essence manipulation eliminates the freedom to will). Consequently, there is no future without the freedom to will, and there are no subjects but only ‘objects’.Footnote ¹¹⁶ As the author puts it, ‘the assertion of freedom of will also asserts the right to the future tense as a condition of a fully human life’.Footnote ¹¹⁷ While arguing for the recognition of such a right as a human right, Zuboff relies on Searle, who argues that elemental rights are crystallized as formal human rights only at that moment in history when they come under systematic threat. Hence, given the development of surveillance capitalism, it is necessary to recognize it as a human right. To illustrate, Zuboff argues that no one is recognizing, for example, a right to breathe because it is not under attack, which cannot be said about the right to the future tense.Footnote ¹¹⁸

German scholar Jan Christoph Bublitz argues for the ‘right to cognitive liberty’ (phrased alternatively a ‘right to mental self-determination’), relying in essence on the fact that the right to freedom of thought has been insignificant in practice, despite its theoretical importance.Footnote ¹¹⁹ Bublitz calls for the law to redefine the right to freedom of thought in terms of its theoretical significance in light of technological developments capable of altering thoughts.Footnote ¹²⁰ The author argues that such technological developments require the setting of normative boundaries ‘to secure the freedom of the forum internum’.Footnote ¹²¹

In their report for the Council of Europe analyzing human rights in the robot age, Dutch scholars Rinie van Est and Joost Gerritsen from the Rathenau Institute suggest reflecting on two novel human rights, namely, the right to not be measured, analyzed or coached and the right to meaningful human contact.Footnote ¹²² They argue that such rights are indirectly related to and aim to elaborate on existing human rights, in particular, the classic privacy right to be let alone and the right to respect for family life (i.e., the right to establish and develop relationships with other human beings).Footnote ¹²³ While discussing the rationale of a potential right not to be measured, analyzed, or coached, they rely on scholarly work revealing detrimental effects of ubiquitous monitoring, profiling or scoring, and persuasion.Footnote ¹²⁴ They argue that what is at stake given the technological development is not only the risk of abuse but the right to remain anonymous and/or the right to be let alone, ‘which in the robot age could be phrased as the right to not be electronically measured, analyzed or coached’.Footnote ¹²⁵ However, their report ultimately leaves it unclear whether they assume it is necessary to introduce the proposed rights as new formal human rights. Rather, it calls for the CoE to clarify how these rights – the right to not be measured, analyzed, or coached, and the right to meaningful human contact – could be included within the right to privacy and the right to family life respectively.Footnote ¹²⁶ In addition to considering potential novel rights, the Rathenau report calls for developing fair persuasion principles, ‘such as enabling people to monitor the way in which information reaches them, and demanding that firms must be transparent about the persuasive methods they apply’.Footnote ¹²⁷

According to UK scholar Karen Yeung, manipulation may threaten individual autonomy and the ‘right to cognitive sovereignty’.Footnote ¹²⁸ While arguing about the rationale of such a right, Yeung relies on the importance of individual autonomy and on the Razian approach comparing manipulation to coercion,Footnote ¹²⁹ as discussed previously. In addition, Yeung relies on Nissenbaum, who observes that the risks of manipulation are even more acute in a digital world involving ‘pervasive monitoring, data aggregation, unconstrained publication, profiling, and segregation’, because the manipulation that deprives us of autonomy is more subtle than the world in which lifestyle choices are punished and explicitly blocked.Footnote ¹³⁰ When it comes to arguing about the need to introduce a new formal human right, Yeung notes that human dignity and individual autonomy are not sufficiently protected by Articles 8, 9, and 10 of the ECHR; however, the study in question does not provide detailed arguments in that regard. The author also refrains from elaborating on the content of such a right.Footnote ¹³¹

Some novel rights are discussed at the institutional level as well. For example, the CoE’s Parliamentary Assembly has proposed working on guidelines which would cover, among other things, the recognition of some new rights, including the right not to be manipulated.Footnote ¹³²

Further research is undoubtedly necessary to assess whether the current legal framework is not already capable of accommodating the developments properly. While the introduction of novel constitutional rights may indeed contribute to defining normative beacons, we should at the same time be cautious not to dilute the significance of constitutional rights by introducing new ones that could, in fact, be considered as manifestations of existing constitutional rights.Footnote ¹³³ Hence, it is particularly important to delineate, as noted by Clifford, between primary and secondary law, and to assess the capabilities of the latter in particular.Footnote ¹³⁴ In other words, it is necessary to exercise restraint and consider what already exists and also to delineate between rights and the specific manifestation of these rights in their operation and/or in secondary law protections (i.e., derived sub-rights). For example, key data subject rights like the right to erasure, object, access, and portability are all manifestations of the aim of respecting the right to data protection as balanced with other rights and interests. Admittedly, while the right to data protection has been explicitly recognized as a distinct fundamental right in the CFREU, this is not the case in the context of the ECHR, where the ECtHR has interpreted the right to privacy in Article 8 ECHR as encompassing informational privacy.Footnote ¹³⁵ The rich debate on the relation between the right to privacy and the right to data protection, and how this impacts secondary law like the GDPR and Convention 108+, clearly exceeds the scope of this chapter.Footnote ¹³⁶

4.4 Blueprint for a Future Research Agenda

The field of affective computing, and more specifically the technologies capable of detecting, classifying, and responding to emotions – in this chapter referred to as emotional AI – hold promises in many application sectors, for instance, for patient well-being in the health sector, for road safety, consumer satisfaction in retail sectors, and so forth. But, just like most (if not all) other forms of artificial intelligence, emotional AI brings with it a number of challenges and calls for assessing whether the existing legal frameworks are capable of accommodating the developments properly. Due to its manipulative capabilities, its potential harmful affective impact and potential profiling errors, emotional AI puts pressure on a whole range of constitutional rights, such as the right to respect for private and family life, non-discrimination, and freedom of thought, conscience, and religion. Moreover, the deployment of emotional AI poses challenges to individual autonomy and human dignity as underlying values underpinning the entirety of international human rights law, as well as to the underlying rationality-based paradigm imbued in law.

Despite the constitutional protection already offered at the European level, some scholars argue, in particular in the context of the ECHR, that this framework does not offer sufficient protection in light of the manipulative capabilities of emotional AI. They suggest (contemplating or introducing) novel rights such as the right to the future tense; the right to cognitive liberty (or, alternatively, the right to mental self-determination); the right to not be measured, analyzed, or coached; the right to cognitive sovereignty; and the right not to be manipulated.

At the same time, it should be noted that the field of constitutional law (in this chapter meant to cover the field of European human rights law) is a very dynamic area that is further shaped through case law, along with societal, economic, and technological developments. The way in which the ECtHR has given a multifaceted interpretation of the right to privacy in Article 8 ECHR is a good example of this.

This motivates the relevance of further research into the scope of existing constitutional rights and secondary sub-rights, in order to understand whether there is effectively a need to introduce new constitutional rights. A possible blueprint for IACL’s Research Group ‘Algorithmic State, Society and Market – Constitutional Dimensions’ could include

• empirical research into the effects of fine-grained, subconscious, and personalised levels of algorithmic persuasion based on affective computing (in general or for specific categories of vulnerable groups, like childrenFootnote ¹³⁷);

• interdisciplinary research into the rise of new practices, such as the trading or renting of machine learning models for emotion classification, which may escape the traditional legal protection frameworks;Footnote ¹³⁸

• doctrinal research into the scope and limits of existing constitutional rights at European level in light of affective computing; Article 9 ECHR and Article 8 CFREU seem particularly interesting from that perspective;

• comparative research, on the one hand, within the European context into constitutional law traditions and interpretations at the national level (think of Germany, where the right to human dignity is explicitly recognised in Article 1 Grundgesetz, versus Belgium or France, where this is not the case), and on the other hand, within the global context (comparing, for instance, the fundamental rights orientated approach to data protection in the EU and the more market-driven approach in other jurisdiction such as the US and AustraliaFootnote ¹³⁹); and

• policy research into the level of jurisdiction, and type of instrument, best suited to tackle the various challenges that emotional AI brings with it. (Is there, for instance, a need for a type of ‘Oviedo Convention’ in relation to (emotional) AI?)

At the beginning of this chapter, reference was made to the CoE’s Declaration on the Manipulative Capabilities of Algorithmic Processes of February 2019.Footnote ¹⁴⁰ In that Declaration, the Committee of Ministers invites member States to

Aspiring to deliver a modest contribution to this much-needed debate, this chapter has set the scene and hopefully offers plenty of food for thought for future activities of the IACL Research Group on Algorithmic State Market & Society – Constitutional Dimensions.

5.1 Introduction

Online human interactions are a continuous matching of data that affects both our physical and virtual life. How data are coupled and aggregated is the result of what algorithms constantly do through a sequence of computational steps that transform the input into the output. In particular, machine learning techniques are based on algorithms that identify patterns in datasets. The paper explores how algorithmic rationality may fit into Weber’s conceptualization of legal rationality. It questions the idea that technical disintermediation may achieve the goal of algorithmic neutrality and objective decision-making.Footnote ¹ It argues that such rationality is represented by surveillance purposes in the broadest meaning. Algorithmic surveillance reduces the complexity of reality calculating the probability that certain facts happen on the basis of repeated actions. Algorithms shape human behaviour, codifying situations and facts, stigmatizing groups rather than individuals, and learning from the past: predictions may lead to static patterns that recall the idea of caste societies, in which the individual potential of change and development is far from being preserved. The persuasive power of algorithms (the so-called nudging) largely consists of small changes aimed at predicting social behaviours that are expected to be repeated in time. This boost in the long run builds a model of anti-social mutation, where actions are oriented. Against such a backdrop, the role of law and legal culture is relevant for individual emancipation and social change in order to frame a model of data production by law. This chapter is divided into four sections: the first part describes commonalities and differences between legal bureaucracy and algorithms, the second part examines the linkage between a data-driven model of law production and algorithmic rationality, the third part shows the different perspective of the socio-legal approach to algorithmic regulation, and the fourth section questions the idea of law production by data as a product of legal culture.

5.2 Bureaucratic Algorithms

‘On-life’ dimensions represent the threshold for a sustainable data-driven rationality.Footnote ² As stated in the White Paper on AI, ‘today 80% of data processing and analysis that takes place in the cloud occurs in data centres and centralized computing facilities, and 20% in smart connected objects, such as cars, home appliances or manufacturing robots, and in computing facilities close to the user (“edge computing”)’. By means of unceasing growth of categorizations and classifications, algorithms develop mechanisms of social control connecting the dots. This entails that our actions mostly depend or are somehow affected by the usable form in which the algorithm code is rendered. In order to enhance their rational capability in calculating every possible action, algorithms aim at reducing human discretion and at structuring behaviours and decisions similarly to bureaucratic organizations. Algorithms act as normative systems that formalize certain patterns. As pointed out by Max Weber, modern capitalist enterprise is mainly based on calculation. For its existence, it requires justice and an administration whose operation can at least in principle be rationally calculated on the basis of general rules – in the same way in which the foreseeable performance of a machine is calculated.Footnote ³ This entails that, on the one hand, like bureaucracy, algorithms, in fact, use impersonal laws requiring obedience that impede free not predictable choices.Footnote ⁴ In fact, according to the Weberian bureaucratic ideal types, the separation between the administrative body and the material means of the bureaucratic enterprise is quintessential to the most perfect form of bureaucratic administration: political expropriation towards specialized civil servants.Footnote ⁵ Nonetheless, impersonality of legal rules does not entail in any case lack of responsibility by virtue of the principle of the division of labour and the hierarchical order on which modern bureaucracy is based:Footnote ⁶ civil servants’ responsibility is indeed to obey impersonal rules or pretend they are impersonal, whereas the exclusive and personal responsibility belongs to the political boss for his actions.Footnote ⁷ Bureaucracy is characterized by the objective fulfilment of duties, ‘regardless of the person’ and based on foreseeable rules and independent from human considerations.Footnote ⁸

On the contrary, the risk of algorithmic decision-making is that no human actor is to take responsibility for the decision.Footnote ⁹ The supervision and the attribution of specialized competences from the highest bureaucratic levels towards the lowest ones (Weber uses the example of ‘procurement’)Footnote ¹⁰ assures that the exercise of authority is compliant to precise competences and technical qualities.Footnote ¹¹ Standardization, rationalization, and formalization are common aspects both for bureaucratic organizations and algorithms. Bureaucratic administration can be considered economic as far as it is fast, precise, continuous, specialized, and avoids possible conflicts.Footnote ¹² Testing algorithms as legal rational means imposes a double question: (1) whether through artificial intelligence and isocratic forms of administration the explainability of algorithmic processes improves the institutional processes and in what respect towards staff competence and individual participation, and (2) whether algorithms take on some of the role of processing institutional and policy complexity much more effectively than humans.Footnote ¹³

According to Aneesh, ‘bureaucracy represents an “efficient” ideal-typical apparatus characterized by an abstract regularity of the exercise of authority centred on formal rationality’.Footnote ¹⁴ In fact, algorithms ‘are trained to infer certain patterns based on a set of data. In such a way actions are determined in order to achieve a given goal’.Footnote ¹⁵ The socio-technical nature of public administration consists in the ability to share data: this is the enabler of artificial intelligence for rationalization. Like bureaucracy, algorithms would be apparently compatible with three Weberian rationales: the Zweckverein (purpose union), as an ideal type of the voluntary associated action; the Anstalt (institution), as an ideal type of institutions, rational systems achieved throughout coercive measures; the Verband (social group), as an ideal type of common action that aims to an agreement for a common purpose.Footnote ¹⁶ According to the first rationale, algorithms are used to smoothly guide a predictable type of social behaviour through data extraction on an ‘induced’ and mostly accepted voluntary basis;Footnote ¹⁷ as for the second, the induction of needs is achieved through forms of ‘nudging’, such as the customization of contractual forms and services based on profiling techniques and without meaningful mechanisms of consent; finally, the legitimacy is based on the social agreement on their utility to hasten and cheapen services (automation theory) or also improve them (augmentation system).Footnote ¹⁸

However, unlike bureaucracy, technology directly legitimizes action enabling users with the bare option ‘can/cannot’. Legitimacy is embedded within the internal rationality of technology. As Pasquale observes, ‘authority is increasingly expressed algorithmically’.Footnote ¹⁹ Moreover, similar to the rise of bureaucratic action, technologies have been thought to be controlled by the exercise of judicial review not to undermine civil liberties and equality. As a matter of fact, algorithmic systems are increasingly being used as part of the continuous process of Entzauberung der Welt (disenchantment of the world) – the achievement of rational goals through organizational measures – with potentially significant consequences for individuals, organizations and societies as a whole.

There are essentially four algorithmic rational models of machine learning that are relevant for law-making: the Neural Networks that are algorithms learning from examples through neurons organized in layers; the Tree Ensemble methods that combine more than one learning algorithm to improve the predictive power of any of the single learning algorithms that they combine; the Support Vector Machines that utilize a subset of the training data, called support vectors, to represent the decision boundary; the Deep Neural Network that can model complex non-linear relationship with multiple hidden layers.Footnote ²⁰

Opaqueness and automation are their main common features, consisting of the secrecy of the algorithmic code and the very limited human input.Footnote ²¹ This typical rationality is blind, as algorithms – Zuboff notes – inform operations given the interaction of these two aspects. Nonetheless, explainability and interpretability are also linked to the potential of algorithmic legal design as rational means.Footnote ²² Rational algorithmic capability is linked to the most efficient use of data and inferences based on them. However, the development of data-driven techniques in the algorithmic architecture determines a triangulation among market, law, and technology. To unleash the full potential of data, rational means deployed to create wider data accessibility and sharing for private and public actors are now being devised in many areas of our lives. However, it should be borne in mind that the use of algorithms as a tool for speeding up the efficiency of the public sector cannot be separately examined from the risk of algorithmic surveillance based on indiscriminate access to private-sector data.Footnote ²³ This is due to the fact that the entire chain of services depends upon more or less overarching access to private sector data. Access to those data requires a strong interaction between public actors’ political power and private actors’ economic and technological capability. This dynamic is pervasive as much as it entirely dominates our daily life from market strategy to economic supply. Furthermore, once the ‘sovereigns’ of the nation-states and their borders have been trumped, data flows re-articulate space in an endless way. The paradox of creating space without having a territory is one of the rationales of the new computational culture that is building promises for the future.

5.3 Law Production by Data

Law production is increasingly subjected to a specialized rationality.Footnote ²⁴ Quantitative knowledge feeds the aspiration of the state bureaucracy’s ‘rationality’, since it helps dress the exercise of public powers of an aura of technical neutrality and impersonality, apparently leaving no room to the discretion of the individual power.Footnote ²⁵ Behind the appearance of the Weberian bureaucratic principle sine ira et studio – which refers to the exclusion of affective personal, non-calculable, and non-rational factors in the fulfilment of civil servants’ dutiesFootnote ²⁶ – the use of classification and measurement techniques affecting human activities generate new forms of power that standardize behaviours for forecasting expectations, performances and conducts of agents.Footnote ²⁷ As correctly highlighted by Zuboff, ‘instrumentarian power reduces the human experience to measurable observable behaviour while remaining steadfastly indifferent to the meaning of that experience’.Footnote ²⁸ However, even though the production of the law through customized and tailored solutions can be a legitimate goal of computational law, it is not all. Social context may change while the law is ruling, but technology reflects changing social needs in a more visible way than the law and apparently provides swifter answers.Footnote ²⁹ On the contrary, the law should filter daily changes, including technological ones, into its own language, while it is regulating a mutable framework. To be competing with other prescriptive systems, the law may be used either as an element of computational rationality or a tool to be computable itself for achieving specific results. In the first case, the law guides and shrinks the action of rational agents through the legal design of algorithms, as an external constraint. In the second case, regulatory patterns are conveyed by different boosts that use the law in a predetermined way for achieving a given goal. Depending on which of those models is chosen, there could be a potential risk for the autonomy of the law in respect to algorithmic rationality. Both the autonomy of the law and the principle of certainty applicable to individuals are at stake. This is an increasingly relevant challenge since the whole human existence is fragmented through data.

Against these two drawbacks, the law may develop its internal rationality even in a third way: as the product of the legal culture that copes with social challenges and needs. Essentially, legal culture is a tough way in which society reflects upon itself, through doctrinal and conceptual systems elaborated by lawyers; through interpretation; and through models of reasoning.Footnote ³⁰ This entails the law being a rational means not only due to its technical linguistic potentialFootnote ³¹ but also due to its technical task aimed at producing social order.Footnote ³² As Weber notes, the superiority of bureaucratic legal rationality over other rational systems is technical.

Nonetheless, not all times reflect a good legal culture, as this can be strongly affected by political and social turmoil. In the age of datification, all fragments of daily life are translated into data, and it is technically possible to shape different realities on demand, including information politics and market. The creation of propensities and assumptions through algorithms as a basis of a pre-packaged concept of the law – driven by colonizing factors – breaks off a spontaneous process through which legal culture surrounds the law. As a result, the effects of algorithmic legal predictions contrast with the goal of legal rationality, which is to establish certain hypotheses and to cluster factual situations into them. The production of the legal culture entails the law being the outcome of a specific knowledge and normative meanings as the result of a contextual Weltanschauung. This aspect has nothing to do either with the legitimacy or with the effectiveness, rather with the way in which the law relies on society. In particular, the capability to produce social consequences that are not directly addressed by the law, by suggesting certain social behaviours and by activating standardized decisions on a large scale, represents such a powerful tool that has been considered the core of algorithmic exception states.Footnote ³³ The idea of exception is explained by the continuous confusion between the rule of causality and the rule of correlation.Footnote ³⁴ Such a blurring between cause and effects, evidences and probabilities, causal inferences and variables, affects database structures, administrative measures that are showed under the form of the algorithmic code, and ultimately rules.Footnote ³⁵ Algorithms lack adaptability because they are based on a casual model that cannot replicate the inferential process of humans to which the general character of the law refers. Human causal intuitions dominate uncertainty differently from machine learning techniques.Footnote ³⁶

Data is disruptive for its capability to blur the threshold between what is inside and what is outside the law. The transformation of the human existence into data is at the crossroad of the most relevant challenges for law and society. Data informs the functioning of legal patterns, but it can be also a component of law production. A reflection on the social function of the law in the context of algorithmic rationality is useful in order to understand what type of data connections are created for regulatory purposes within an ‘architecture of assumptions’, to quote McQuillan. Decoding algorithms sometimes allows one to interpret such results, even though the plurality and complexity of societal patterns cannot be reduced to the findings of data analysis or inferential interpretation generated by automated decision-making processes. The growing amount of data, despite being increasingly the engine of law production, does reflect the complexity of the social reality, which instead refers to possible causal interactions between technology, reality and regulatory patterns, and alternative compositions of them, depending upon uncertain variables. Datification, on which advanced technologies are generally based, has profoundly altered the mechanisms of production of legal culture, which cannot be easily reduced to what data aggregation or data analysis is. Relevant behaviours and social changes nourish inferences that can be made from data streams: despite the fact that they can be the output of the law, they will never be the input of the legal culture. Between the dry facts and the causal explanation, there is a very dense texture for the elaboration of specialized jurists, legal scholars, and judges. Furthermore, globalization strongly shapes common characters across different legal traditions no longer identifiable with an archetypal idea of state sovereignty. This depends upon at least two factors: on the one hand, the increasing cooperation between private and public actors in data access and information management beyond national borders; on the other hand, the increasing production of data from different sources. Nonetheless, not much attention has been paid to the necessity of safeguarding the space of the legal culture in respect to law overproduction by data. Regulation of technology combined with the legal design of technology tends to create a misleading overlap between both, because technological feasibility is becoming the natural substitution of legal rationales. Instead, I argue that the autonomous function of the legal culture should be revenged and preserved as the theoretical grid for data accumulation. What legal culture calls into question is the reflexive social function of the law that data-driven law erases immediately by producing a computational output. In addition, the plurality of interconnected legal systems cannot be reduced to data. The increasing production of the law resulting from data does not reflect the complexity of social reality. How data and technologies based on them affect the rise of legal culture and the production of data-driven laws has not only to do with data. According to a simple definition of legal culture as ‘one way of describing relatively stable patterns of legally oriented social behaviour and attitudes’,Footnote ³⁷ one may think of data-driven law as a technologically oriented legal conduct.

‘Commodification of “reality” and its transformation into behavioural data for analysis and sales’,Footnote ³⁸ defined by Zuboff as surveillance capitalism, has made the private human experience a ‘free raw material’Footnote ³⁹ that can be elaborated and transformed into behavioural predictions feeding production chain and business. Data extraction allows the capitalistic system to know all about all. It is a ‘one-way process, not a relationship’, which produces identity fragmentation and attributes an exchange value to single fragments of the identity itself.Footnote ⁴⁰ Algorithmic surveillance indeed produces a twofold phenomenon: on the one hand, it forges the extraction process itself, which is predetermined to be predictive; on the other hand, it determines effects that are not totally explainable, despite all accurate proxies input into the system. Those qualities are defined operational variables that are processed at a very high speed so that it is hard for humans to monitor them.Footnote ⁴¹

In the light of an unprecedented transformation that is radically shaping the development of personality as well as common values, the role of the law should be not only to guarantee ex post legal remedies but also to reconfigure the dimension of human beings, technology, and social action within integrated projects of coexistence with regulatory models. When an individual is subject to automation – the decision-making process, which determines the best or worst chances of well-being, the easiest or least opportunities to find a good job, or in the case of the predictive police, a threat to the presumption of innocence – the social function of the law is necessary to cope with the increasing complexity of relevant variables and to safeguard freedom. Power relationships striving to impose subjugation vertically along command and obedience relationships are replaced by a new ‘axiomatic’ one: the ability to continuously un-code and re-code the lines along which information, communication, and production intertwine, combining differences rather than forcing unity.

5.4 The Socio-legal Approach

The current socio-legal debate on algorithmic application on legal frameworks is very much focused on issues related to data-driven innovation. Whereas the internal approach is still dominant in many regulatory areas, the relationship between law and technology requires an external perspective that takes into account different possibilities. As the impact of artificial intelligence on the law produces social and cultural patterns, a purely internal legal approach cannot contribute to a comprehensive understanding. However, whereas the law produces bindings effects depending on if certain facts may or not happen, algorithms are performative in the sense that the effect that they aim to produce is encompassed in the algorithmic code. The analysis of both the benefits and the risks of algorithmic rationality have societal relevance for the substantial well-being of individuals. On one hand, the lack of an adequate sectoral regulatory framework requires a cross-cutting analysis to highlight potential shortcomings in the existing legal tools and their inter-relationships. In addition, operational solutions should be proactive in outlining concrete joined-up policy actions, which also consider the role of soft-law solutions. On the other hand, the potential negative impact of biased algorithms on rights protection and non-discrimination risks establishing a legal regime for algorithmic rationality that does not meet societal needs. In order to address the interplay between societal needs, rights, and algorithmic decision-making, it is relevant to pinpoint several filters on the use of AI technology in daily life.

For example, a social filter sets a limits for the manner in which technology is applied on the basis of the activities of people and organizations. A well-known recent example of a social filter is how taxi drivers and their backing organizations have opposed transport platforms and services. An institutional filter sets institutionally determined limits on the ways in which technology can be applied. This type of institutional system includes the corporate governance model, the education system, and the labour market system. A normative filter sets regulatory and statute-based limitations on the manner in which technology can be applied. For example, the adoption of self-steering vehicles in road traffic will be slow until the related issues regarding responsibilities have been conclusively determined in legislation. Last but not least, an ethical filter sets restrictions on the ways in which technology is applied.

A further step requires identifying a changing legal paradigm that progressively shifts attention from the idea of a right to a reasonable explanation of the algorithm as a form of transparency to the right to reasonable inferences (through the extensive interpretation of the notion of personal data that it includes the notion of decisional inference) or towards an evolutionary interpretation of the principle of good administration.Footnote ⁴² The evolutionary interpretation of the principle of good administration has hinged on the algorithmic ‘black box’ within a more fruitful path, oriented towards the legality and responsibility of the decision maker in the algorithmic decision-making process. This is particularly relevant in the field of preventive surveillance, for example, as it is mainly a public service whose technological methods can be interpreted in the light of the principle of good administration.

More broadly, the rationale of AI in the digital single market should inter alia guarantee: (1) better services that are cost-efficient; (2) unifying cross-border public services, increasing efficiency and improving transparency; (3) promoting the participation of individuals in the decision-making process; and (4) improving the use of AI in the private sector as a potential to improve business and competitiveness.Footnote ⁴³

In order to achieve these objectives, it is necessary to evaluate the social impact, as well as the risks and opportunities, that the interaction between public and private actors in accessing data through the use of algorithmic rationality combined with legal rationality entails. However, the optimization of organizational processes in terms of efficiency, on the one hand, and the degree of users’ satisfaction, on the other hand, are not relevant factors to face the impact of algorithms on rights. The law preserving individual chances of emancipation is at the centre of this interaction, constituting the beginning and the end of the causal chain, since both the production of law for protecting rights and the violation of rights significantly alter this relationship. This aspect is significant, for instance, in the field of machine learning carried out on the basis of the mass collection of data flows, from which algorithms are able to learn. The ability of machine learning techniques to model human behaviour, to codify reality and to stigmatize groups, increases the risk of couching static social situations, undermining the free and self-determined development of personality. Such a risk is real, irrespective of the fact that algorithms are used to align a legal system to a predetermined market model or to reach a precise outcome of economic policy. In both cases, algorithms exceed the primary function of the law, which is to match the provision of general and abstract rules with concrete situations through adaptive solutions. Such an adaptation process is missing in the algorithmic logic, because the algorithmic code is unchangeable.

Law as a social construction is able to address specific situations and change at the same time in its interpretation or according to social needs. Indeed, law should advocate an emancipatory function for human beings who are not subject to personal powers. If applied to algorithmic decision-making in the broadest context, the personality of laws may result in tailored and fragmented pictures corresponding to ‘social types’ on the basis of profiling techniques. This is the reason why law production by data processed through algorithms cannot be the outcome of any legal culture, as it would be a pre-packaged solution regardless of the institutional and political context surrounding causes and effects. Nonetheless, the increasing tailored production of data-driven law through algorithmic rationality cannot overcome such a threshold in a way that enables a decision-making process – at every level of daily life – being irrespective of autonomy, case-by-case evaluation, and freedom.

The alignment of legal requirements and algorithmic operational rules must always be demonstrated ex post both at a technical level and at a legal level in relation to the concrete case.

5.5 Data Production by Law

Against the backdrop of data-driven law, legal rationality should be able to frame a model rather based on data production by law. However, a real challenge that should be borne in mind is that algorithmic bureaucracy does not need a territory as legal bureaucracy.Footnote ⁴⁴ Algorithmic systems are ubiquitous, along with data that feed machine learning techniques. Whereas a bureaucratic state is a way to organize and manage the distribution of power over and within a territory, algorithms are not limited by territory. Sovereignty’s fragmentation operated by data flows shows that virtual reality is a radical alternative form to territorial sovereignty and cannot be understood as a mere assignment of sovereign powers upon portions of data. The ubiquity of data requires a new description of regulatory patterns in the field of cross-border data governance as data location that would create under certain conditions the context of the application of legal regime, and the exclusion of another is not necessarily a criterion which is meaningfully associated with the data flow. Data is borderless, as it can be scattered everywhere across different countries.Footnote ⁴⁵ Although data can be accessed everywhere irrespective of where it is located, its regulation and legal effects are still anchored to the territoriality principle. Access to data does not depend on physical proximity; nor are regulatory schemes arising from data flows intrinsically or necessarily connected to any particular territory. Connection with territory must justify jurisdictional concerns but does not have much to do with physical proximity. Such disconnection between information and its un-territorial nature potentially generates conflicts of law and may produce contrasting claims of sovereign powers.Footnote ⁴⁶ This is magnified by algorithmic systems that do not have a forum loci because they are valid formulations regardless of the geographical space where they are applied. Furthermore, they gather data sets irrespective of borders or jurisdictions. Bureaucracy’s functioning much depends upon borders, as it works only within a limited territory.Footnote ⁴⁷ On the contrary, algorithms are unleashed from territories but can affect multiple jurisdictions, as the algorithmic code is territorially neutral. This may be potentially dangerous for two reasons: on the one hand, algorithms can transversally impact different jurisdictions, regardless of the legal systems and regulatory regimes involved; on the other hand, the disconnection of the algorithmic code from territory and implies a law production that does not emerge from legal culture. Even though legal culture is not necessarily bound to the concept of state sovereignty,Footnote ⁴⁸ it is inherent to a territory as a political and social space. Weber rejects the vision of the modern judge as a machine in which ‘documents are input together with expenses’ and which spits out the sentence together with the motives mechanically inferred from the paragraphs. Indeed, there is the space for the individualizing assessment in respect of which the general norms have a negative function in that they limit the official’s positive and creative activity.Footnote ⁴⁹ This massive difference between legal rationality and algorithmic rationality imposes rethinking the relationship between law, technology, and legal culture. Data production by law can be a balanced response to reconnect algorithmic codes to the boundaries of jurisdictions. Of course, many means of data production by law exist. A simple legal design of data production is not the optimal option. Matching algorithmic production of data and legal compliance can be mechanically ensured through the application of certain patterns that are inserted in the algorithmic process. Instead, the impact of legal culture over the algorithmic production of data shape a socio-legal context inspiring the legal application of rules on data production.

The experience of the Italian Administrative Supreme Court (Council of State) is noteworthy. After the leading case of 8 April 2019 n. 2270 that opened the path to administrative algorithmic decision-making, the Council of State confirmed its case law.Footnote ⁵⁰ It holds the lawfulness of automated decision-making in administrative law, providing limits and criteria.Footnote ⁵¹ It extended for the first time the automated decision-making both to public administration’s discretionary and binding activities. The use of algorithmic administrative decision-making is encompassed by the principle of good performance of administration pursuant to article 97 of the Italian Constitution. The Council stated that the fundamental need for protection posed by the use of the so-called IT tool algorithmic is transparency due to the principle of motivation of the decision.Footnote ⁵² It expressly denied algorithmic neutrality, holding that predictive models and criteria are the result of precise choices and values. Conversely, the issue of the dangers associated with the instrument is not overcome by the rigid and mechanical application of all detailed procedural rules of Law no. 241 of 1990 (such as, for example, the notice of initiation of the proceeding).

The underlying innovative rationale is that the ‘multidisciplinary character’ of the algorithm requires not only legal but technical, IT, statistical, and administrative skills, and does not exempt from the need to explain and translate the ‘technical formulation’ of the algorithm into the ‘legal rule’ in order to make it legible and understandable.

Since algorithm becomes a modality of the authoritative decision, it is necessary to determine specific criteria for their use. Surprisingly, the Council made an operation of legal blurring, affirming that knowability and transparency must be interpreted according to articles 13, 14, and 15 GDPR. In particular, the interested party must be informed of the possible execution of an automated decision-making process; in addition, the owner of algorithms must provide significant information on the logic used, as well as the importance and expected consequences of this treatment for the interested party.

Additionally, the Council adopted three supranational principles: (1) the full knowability of the algorithm used and the criteria applied pursuant to article 42 of the EU Charter (‘Right to a good administration’), according to which everyone has the right to know the existence of automated decision-making processes concerning him or her and, in this case, to receive significant information on the logic used; (2) the non-exclusivity of automated decision-making, according to which everyone has the right not to be subjected to solely automated decision-making (similarly to article 22 GDPR); and (3) the non-discrimination principle, as a result of the application of the principle of non-exclusivity, plus data accuracy, minimization of risks of errors, and data security.Footnote ⁵³ In particular, the data controller must use appropriate mathematical or statistical procedures for profiling, implementing adequate technical and organizational measures in order to ensure correction of the factors that involve data inaccuracy, thus minimizing the risk of errors.Footnote ⁵⁴ Input data should be corrected to avoid discriminatory effects in decision-making output. This operation requires the necessary cooperation of those who instruct the machines that produce these decisions. The goal of a legal design approach is to filter data production through the production of potential algorithmic harms and the protection of individual rights, and figure out which kind of legal remedies are available and also useful to individuals. The first shortcoming of such endeavour is that – given for granted the logic of garbage in/garbage out, according to which inaccurate inputs produce wrong outputs – it is noteworthy that a legal input is not a sufficient condition to produce a lawful output. Instead, an integrated approach such as the one adopted by the Council of State is based on more complex criteria to consider the lawfulness of algorithmic decision-making, also in respect of actors involved. First, it is necessary to ensure the traceability of the final decision to the competent body pursuant to the law conferring the power of the authoritative decision to the civil servants in charge.Footnote ⁵⁵ Second, the comprehensibility of algorithms must involve all aspects but cannot result in harm for IP rights. In fact, pursuant to art. 22, let. c, Law 241/90 holders of an IP right on software are considered counter-interested,Footnote ⁵⁶ but Consiglio di Stato does not specifically address the issue of holders of trade secrets.

5.6 Conclusions: TOWARDS DATA PROTECTION OF LAW

While discussing similarities between bureaucratic and algorithmic rationality, I voluntarily did not address the issue of secrecy. According to Weber, each power that aims to its preservation is a secret power in one of its features. Secrecy is functional for all bureaucracies to the superiority of their technical tasks towards other rational systems.Footnote ⁵⁷ Secrecy is also the fuel of algorithmic reasoning, as its causal explanation is mostly secret. This common aspect, if taken for granted as a requirement of efficient rational decision-making, should be weighted in a very precise way in order to render algorithms compliant with the principle of legality.

This chapter has explored how algorithmic bureaucracy proves to be a valuable form of rationality as far as it does not totally eliminate human intermediation under the form of imputability, responsibility, and control.Footnote ⁵⁸ To be sure, this may happen only under certain conditions that are summarized as follows: (1) Technological neutrality for law production cannot be a space ‘where legal determinations are de-activated’Footnote ⁵⁹ in such a way that externalizes control. (2) Law production by data is not compatible with Weberian’s legal rationality. (3) Translation of technical rules into legal rules needs to be filtered through legal culture. (4) Data production by law is the big challenge of algorithmic rationality. (5) Algorithmic disconnection from territory cannot be replaced by algorithmic global surveillance. (6) Legal design of algorithmic functioning is not an exhaustive solution. (7) The linkage of automated decision-making to the principle of good administration is a promising trajectory along which concepts such as traceability, knowability, accessibility, readability, imputability, responsibility, and non-exclusivity of the automated decision have been developed in the public interest.

All these conditions underlie a regulatory idea that draws the role of lawyers from what Max Weber defined as die geistige Arbeit als Beruf (the spiritual work as profession). In this respect, algorithmic rationality may be compatible with a legal creative activity as long as a society is well equipped with good lawyers.Footnote ⁶⁰ The transformation of law production by data into data production by law is a complex challenge that lawyers can drive if they do not give up being humanists for being only specialized experts.Footnote ⁶¹ From this perspective, algorithmic bureaucratic power has a good chance of becoming an ‘intelligent humanism’.Footnote ⁶² To accomplish this task, the law should re-appropriate its own instruments of knowledge's production. This does not mean to develop a simplistic categorization of legal compliance requirements for machine-learning techniques. Nor it only relies on the formal application of legal rationality to the algorithmic process. In the long run, it shall bring towards increasing forms of data production of law. Data production of law defines the capability of the law to pick and choose those data that are relevant to elaborate new forms of legal culture. How the law autonomously creates knowledge from experiences that impact on society is a reflexive process that needs institutions as well as individuals. As much as this process is enshrined in a composite legal culture, the law has more chances to recentre its own role in the development of democratic societies.

6.1 Introduction

Artificial intelligence (AI) constitutes a major form of scientific and technological progress. For the first time in human history, it is possible to create autonomous systems capable of performing complex tasks, such as processing large quantities of information, calculating and predicting, learning and adapting responses to changing situations, and recognizing and classifying objects.Footnote ¹ For instance, algorithms, or so-called Algorithmic Decision Systems (ADS),Footnote ² are increasingly involved in systems used to support decision-making in many fields,Footnote ³ such as child welfare, criminal justice, school assignment, teacher evaluation, fire risk assessment, homelessness prioritization, Medicaid benefit, immigration decision systems or risk assessment, and predictive policing, among other things.

An Automated Decision(-making/-support) System (ADS) is a system that uses automated reasoning to facilitate or replace a decision-making process that would otherwise be performed by humans.Footnote ⁴ These systems rely on the analysis of large amounts of data from which they derive useful information to make decisions and to inferFootnote ⁵ correlations,Footnote ⁶ with or without artificial intelligence techniques.Footnote ⁷

Law enforcement agencies are increasingly using algorithmic predictive policing systems to forecast criminal activity and allocate police resources. For instance, New York, Chicago, and Los Angeles use predictive policing systems built by private actors, such as PredPol, Palantir, and Hunchlab,Footnote ⁸ to assess crime risk and forecast its occurrence, in hope of mitigating it. More often, such systems predict the places where crimes are most likely to happen in a given time window (place-based) based on input data, such as the location and timing of previously reported crimes.Footnote ⁹ Other systems analyze who will be involved in a crime as either victim or perpetrator (person-based). Predictions can focus on variables such as places, people, groups, or incidents. The goal is also to better deploy officers in a time of declining budgets and staffing.Footnote ¹⁰ Such tools are mainly used in the United States, but European police forces have expressed an interest in using them to protect the largest cities.Footnote ¹¹ Predictive policing systems and pilot projects have already been deployed,Footnote ¹² such as PredPol, used by the Kent Police in the United Kingdom.

However, these predictive systems challenge fundamental rights and guarantees of the criminal procedure (Section 6.2). I will address these issues by taking into account the enactment of ethical norms to reinforce constitutional rights (Section 6.3),Footnote ¹³ as well as the use of a practical tool, namely Algorithmic Impact Assessment, to mitigate the risks of such systems (Section 6.4).

6.2 Human Rights Challenged by Predictive Policing Systems

In proactive policing, law enforcement uses data and analyzes patterns to understand the nature of a problem. Officers attempt to prevent crime and mitigate the risk of future harm. They refer to the power of information, geospatial technologies, and evidence-based intervention models to predict what and where something is likely to happen, and then deploy resources accordingly.Footnote ¹⁴

6.3 Human Rights Reinforced by Ethical Principles to Govern AI

Before considering the enactment of ethical principles to reinforce fundamental rights in the use of ADS, one needs to identify whether or not efficient legal provisions are already enacted.

6.4 Human Rights Reinforced by Practical Tools to Govern ADS

In order to identify solutions and practical tools, excluding the instruments of self-regulation,Footnote ⁵⁹ the ‘Trustworthy AI Assessment List’ proposed by the Group of Experts can first be considered. Aiming to operationalize the ethical principles and requirements, the Guidelines present an assessment list that offers guidance on the practical implementation of each requirement. This assessment list will undergo a piloting process in which all interested stakeholders can participate, in order to gather feedback for its improvement. In addition, a forum to exchange best practices for the implementation of Trustworthy AI has been created. However, the goal of these Guidelines and the List is to regulate the activities linked with AI technologies via a general approach. Consequently, the measures proposed are broad enough to cover many situations and different applications of AI, such as climate action and sustainable infrastructure, health and well-being, quality education and digital transformation, tracking and scoring individuals, and lethal autonomous weapon systems (LAWS). But while our study concerns predictive policing activities, it is more relevant to consider specific, practical tools which regulate the governmental activities and ADS.Footnote ⁶⁰ In this sense, the Canadian government enacted in February 2019 a Directive on Automated Decision-MakingFootnote ⁶¹ and a method on AIA.Footnote ⁶² These tools pursue the goal of offering governmental institutions a practical method to comply with fundamental rights, laws, and ethical principles. I argue that these methods are relevant to assess the activity of predictive policing in theory.

6.5 Conclusion

Human rights are a representation of the fundamental values of a society and are universal. However, in an algorithmic society, even if a European lawmaker pretends to reinforce the protection of these rights through ethical principles, I have demonstrated that the current system is not good enough when it comes to guaranteeing their respect in practice. Constitutional rights must be reinforced not only by ethical principles but even more by specific practical tools taking into account the risks involved in ADS, especially when the decision-making concerns sensitive issues such as predictive policing. Beyond the Ethics Guidelines for Trustworthy AI, I argue that the European lawmaker should consider enacting similar tools as the Canadian Directive on Automated Decision Making and AIAs policies that must be made applicable to police services to make them accountable.Footnote ⁶⁴ AIAs will not solve all of the problems that algorithmic systems might raise, but they do provide an important mechanism to inform the public and to engage policymakers and researchers in productive conversation.Footnote ⁶⁵ Even if this tool is certainly not perfect, it constitutes a good starting point. Moreover, I argue this policy should come from the European Union and not its member states. The protection of human rights in an algorithmic society may be considered globally as a whole system integrating human rights. The final result is providing a robust theoretical and practical framework, while human rights keep a central place within this broad system.

7.1 Introduction

Technological progress could constitute a huge benefit for law enforcement and criminal justice more broadly.Footnote ¹ In the security context,Footnote ² alleged opportunities and benefits of applying big data analytics are greater efficiency, effectiveness, and speed of law enforcement operations, as well as more precise risk analyses, including the discovery of unexpected correlations,Footnote ³ which could nourish profiles.Footnote ⁴

The concept of ‘big data’ refers to the growing ability of technology to capture, aggregate, and process an ever-greater volume and variety of data.Footnote ⁵ The combination of mass digitisation of information and the exponential growth of computational power allows for their increasing exploitation.Footnote ⁶

A number of new tools have been developed. Algorithms are merely an abstract and formal description of a computational procedure.Footnote ⁷ Besides, law enforcement can rely on artificial intelligence (i.e., the theory and development of computer systems capable of performing tasks which would normally require human intelligence), such as visual perception, speech recognition, decision-making, and translation between languages.Footnote ⁸ For the purpose of this contribution, these systems are relevant because they do not simply imitate the intelligence of human beings; they are meant to formulate and often execute decisions. The notion of an allegedly clever agent, capable of taking relatively autonomous decisions, on the basis of its perception of the environment, is in fact, pivotal to the current concept of artificial intelligence.Footnote ⁹ With machine learning, or ‘self-teaching’ algorithms, the knowledge in the system is the result of ‘data-driven predictions’, the automated discovery of correlations between variables in a data set, often to make estimates of some outcome.Footnote ¹⁰ Correlations are relationships or patterns, thus more closely related to the concept of ‘suspicion’ rather than the concept of ‘evidence’ in criminal law.Footnote ¹¹ Data mining, or ‘knowledge discovery from data’, refers to the process of discovery of remarkable patterns from massive amounts of data.

Such tools entail new scenarios for information gathering, as well as the monitoring, profiling, and prediction of individual behaviours, thus allegedly facilitating crime prevention.Footnote ¹² The underlying assumption is that data could change public policy, addressing biases and fostering a data-driven approach in policy-making. Clearer evidence could support both evaluations of existing policies and impact assessments of new proposals.Footnote ¹³

Law enforcement authorities have already embraced the assumed benefits of big data, irrespective of criticism questioning the validity of crucial assumptions underlying criminal profiling.Footnote ¹⁴ In a range of daily operations and surveillance activities, such as patrol, investigation, as well as crime analysis, the outcomes of computational risk assessment are increasingly the underlying foundation of criminal justice policies.Footnote ¹⁵ Existing research on the implications of ‘big data’ has mostly focused on privacy and data protection concerns.Footnote ¹⁶ However, potential gains in security come also at the expenses of accountabilityFootnote ¹⁷ and could lead to the erosion of fundamental rights, emphasising coercive control.Footnote ¹⁸

This contribution first addresses the so-called rise of the algorithmic society and the use of automated technologies in criminal justice to assess whether and how the gathering, analysis, and deployment of big data are changing law enforcement activities. It then examines the actual or potential transformation of core principles of criminal law and whether the substance of legal protectionFootnote ¹⁹ may be weakened in a ‘data-driven society’.Footnote ²⁰

7.2 The Rise of the Algorithmic Society and the Use of Automated Technologies in Criminal Justice

7.3 The Transformation of Core Principles of Criminal Law

7.4 Concluding Remarks

Law enforcement authorities and intelligence services have already embraced the assumed benefits of big data analyses. It is yet difficult to assess how and to what extent big data are applied to the field of security, irrespective of exploring whether or not their use fosters efficiency or effectiveness. This is also because of secrecy often surrounding law enforcement operations, the experimental nature of new means, and authorities’ understandable reluctance to disclose their functioning to public opinion. ‘Algorithms are increasingly used in criminal proceedings for evidentiary purposes and for supporting decision-making. In a worrying trend, these tools are still concealed in secrecy and opacity preventing the possibility to understand how their specific output has been generated’,Footnote ⁷⁴ argues Palmiotto, addressing the Exodus case,Footnote ⁷⁵ while questioning whether opacity represents a threat to fair trial rights.

However, there is still a great need for an in-depth debate about the appropriateness of using algorithms in machine-learning techniques in law enforcement, and more broadly in criminal justice. In particular, there is a need to assess how the substance of legal protection may be weakened by the use of tools such as algorithms and artificial intelligence.Footnote ⁷⁶

Moreover, given that big data, automation, and artificial intelligence remain largely under-regulated, the extent to which data-driven surveillance societies could erode core criminal law principles such as reasonable suspicion and the presumption of innocence ultimately depends on the design of the surveillance infrastructures. There is thus a need to develop a regulatory framework adding new layers of protection to fundamental rights and safeguards against their erroneous use.

There are some improvements which could be made to increase the procedural fairness of these tools. First, more transparent algorithms could increase their trustworthiness. Second, if designed to remove pre-existing biases in the original data sets, algorithms could also improve their neutrality. Third, when algorithms are in use profiling and (semi-)automated decision-making should be regulated more tightly.Footnote ⁷⁷

Most importantly, the ultimate decision should always be human. The careful implementations by humans involved in the process could certainly mitigate the vulnerabilities of automated systems. It must remain for a human decision maker or law enforcement authority to decide how to act on any computationally suggested result.

For instance, correlation must not be erroneously interpreted as a causality link, so that ‘suspicion’ is not confused with ‘evidence’. Predictions made by big data analysis must never be sufficient for the purpose of initiating a criminal investigation.

Trust in algorithms both in fully and partially automated decision processes is grounded on their supposed infallibility. There is a tendency (as has been the case in the use of experts in criminal casesFootnote ⁷⁸) among law enforcement authorities to blindly follow them. Rubberstamping algorithms’ advice could also become a trick to minimise the responsibility of decision maker.

Algorithm-based decisions require time, context, and skills to be adequate in each individual case. Yet, given the complexity of algorithms, judges and law enforcement authorities can at times hardly understand the underlying calculus, and it is thus difficult to question their accuracy, effectiveness, or fairness. This is linked with the transparency paradox surrounding the use of big data:Footnote ⁷⁹ citizens become increasingly transparent to government, while the profiles, algorithms, and methods used by government organisations are hardly transparent or comprehensible to citizens.Footnote ⁸⁰ This results in a shift in the balance of power between state and citizen, in favour of the state.Footnote ⁸¹