Fundamental Rights and the Rule of Law in the Algorithmic Society

doi:10.1017/9781108914857.003

2 - Fundamental Rights and the Rule of Law in the Algorithmic Society

from Part I - Algorithms, Freedom, and Fundamental Rights

Published online by Cambridge University Press: 01 November 2021

Andrea Simoncini and

Edited by

Giovanni Sartor and

Hans-W. Micklitz: Affiliation:
European University Institute, Florence
Oreste Pollicino: Affiliation:
Bocconi University
Amnon Reichman: Affiliation:
University of California, Berkeley
Andrea Simoncini: Affiliation:
University of Florence
Giovanni Sartor: Affiliation:
European University Institute, Florence
Giovanni De Gregorio: Affiliation:
University of Oxford

Book contents

Summary

This introductory chapter opens the book’s part on ‘Fundamental rights and rule of law in the Algorithmic society.’ By recalling the old prophecy of Herbert Marcuse, in the first and second sections, we outline pitfalls of the prevailing dominance of algorithmic decision-making on the pillars of Constitutional law. In the third section, we analyse how the fast-growing use of algorithms in the fields of justice, policing, public welfare, etc., could end in biased and erroneous decisions, boosting inequality, discrimination, unfair consequences, and undermining constitutional rights, such as privacy, freedom of expression, and equality. The final section is devoted to draw the roadmap of the entire book’s part, which covers chapters on ‘due process,’ ‘emotional Artificial Intelligence,’ ‘algorithmic administration,’ and ‘predictive policing.’

Keywords

Algorithmic society constitutional law artificial intelligence fundamental rights

Type: Chapter
Information: Constitutional Challenges in the Algorithmic Society , pp. 27 - 41

DOI: https://doi.org/10.1017/9781108914857.003 [Opens in a new window]

Publisher: Cambridge University Press

Print publication year: 2021
Creative Commons: This content is Open Access and distributed under the terms of the Creative Commons Attribution licence CC-BY-NC-ND 4.0 https://creativecommons.org/cclicenses/

2.1 New Technologies and the Rise of the Algorithmic Society

New technologies offer human agents entirely new ways of doing things.Footnote ¹ However, as history shows, ‘practical’ innovations always bring with them more significant changes. Each new option introduced by technological evolution allowing new forms affects the substance, eventually changing the way humans think and relate to each other.Footnote ² The transformation is especially true when we consider information and communication technologies (so-called ICT); as indicated by Marshall McLuhan, ‘the media is the message’.Footnote ³ Furthermore, this scenario has been accelerated by the appearance of artificial intelligence systems (AIS), based on the application of machine learning (ML).

These new technologies not only allow people to find information at an incredible speed; they also recast decision-making processes once in the exclusive remit of human beings.Footnote ⁴ By learning from vast amounts of data – the so-called Big Data – AIS offer predictions, evaluations, and hypotheses that go beyond the mere application of pre-existing rules or programs. They instead ‘induce’ their own rules of action from data analysis; in a word, they make autonomous decisions.Footnote ⁵

We have entered a new era, where big multinational firms (called ‘platforms’) use algorithms and artificial intelligence to govern vast communities of people.Footnote ⁶ Conversely, data generated by those platforms fuel the engine of the ‘Algorithmic Society’.Footnote ⁷

From this point of view, the Algorithmic Society is a distinctive evolution of the ‘Information Society’,Footnote ⁸ where a new kind of ‘mass-surveillance’ becomes possible.Footnote ⁹

This progress generates a mixture of excitement and anxiety.Footnote ¹⁰ The development of algorithms and artificial intelligence technologies is becoming ubiquitous, omnipresent, and seemingly omnipotent. They promise to eliminate our errors and make our decisions better suited for any purpose.Footnote ¹¹

In this perspective, a relatively old prophecy, predicted by Herbert Marcuse in one of the ‘red books’ of that massive socio-political movement usually known as ‘1968’, The One-Dimensional Man, becomes reality. Marcuse starts the first page of that seminal book as follows:

A comfortable, smooth, reasonable, democratic unfreedom prevails in advanced industrial civilization, a token of technical progress.

Indeed, what could be more rational than the suppression of individuality in the mechanization of socially necessary but painful performances; … That this technological order also involves a political and intellectual coordination may be a regrettable and yet promising development. The rights and liberties which were such vital factors in the origins and earlier stages of industrial society yield to a higher stage of this society: they are losing their traditional rationale and content. …

To the degree to which freedom from want, the concrete substance of all freedom, is becoming a real possibility. The liberties that pertain to a state of lower productivity are losing their former content. … In this respect, it seems to make little difference whether the increasing satisfaction of needs is accomplished by an authoritarian or a non-authoritarian system.Footnote ¹²

If technology replaces all ‘socially necessary but painful performances’ – work included – personal freedom reaches its final fulfilment (that is, its very end). In Marcuse’s eyes, this is how technological power will take over our freedom and political system: not through a bloody ‘coup’ but by inducing people – practically and happily – to give up all their responsibilities.

However, this dystopic perspective – a future of ‘digital slavery’, where men and women will lose their liberty and quietly reject all democratic principlesFootnote ¹³ – produces a reaction. It is not by chance that the European Commission’s strategically endorsing the transformation of the EU into an AI-led economy, at the same time, requires great attention to people’s trust and a high level of fundamental rights protection.Footnote ¹⁴

One of the most common areas where we experience the rise of these concerns is public and private security.Footnote ¹⁵ For a large part of the 2010s onward, technological innovations have focused on safety and control; the consequence has been an alarming increase in public and private surveillance, coupled with growing threats to political and civil liberties.Footnote ¹⁶ In addition to this, the global ‘COVID-19’ pandemic has doubtlessly boosted the already fast-growing ‘surveillance capitalism’.Footnote ¹⁷

While at the beginning of the twenty-first century, there was an increasing awareness of the risks of the new pervasive surveillance technologies, today, hit by the pandemic and searching for practical tools to enforce social distancing or controlling policies, the general institutional and academic debate seems to be less worried by liberty-killing effects and more allured by health-preserving results.Footnote ¹⁸

Regardless, the most worrying challenges stem from the increasing power of algorithms, created through Big Data analytics such as machine learning and used to automate decision-making processes.Footnote ¹⁹ Their explicability,Footnote ²⁰ liability, and culpability are still far from being clearly defined.Footnote ²¹ As a consequence, several scholars and policymakers are arguing, on the one hand, to aggressively regulate tech firmsFootnote ²² (since classic antitrust law is unfit for this purpose) or, on the other, to require procedural safeguards, allowing people to challenge the decisions of algorithms which can have significant consequences on their lives (such as credit score systems).Footnote ²³

2.2 The Impact of the Algorithmic Society on Constitutional Law

As we know, at its very origin, constitutional theory wrestles with the problem of power control.Footnote ²⁴ Scholars commonly consider constitutional law that part of the legal system whose function is to legallyFootnote ²⁵ delimit power.Footnote ²⁶ In the ‘modern sense’,Footnote ²⁷ this discipline establishes rules or builds institutions capable of shielding personal freedoms from external constraints.Footnote ²⁸ According to this idea, constitutionalism historically always ‘adapted’ itself to power’s features; that is to say, the protection of freedoms in constitutions has been shaped following the evolving character of the threats to those same freedoms.Footnote ²⁹

At the beginning of the modern era, the power to be feared was the king’s private force.Footnote ³⁰ The idea of ‘sovereignty’, which appeared at the end of the Middle Ages, had its roots in the physical and military strength of the very person of the ‘Sovereign’.Footnote ³¹ Sovereignty evoked an ‘external power’Footnote ³² grounded on the monopoly (actual or potential) of the physical ‘force’Footnote ³³ used against individuals or communities (e.g., ‘military force’ or the ‘force of law’).Footnote ³⁴ Consequently, liberties were those dimensions of human life not subjected to that power (e.g., habeas corpus). As the offspring of the French and American Revolutions, the ‘rule of law’ doctrine was the main legal tool ‘invented’ by constitutional theory to delimit the king’s power and protect personal freedom and rights. To be ‘legitimate’, any power has to be subjected to the rule of law.

The other decisive turning point in the history of constitutionalism was World War II and the end of twentieth-century European totalitarian regimes. It may sound like a paradox, but those regimes showed that the ‘legislative state’, built on the supremacy of law and therefore exercising a ‘legitimate power’, can become another terrible threat to human freedom and dignity.

If the law itself has no limits, whenever it ‘gives’ a right, it can ‘withdraw’ it. This practice is the inhuman history of some European twentieth-century states that cancelled human dignity ‘through the law’.

With the end of World War II, a demolition process of those regimes began, and learning from the American constitutional experience, Europe transformed ‘flexible’ constitutions – until then, mere ordinary laws – into ‘rigid’ constitutions,Footnote ³⁵ which are effectively the ‘supreme law’ of the land.Footnote ³⁶

In this new scenario, the power that instils fear is no longer the king’s private prerogative; the new limitless force is the public power of state laws, and the constitutional tool intended to effectively regulate that power is vested in the new ‘rigid’ constitution: a superior law, ‘stronger’ than ordinary statutes and thus truly able to protect freedoms, at least apparently, even against legal acts.

With the turn of the twenty-first century, we witness the rise of a new kind of power. The advent of new digital technologies, as discussed previously, provides an unprecedented means of limiting and directing human freedom that has appeared on the global stage; a way based on not an ‘external’ force (as in the two previous constitutional scenarios, the private force of the king or the public ‘force of the law’) but rather an ‘internal’ force, able to affect and eventually substitute our self-determination ‘from inside’.Footnote ³⁷

This technological power is at the origin of ‘platform capitalism’,Footnote ³⁸ which is a vast economic transformation induced by the exponentially fast-growing markets of Internet-related goods and services – for example, smart devices (Apple, Samsung, Huawei, Xiaomi), web-search engines (Google), social media corporations (Facebook, Instagram, Twitter), cloud service providers (Amazon, Microsoft, Google), e-commerce companies (Amazon, Netflix), and social platforms (Zoom, Cisco Webex).

Consider that today,Footnote ³⁹ the combined value of the S&P 500’s five most prominent companiesFootnote ⁴⁰ now stands at more than $7 trillion, accounting for almost 25 per cent of the market capitalization of the index, drawing a picture of what a recent doctrine accurately defined as a ‘moligopoly’.Footnote ⁴¹

These ‘moligopolists’Footnote ⁴² are not only creating communities and benefitting from network effects generated by users’ transactions, but they also develop a de facto political authority and influence once reserved for legal and political institutions. More importantly, they are taking on configurations that are increasingly similar to the state and other public authorities.Footnote ⁴³ Their structure reflects a fundamental shift in the political and legal systems of Western democracies – what has been called a new type of ‘functional sovereignty’.Footnote ⁴⁴ Elsewhere we used the term ‘cybernetic power’,Footnote ⁴⁵ which perhaps sounds like an old-fashioned expression. Still, it is more accurate in its etymology (‘cyber’, from its original ancient Greek meaning,Footnote ⁴⁶ shares the same linguistic root as ‘govern’ and ‘governance’) to identify how automation and ICT have radically transformed our lives.

As algorithms begin to play a dominant role in the contemporary exercise of power,Footnote ⁴⁷ it becomes increasingly important to examine the ‘phenomenology’ of this new sovereign power and its unique challenges to constitutional freedoms.

2.3 The ‘Algorithmic State’ versus Fundamental Rights: Some Critical Issues

As already stated, the main force of algorithms is their practical convenience, so their interference with our freedom is not perceived as an ‘external’ constraint or a disturbing power. Instead, it is felt as evidence-based support for our decisions, capturing our autonomy by lifting our deliberation burden.

Who would like to switch back to searching for information in volumes of an encyclopaedia? Who would want to filter their email for spam manually anymore? Who would like to use manual calculators instead of a spreadsheet when doing complex calculations? We are not just living in an increasingly automated world; we are increasingly enjoying the many advantages that come with it. Public administrations are using more and more algorithms to help public-sector functions, such as welfare, the labour market, tax administration, justice, crime prevention, and more. The use of algorithms in decision-making and adjudications promises more objectivity and fewer costs.

However, as we said, algorithms have a darker side, and the following chapters of this section of the book illustrate some of the facets of the Algorithmic State phenomenology.

The fast-growing use of algorithms in the fields of justice, policing, public welfare, and the like could end in biased and erroneous decisions, boosting inequality, discrimination, unfair consequences, and undermining constitutional rights, such as privacy, freedom of expression, and equality.Footnote ⁴⁸

And these uses raise considerable concerns not only for the specific policy area in which they are operated but also for our society as a whole.Footnote ⁴⁹ There is an increasing perception that humans do not have complete control over Algorithmic State decision-making processes.Footnote ⁵⁰ Despite their predictive outperformance over analogue tools, algorithmic decisions are difficult to understand and explain (the so-called black box effect).Footnote ⁵¹ While producing highly effective practical outcomes, algorithmic decisions could undermine procedural and substantive guarantees related to democracy and the rule of law.

Issues related to the use of algorithms as part of the decision-making process are numerous and complex, but at the same time, the debate is at an early stage. However, efforts towards a deeper understanding of how algorithms work when applied to legally tricky decisions will be addressed soon.

In this section, we will examine four profiles of the use of algorithmic decisions: the relation between automation and due process, the so-called ‘emotional’ AI, the algorithmic bureaucracy, and predictive policing.

Due Process in the Age of AI

In Chapter 3, entitled ‘Inalienable Due Process in an Age of AI: Limiting the Contractual Creep toward Automated Adjudication’, Frank Pasquale argues that robust legal values must inspire the current efforts to ‘fast track’ cases by judges and agencies, via statistical methods, machine learning, or artificial intelligence. First, he identifies four core features to be included in due process rights when algorithmic decisions are under consideration. They are related to the ‘ability to explain one’s case’, the ‘necessity of a judgment by a human decision-maker’, an ‘explanation for that judgment’, and an ‘ability to appeal’. As a second step, he argues that given that legal automation threatens due process rights, we need proper countermeasures, such as explainability and algorithmic accountability. Courts should not accept legal automation because it could be a hazard for vulnerable and marginalized persons, despite all good intentions. In the last part of his article, Pasquale traces a way to stem the tide of automation in the field of justice and administration, recalling the doctrine of Daniel Farber concerning ‘unconstitutional conditions’, which sets principles and procedures to block governments from requiring waiver of a constitutional right as a condition of receiving some governmental benefit.Footnote ⁵²

Far from a solution that brings us back to an ‘analogic’ world, we agree with Frank Pasquale. In his article, he calls for a more robust and durable theory of constitutionalism to pre-empt the problems that may arise from using automation. However, this is not sufficient, since we need a parallel theory and practice of computer science to consider ethical values and constitutional rights involved in the algorithmic reasoning and to empower officials with the ability to understand when and how to develop and deploy the technology.Footnote ⁵³ Besides, it is necessary to maintain a ‘human-centric’ process in judging for the sake of courts and citizens, who could be destroyed, as Pasquale warns, by the temptation of the acceleration, abbreviation, and automation of decisional processes.

Constitutional Challenges from ‘Emphatic’ Media

Chapter 4, by Peggy Valcke, Damian Clifford, and Viltė Kristina Steponėnaitė, focuses on ‘Constitutional Challenges in the Emotional AI Era’. The emergence of ‘emotional AI’, meaning technologies capable of using computing and artificial intelligence techniques to sense, learn about, and interact with human emotional life (so-called ‘emphatic media’)Footnote ⁵⁴ raises concerns and challenges for constitutional rights and values from the point of view of its use in the business to consumer context.Footnote ⁵⁵

These technologies rely on various methods, including facial recognition, physiological measuring, voice analysis, body movement monitoring, and eye-tracking. The social media business gauges several of these techniques to quantify, track, and manipulate emotions to increase their business profits.

In addition to technical issues about ‘accuracy’, these technologies pose several concerns related to protecting consumers’ fundamental rights and the rights of many other individuals, such as voters and ordinary people. As Peggy Valcke, Damian Clifford, and Viltė Kristina Steponėnaitė claim, emotional AI generates a growing pressure on the whole range of fundamental rights involved with the protection against the misuse of AI, such as privacy, data protection, respect for private and family life, non-discrimination, freedom of thought, conscience, and religion.

Although the authors argue for the necessity of constitutional protection against the possible impacts of emotional AI on existing constitutional freedoms, they ask themselves whether we need new rights in Europe in the light of growing practices of manipulation by algorithms and emotional AI. By highlighting the legal and ethical challenges of manipulating emotional AI tools, the three authors suggest a new research agenda that harnesses the academic scholarship and literature on dignity, individual autonomy, and self-determination to inquiring into the need for further constitutional rights capable of preventing or deterring emotional manipulation.

Algorithmic Surveillance as a New Bureaucracy

Chapter 5 is entitled ‘Algorithmic Surveillance as a New Bureaucracy: Law Production by Data or Data Production by Law?’, in which Mariavittoria Catanzariti explores the vast topic of algorithmic administration. Her argument deals with the legitimation of administrative power, questioning the rise of a ‘new bureaucracy’ in Weberian terms. Like bureaucracy, algorithms have a rational power requiring obedience and excluding non-predictable choices. Whereas many aspects of public administration could undoubtedly benefit from applying machine learning algorithms, their substitution for human decisions would ‘create a serious threat to democratic governance, conjuring images of unaccountable, computerized overlords’.Footnote ⁵⁶

Catanzariti points out that with private sectors increasingly relying on machine learning power, even administration and public authorities, in general, keep pace and make use of the same rationale, giving birth to an automated form of technological rationality. The massive use of classification and measurement techniques affect human activity, generating new forms of power that standardize behaviours for inducing specific conduct. The social power of algorithms is currently visible in the business of many governmental agencies in the United States.

While producing a faster administration, decision-making with algorithms is likely to generate multiple disputes. The effects of algorithmic administration are far from being compliant with the same rationality as law and administrative procedures. Indeed, the use of algorithms determines results that are not totally ‘explainable’, a fact that is often accused of being ‘obscure, crazy, wrong, in short, incomprehensible’.Footnote ⁵⁷

As Catanzariti explains, algorithms are not neutral, and technology is not merely a ‘proxy’ for human decisions. Whenever an automated decision-making technology is included in a deliberative or administrative procedure, it tends to ‘capture’ the process of deciding or make it extremely difficult to ignore it. Consequently, the author argues that law production by data ‘is not compatible with Weberian legal rationality’, or as we have claimed, automation, far from appearing a mere ‘slave’, unveils its true nature of being the ‘master’ of decision-making when employed, due to its ‘practical appeal’.Footnote ⁵⁸ Indeed, algorithms put a subtle but potent spell on administrations: by using them, you can save work, time, and above all, you are relieved of your burden of motivating. Yet is this type of algorithmic administration really accountable? Coming back to Frank Pasquale’s question, are ‘due process’ principles effectively applicable to this kind of decision?

Predictive Policing

Finally, Chapters 6 and 7, ‘Human Rights and Algorithmic Impact Assessment for Predictive Policing’ by Céline Castets-Renard and ‘Law Enforcement and Data-Driven Predictions at the National and EU Level: A Challenge to the Presumption of Innocence and Reasonable Suspicion?’ by Francesca Galli, touch upon the issue of law enforcement and technology.Footnote ⁵⁹ The first addresses the dilemma of human rights challenged by ‘predictive policing’ and the use of new tools such as the ‘Algorithmic Impact Assessment’ to mitigate the risks of such systems. The second explores the potential transformation of core principles of criminal law and whether the techniques of a data-driven society may hamper the substance of legal protection. Both the authors argue for the necessity to protect fundamental rights against the possible increase of coercive control of individuals and the development of a regulatory framework that adds new layers of fundamental rights protection based on ethical principles and other practical tools.

In some countries, police authorities have been granted sophisticated surveillance technologies and much more intrusive investigative powers to reduce crime by mapping the likely locations of future unlawful conduct so that the deployment of police resources can be more effective.Footnote ⁶⁰

Here again, the problem regards the ability and sustainability of decisions by intelligent machines and their consequences for the rights of individuals and groups.Footnote ⁶¹ Machine learning and other algorithmic tools can now correlate multiple variables in a data set and then predict behaviours. Such technologies open new scenarios for information gathering, monitoring, surveilling, and profiling criminal behaviour. The risk here is that predictive policing represents more than a simple shift in tools and could result in less effective and maybe even discriminatory police interventions.Footnote ⁶²

2.4 The Effects of the ‘Algorithmic State’ on the Practice of Liberty

Trying to synthetize some of the most critical issues brought about by the advent of what we call the Algorithmic State on the practice of constitutional liberties, there appear to be two main sensitive areas: surveillance and freedom.

Surveillance

As we have already seen, the rise of the algorithmic state has produced the change foreseen more than forty years ago by Herbert Marcuse. In general, technology is improving people’s lives. However, we know that this improvement comes at a ‘price’. We are increasingly dependent on big-tech-platform services, even if it is clear that they make huge profits with our data. They promise to unchain humans from needs and necessities, but they themselves are becoming indispensable.

Therefore, we are taking for granted that the cost of gaining such benefits – security, efficiency, protection, rewards, and convenience – is to consent to our personal data being recorded, stored, recovered, crossed, traded, and exchanged through surveillance systems. Arguing that people usually have no reason to question surveillance (the ‘nothing to hide’ misconception)Footnote ⁶³ strengthens the order built by the system, and people become ‘normalized’ (as Foucault would have said).Footnote ⁶⁴

Because of this massive use of technology, we are now subject to a new form of surveillance, which profoundly impacts individual freedom, as it is both intrusive and invasive in private life.Footnote ⁶⁵ Both explicit and non-explicit forms of surveillance extend to virtually all forms of human interaction.Footnote ⁶⁶

As the EU Court of Justice pointed out, mass surveillance can be produced by both governments and private companies. This is likely to create ‘in the minds of the persons concerned the feeling that their private lives are the subject of constant surveillance’.Footnote ⁶⁷ In both cases, we have a kind of intrusive surveillance on people’s lives, and this is evidence of individuals’ loss of control over their personal data.

Freedom

This process also affects the very idea of the causal link between individual or collective actions and their consequences, therefore, the core notion of our freedom. Replacing causation with correlation profoundly affects the fundamental distinction embedded in our moral and legal theory between instruments and ends.Footnote ⁶⁸ Today’s cybernetic power is no longer just an instrument to achieve ends decided by human agents. Machines make decisions autonomously on behalf of the person, thus interfering with human freedom.

As it is very clearly described in the following chapters, human agents (individual or collective) explicitly delegate the power to make decisions or express assessments on their behalf to automated systems (judicial support systems, algorithmic administration, emotional assessments, policing decisions). But we must be aware of another crucial dimension of that substitution.

There are two ways to capture human freedom: the first, as we saw in the previously noted cases, occurs whenever we ask a technological system to decide directly on our behalf (we reduce our self-determination to choose our proxy) and the second is when we ask automated machinery to provide the information upon which we take a course of action. Knowledge always shapes our freedom. One key factor (although not the only one) influencing our decisions is the information background we have. Deciding to drive a specific route rather than another to reach our destination is usually affected by information we have either on traffic or roadworks; the choice to vote for one political candidate instead of another depends on the information we get about his or her campaign or ideas. If we ask ourselves which channel we will use today to get information about the world beyond our direct experience, the answer will be more than 80 per cent from the Internet.Footnote ⁶⁹

Automated technological systems increasingly provide knowledge.Footnote ⁷⁰ Simultaneously, ‘individual and collective identities become conceivable as fluid, hybrid and constantly evolving’ as the result of ‘continuous processes bringing together humans, objects, energy flows, and technologies’.Footnote ⁷¹ This substitution profoundly impacts the very idea of autonomy as it emerged in the last two centuries and basically alters the way people come to make decisions, have beliefs, or take action.

In this way, two distinctive elements of our idea of freedoms’ violations seem to change or disappear in the Algorithmic Society. In the first case – when we explicitly ask technology to decide on our behalf – we cannot say that the restriction of our freedom is unwanted or unvoluntary because we ourselves consented to it. We expressly ask those technologies to decide, assuming they are ‘evidence-based’, more effective, more neutral, science-oriented, and so forth. Therefore, we cannot say that our freedom has been violated against our will or self-determination, given that we expressly asked those systems to make our decisions.

On the other hand, when our decisions are taken on the informative basis provided by technology, we can no longer say that such threats to our liberty are ‘external’; as a matter of fact, when we trust information taken from the Internet (from web search engines, like Google, or from social media, like Facebook or Twitter), there is no apparent coercion, no violence. That information is simply welcomed as a sound and valid basis for our deliberations. Yet there is a critical point here. We trust web-sourced information provided by platforms, assuming they are scientifically accurate or at least trustworthy. However, this trust has nothing to do with science or education. Platforms simply use powerful algorithms that learn behavioural patterns from previous preferences to reinforce individuals or groups in filtering overwhelming alternatives in our daily life. The accuracy of these algorithms in predicting and giving us helpful information with their results only occurs because they confirm – feeding a ‘confirmation bias’Footnote ⁷² – our beliefs or, worst, our ideological positions (‘bubble effect’).Footnote ⁷³

There is something deeply philosophically and legally problematic about restricting people’s freedom based on predictions about their conduct. For example, as an essential requirement for a just society, liberal and communitarian doctrines share not only the absence of coercion but also independence and capacity when acting; from this point of view, new algorithmic decision-making affects the very basis of both liberal and communitarian theories. As Lawrence Lessig wrote, we have experienced, through cyberspace, a ‘displacement of a certain architecture of control and the substitution with an apparent freedom.’Footnote ⁷⁴

Towards the Algorithmic State Constitution: A ‘hybrid’ Constitutionalism

Surveillance capitalism and the new algorithmic threats to liberty share a common feature: when a new technology has already appeared, it is often too late for the legal system to intervene. The gradual anticipation in the field of privacy rights, from subsequent to preventive (from protection by regulation, to protection ‘by design’ and finally ‘by default’), exactly traces this sort of ‘backwards’ trajectory. This is the main feature of the Algorithmic State constitutionalism.

It is necessary to incorporate the values of constitutional rights within the ‘design stage’ of the machines; for this, we need what we would define as a ‘hybrid’ constitutional law – that is, a constitutional law that still aims to protect fundamental human rights and at the same time knows how to express this goal in the language of technology.Footnote ⁷⁵ Here the space for effective dialogue is still abundantly unexplored, and consequently, the rate of ‘hybridization’ is still extraordinarily low.

We argue that after the season of protection by design and by default, a new season ought to be opened – that of protection ‘by education’, in the sense that it is necessary to act when scientists and technologists are still studying and training, to communicate the fundamental reasons for general principles such as personal data protection, human dignity, and freedom protection, but also for more specific values as the explainability of decision-making algorithms or the ‘human in the loop’ principle.

Technology is increasingly integrated with the life of the person, and this integration cannot realistically be stopped, nor it would be desirable, given the huge importance for human progress that some new technologies have had.

The only possible way, therefore, is to ensure that the value (i.e., the meaning) of protecting the dignity of the person and his or her freedom becomes an integral part of the training of those who will then become technicians. Hence the decisive role of school, university, and other training agencies, professional or academic associations, as well as the role of soft law.

Footnotes

¹ Pedro Domingos, The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World (Basic Books 2015).

² One of the most prominent prophets of the idea of a new kind of progress generated through the use of technologies is surely Jeremy Rifkin. See his book The Zero Marginal Cost Society: The Internet of Things, the Collaborative Commons, and the Eclipse of Capitalism (St. Martin’s Press 2014).

³ Marshall McLuhan and Quentin Fiore, The Medium Is the Massage (Ginko Press 1967).

⁴ Committee of Experts on Internet Intermediaries of the Council of Europe (MSI-NET), ‘Algorithms and Human Rights. Study on the Human Rights Dimensions of Automated Data Processing Techniques and Possible Regulatory Implications’ (2016) DGI(2017)12.

⁵ According to the European Parliament, ‘Resolution of 16 February 2017 with recommendations to the Commission on Civil Law Rules on Robotics (2015/2103(INL))’ (P8_TA(2017)0051, Bruxelles), ‘a robot’s autonomy can be defined as the ability to take decisions and implement them in the outside world, independently of external control or influence.’

⁶ According to Statista, Facebook is the biggest social network platform worldwide, with more than 2.7 billion monthly active users in the second quarter of 2020. During the last reported quarter, the company stated that 3.14 billion people were using at least one of the company’s core products (Facebook, WhatsApp, Instagram, or Messenger) each month. To the contrary, as of the end of 2019, Twitter had 152 million monetizable daily active users worldwide.

⁷ Jack M. Balkin, ‘Free Speech in the Algorithmic Society: Big Data, Private Governance, and New School Speech Regulation’ (2017) 51 UCDL Rev 1149; Agnieszka M. Walorska, ‘The Algorithmic Society’ in Denise Feldner (ed), Redesigning Organizations Concepts for the Connected Society (Springer 2020); Giovanni De Gregorio, ‘From Constitutional Freedoms to the Power of the Platforms: Protecting Fundamental Rights Online in the Algorithmic Society’ (2018) 11 Eur J Legal Stud 65.

⁸ Frank Webster, Theories of the Information Society, 4th ed. (Routledge 2014).

⁹ Neil M. Richards, ‘The Dangers of Surveillance’ (2012) 126 Harv L Rev 1934;

¹⁰ Jonathan Zittrain, The Future of the Internet – And How to Stop It (Yale University Press 2008); Cary Coglianese and David Lehr, ‘Regulating by Robot: Administrative Decision Making in the Machine-Learning Era’ (2016) 105 Geo LJ 1147. Responsibility for the production of recent anxiety in large part can be attributed to Nick Bostrom, Superintelligence (Oxford University Press 2014).

¹¹ E. Morozov uses the expression ‘digital solutionism’ to name the idea that technological innovation should solve every social problem. Evgeny Morozov, To Save Everything, Click Here: The Folly of Technological Solutionism (Public Affairs 2013).

¹² Herbert Marcuse, One-Dimensional Man: Studies in the Ideology of Advanced Industrial Society, 2nd ed. (Beacon Press 2019) 1.

¹³ Michael Veale and Lilian Edwards, ‘Clarity, Surprises, and Further Questions in the Article 29 Working Party Draft Guidance on Automated Decision-Making and Profiling’ (2018) 34 Computer Law Security Review 398.

¹⁴ European Commission, ‘White Paper on Artificial Intelligence: A European Approach to Excellence and Trust’ (COM (2020) 65 final, Bruxelles).

¹⁵ Inga Kroener and Daniel Neyland, ‘New Technologies, Security and Surveillance’ in Kirstie Ball, Kevin Haggerty, and David Lyon (eds), Routledge Handbook of Surveillance Studies (Routledge 2012) 141.

¹⁶ Radha D’Souza, The Surveillance State: A Composition in Four Movements, (Pluto 2019); Cathy O’Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (Broadway Books 2016).

¹⁷ Shoshana Zuboff, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power (PublicAffairs 2019).

¹⁸ Natalie Ram and David Gray, ‘Mass Surveillance in the Age of COVID-19’ (2020) 7 Journal of Law and the Biosciences 1.

¹⁹ In the broadest sense, algorithms are encoded procedures for solving a problem by transforming input data into a desired output. As we know the excitement surrounding Big Data is largely attributable to machine learning. Paul Dourish, ‘Algorithms and Their Others: Algorithmic Culture in Context’ (2016) 3 Big Data & Society 1; Tarleton Gillespie, ‘The Relevance of Algorithms’ in Tarleton Gillespie, Pablo J. Boczkowski. and Kirsten A. Foot (eds), Media Technologies: Essays on Communication, Materiality, and Society (MIT Press 2014); Viktor Mayer-Schoenberger and Kenneth Cukier, Big Data: A Revolution That Will Transform How We Live, Work, and Think (Houghton Mifflin Harcourt 2013).

²⁰ This idea – sometimes abbreviated as XAI (explainable artificial intelligence) – means that machines could give access to data about their own deliberative processes, simply by recording them and making them available as data structures. See Wojciech Samek and Klaus-Robert Müller, ‘Towards Explainable Artificial Intelligence’ in Wojciech Samek et al. (eds), Explainable AI: Interpreting, Explaining and Visualizing Deep Learning (Springer 2019); Tim Miller, ‘Explanation in Artificial Intelligence: Insights from the Social Sciences’ (2019) 267 Artificial Intelligence 1 Brent Mittelstadt, Chris Russell, and Sandra Wachter, Explaining Explanations in AI (ACM 2019).

²¹ Frank Pasquale, The Black Box Society: The Secret Algorithms That Control Money and Information (Harvard University Press 2015); Tal Z. Zarsky, ‘Understanding Discrimination in the Scored Society’ (2014) 89 Wash L Rev 1375.

²² See, for example, the ‘Proposal for a Regulation of the European Parliament and of the Council on a Single Market for Digital Services (Digital Services Act)’ and amending Directive 2000/31/EC, COM (2020) 825 final, 15.12.2020, and the proposals for a Digital Services Act (DSA), COM/2020/842 final, 15.12.2020, and for an Artificial Intelligence Act, COM (2017) 85 final, 21.4.2021.

²³ Danielle Keats Citron and Frank Pasquale, ‘The Scored Society: Due Process for Automated Predictions’ (2014) 89 Wash L Rev 1.

²⁴ Giovanni Sartori, ‘Constitutionalism: A Preliminary Discussion’ (1962) 56 American Political Science Review 853.

²⁵ Being that constitutional theory is part of the legal system, this feature distinctively differentiates constitutional law from political philosophy or political sociology.

²⁶ Giorgio Pino, Il costituzionalismo dei diritti struttura e limiti del costituzionalismo contemporaneo (il Mulino 2017).

²⁷ Benjamin Constant, ‘De la liberté des anciens comparée à celle des modernes’ in Collection complète des ouvrages: publiés sur le gouvernement représentatif et la constitution actuelle ou Cours de politique constitutionelle (Plancher 1820).

²⁸ Richard Bellamy, ‘Constitutionalism’ in Bertrand Badie, Dirk Berg-Schlosser, and Leonardo Morlino (eds), International Encyclopedia of Political Science, vol. 2 (SAGE 2011).

²⁹ Andrea Buratti, Western Constitutionalism: History, Institutions, Comparative Law (Springer-Giappichelli 2019).

³⁰ Martin Loughlin, Foundations of Public Law (Oxford University Press 2010).

³¹ Jean Bodin, On Sovereignty: Four Chapters from the Six Books of the Commonwealth (Cambridge University Press 1992).

³² Carl Schmitt, Political Theology: Four Chapters on the Concept of Sovereignty (University of Chicago Press 1985).

³³ Absolutism was the crucible in which this modern concept of sovereignty was forged. As J. Bodin expressed, ‘sovereignty’ is ‘the greatest power of command’ and is ‘not limited either in power, charge, or time certain’. Jean Bodin, Les six livres de la république (Jacques du Puis 1576).

³⁴ As Hart asserted: ‘In any society where there is law, there actually is a sovereign, characterized affirmatively and negatively by reference to the habit of obedience: a person or body of persons whose orders the great majority of the society habitually obey and who does not habitually obey any other person or persons.’ Herbert L. A. Hart, The Concept of Law, 1st ed. (Oxford University Press 1961).

³⁵ Regarding this distinction, see James Bryce, ‘Flexible and Rigid Constitutions’ (1901) 1 Studies in History and Jurisprudence 145, 124.

³⁶ In Marbury v. Madison, 5 U.S. (1 Cr.) 137, 173–180 (1803), Chief Justice Marshall gave the Constitution precedence over laws and treaties, providing that only laws ‘which shall be made in pursuance of the constitution’ shall be ‘the supreme law of the land’. For further information on this topic, see generally Bruce Ackerman, ‘The Rise of World Constitutionalism’ (1997) Virginia Law Review 771; Alec Stone Sweet, Governing with Judges: Constitutional Politics in Europe (Oxford University Press 2000); Ronald Dworkin, Freedom’s Law: The Moral Reading of the American Constitution (Oxford University Press 1999); Michaela Hailbronner, ‘Transformative Constitutionalism: Not Only in the Global South’ (2017) 65 The American Journal of Comparative Law 527; and Mark Tushnet, Advanced Introduction to Comparative Constitutional Law (Edward Elgar 2018). For a specific insight into the Italian experience, see Vittoria Barsotti et al., Italian Constitutional Justice in Global Context (Oxford University Press 2016), 263; Maurizio Fioravanti, ‘Constitutionalism’ in Damiano Canale, Paolo Grossi, and Basso Hofmann (eds), A Treatise of Legal Philosophy and General Jurisprudence, vol. 9 (Springer 2009).

³⁷ In the view of Bodei, from ‘algorithmic capitalism’ (which will use artificial intelligence and robotics to increasingly link economics and politics to certain forms of knowledge) originates a new ‘occult’ power in which ‘the human logos will be more and more subject to an impersonal logos’. See Remo Bodei, Dominio e sottomissione. Schiavi, animali, macchine, Intelligenza Artificiale (il Mulino 2019).

³⁸ Frank Pasquale, ‘Two Narratives of Platform Capitalism’ (2016) 35 Yale L & Pol’y Rev 309.

³⁹ Lawrence Delevingne, ‘U.S. Big Tech Dominates Stock Market after Monster Rally, Leaving Investors on Edge’, Reuters (28 August 2020) www.reuters.com/article/us-usa-markets-faangs-analysis-idUSKBN25O0FV.

⁴⁰ Apple Inc, AAPL.O; Amazon.com Inc., AMZN.O; Microsoft Corp, MSFT.O; Facebook Inc., FB.O; and Google parent Alphabet Inc., GOOGL.O.

⁴¹ Nicolas Petit, Big Tech and the Digital Economy: The Moligopoly Scenario (Oxford University Press 2020).

⁴² Footnote Ibid.

⁴³ Airbnb, for example, has developed market power to shape urban planning in smaller cities in the United States. Amazon has received offers from democratically elected mayors to assume political power when the company moves its headquarters to these cities. More importantly, Facebook has become one of the most important actors in political campaigns all over the world, not to mention the famous and controversial case of Cambridge Analytica, when we experienced the disruptive force of the social network for people’s lives in terms of political participation, data protection, and privacy. See Emma Graham-Harrison and Carole Cadwalladr, ‘Data Firm Bragged of Role in Trump Victory’ The Guardian (21 March 2018) https://www.theguardian.com/uk-news/2018/mar/20/cambridge-analytica-execs-boast-of-role-in-getting-trump-elected.

⁴⁴ Frank Pasquale, ‘From Territorial to Functional Sovereignty: The Case of Amazon’ (accessed 6 December 2020) http://lpeblog.org/2017/12/06/from-territorial-to-functional-sovereignty-the-case-of-amazon/; Denise Feldner, ‘Designing a Future Europe’ in Denise Feldner (ed), Redesigning Organizations: Concepts for the Connected Society (Springer 2020).

⁴⁵ Andrea Simoncini, ‘Sovranità e potere nell’era digitale’ in Tommaso Edoardo Frosini et al. (eds), Diritti e libertà in internet (Le Monnier Università 2017) 19.

⁴⁶ Wiener decided to call ‘the entire field of control and communication theory, whether in the machine or in the animal, by the name Cybernetics, which we form from the Greek χυβερνήτης or steersman’. See Norbert Wiener, Cybernetics or Control and Communication in the Animal and the Machine (2nd reissued edn, MIT Press 2019) 18.

⁴⁷ Ugo Pagallo, ‘Algo-Rhythms and the Beat of the Legal Drum’ (2018) 31 Philosophy & Technology 507.

⁴⁸ Nicol Turner Lee, ‘Detecting Racial Bias in Algorithms and Machine Learning’ (2018) 16 Journal of Information, Communication and Ethics in Society 252; Jack M. Balkin, ‘Free Speech in the Algorithmic Society: Big Data, Private Governance, and New School Speech Regulation’ (2017) 51 UCDL Rev 1149; Ryan Calo, ‘Privacy, Vulnerability, and Affordance’ (2016) 66 DePaul L Rev 591; Oreste Pollicino and Laura Somaini, ‘Online Disinformation and Freedom of Expression in the Democratic Context’ in Sandrine Boillet Baume, Véronique Martenet Vincent (eds), Misinformation in Referenda (Routledge 2021).

⁴⁹ O’Neil, Weapons of Math Destruction.

⁵⁰ ‘Robotics Ethics’ (SHS/YES/COMEST-10/17/2 REV, Paris); Jon Kleinberg et al., ‘Discrimination in the Age of Algorithms’ (2018) 10 Journal of Legal Analysis 113; McKenzie Raub, ‘Bots, Bias and Big Data: Artificial Intelligence, Algorithmic Bias and Disparate Impact Liability in Hiring Practices’ (2018) 71 Ark L Rev 529.

⁵¹ Pasquale, The Black Box Society.

⁵² Daniel A. Farber, ‘Another View of the Quagmire: Unconstitutional Conditions and Contract Theory’ (2005) 33 Fla St UL Rev 913 914.

⁵³ Coglianese and Lehr, ‘Regulating by Robot’.

⁵⁴ Andrew McStay, Emotional AI: The Rise of Empathic Media (SAGE 2018).

⁵⁵ Vian Bakir and Andrew McStay, ‘Empathic Media, Emotional AI, and the Optimization of Disinformation’ in Megan Boler and Elizabeth Davis (eds), Affective Politics of Digital Media (Routledge 2020) 263.

⁵⁶ Coglianese and Lehr, ‘Regulating by Robot’, 1152.

⁵⁷ Andrea Simoncini, ‘Amministrazione digitale algoritmica. Il quadro costituzionale’ in Roberto Cavallo Perin and Diana-Urania Galletta (eds), Il diritto dell’amministrazione pubblica digitale (Giappichelli 2020) 1.

⁵⁸ Andrea Simoncini, ‘Profili costituzionali dell’amministrazione algoritmica’ (2019) Riv trim dir pubbl 1149.

⁵⁹ Andrew Guthrie Ferguson, The Rise of Big Data: Surveillance, Race and the Future of Law Enforcement (New York University Press 2017).

⁶⁰ Lyria Bennett Moses and Janet Chan, ‘Algorithmic Prediction in Policing: Assumptions, Evaluation, and Accountability’ (2018) 28 Policing and Society 806; Gavin J. D. Smith, Lyria Bennett Moses, and Janet Chan, ‘The Challenges of Doing Criminology in the Big Data Era: Towards a Digital and Data-Driven Approach’ (2017) 57 British Journal of Criminology 259; Wim Hardyns and Anneleen Rummens, ‘Predictive Policing as a New Tool for Law Enforcement? Recent Developments and Challenges’ (2018) 24 European Journal on Criminal Policy and Research 201.

⁶¹ Karen Yeung, ‘Algorithmic Regulation: A Critical Interrogation’ (2018) 12 Regulation & Governance 505.

⁶² Albert Meijer and Martijn Wessels, ‘Predictive Policing: Review of Benefits and Drawbacks’ (2019) 42 International Journal of Public Administration 1031.

⁶³ Bruce Schneier, Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World (W. W. Norton & Company 2015).

⁶⁴ David Lyon, Surveillance after September 11 (Polity 2003).

⁶⁵ Surveillance consists of the ‘collection and processing of personal data, identifiable or not, for the purpose of influencing or controlling those to whom they belong’. Surveillance is a necessary correlative of a risk-based new idea of state power. See David Lyon, Surveillance Society: Monitoring Everyday Life (Open University Press 2001).

⁶⁶ J. Guelke et al., ‘SURVEILLE Deliverable 2.6: Matrix of Surveillance Technologies’ (2013) Seventh Framework Programme Surveillance: Ethical Issues, Legal Limitations, and Efficiency, FP7-SEC-2011-284725.

⁶⁷ Joined cases C-293/12 and C-594/12, Digital Rights Ireland (C-293/12) and Seitlinger (C-594/12), EU:C:2014:238, at 37.

⁶⁸ Andrea Simoncini and Samir Suweis, ‘Il cambio di paradigma nell’intelligenza artificiale e il suo impatto sul diritto costituzionale’ (2019) 8 Rivista di filosofia del diritto 87.

⁶⁹ According to Statista in 2019, around the 50 per cent of the Italian population accesses information from the Internet.

⁷⁰ Brent Daniel Mittelstadt et al., ‘The Ethics of Algorithms: Mapping the Debate’ (2016) 3 Big Data & Society 1.

⁷¹ Holger Pötzsch, ‘Archives and Identity in the Context of Social Media and Algorithmic Analytics: Towards an Understanding of iArchive and Predictive Retention’ (2018) 20 New Media & Society 3304.

⁷² Andreas Kappes et al., ‘Confirmation Bias in the Utilization of Others’ Opinion Strength’ (2020) 23 Nature Neuroscience 130.

⁷³ Users tend to aggregate in communities of interest causing reinforcements and support of conformation bias, segregation, and polarization. Erik Longo, ‘Dai big data alle “bolle filtro”: nuovi rischi per i sistemi democratici’ (2019) XII Percorsi costituzionali 29.

⁷⁴ Lawrence Lessig, Code. Version 2.0 (Basic Books 2006) 2.

⁷⁵ This is the reason we use the ‘hybrid’ image, coming from the world of the automotive industry: a ‘hybrid’ vehicle means it uses both classical combustion engines and electric power.