Part 1

Algorithmic Discrimination

Understanding the algorithmic judgment

There has been a lot of talk lately about algorithmic discrimination. But does it exist? If so, how does it happen? To what extent can the risks of using algorithmic systems compromise their expected benefits? What must be done so that the use of algorithmic systems is reconciled with protecting of fundamental rights and personal data, as recommended by the Constitution and by the General Data Protection Regulation?

To better situate the debate, it is important to understand what algorithms are. And why they have been vastly used and what are their repercussions on our lives at the individual and collective levels.

Algorithms are formulas or processes for performing tasks, solving problems, analyzing data, and making decisions. They have existed for a long time, and it is common to refer to the algorithm of Euclid, the famous Greek mathematician, as one of the first — if not the first — initiatives in this direction. Only more recently, in the 20th century, algorithms were seen in computer science as finite sequences of executable actions for the solution of a specific problem.

The most recent controversy, in which the discussion of algorithmic discrimination shows, concerns the use of increasingly complex and sophisticated algorithms to solve problems that are not objective and technical. Powered by increasingly larger databases, in big data, algorithms have been used for decisions and tasks that involve qualitative and subjective analyses. These are marked by a high-value load, such as classification, ranking, and creation of people’s profiles.

More than that, algorithms have been seen as keys to understanding the past, diagnosing the present, and foreseeing the future, through predictions and predictive analysis of people, both individually and collectively.

Data from the past and present are scrutinized by algorithmic systems that intend to assess people based on objective criteria or metrics. It takes different degrees of coverage: the assessment of users in an application, the evaluation of people to map the possibility and conditions of access to a particular service — such as credit scoring. Even holistic attempts to classify people as with the concept of social scoring that has been used by the Chinese government to define the quality of its citizens.

As noted, algorithmic systems have been key to maintaining a gear that aims to decide and predictions about people regarding their most diverse characteristics: their merits, their profiles, their preferences, their inclinations, and probabilities in different fields — from the tendency to consume a product to the propensity to commit crimes or to relapse in a crime -, their capacities and aptitudes — in the physical, intellectual, emotional, professional, economic fields, etc -, and their weaknesses and vulnerabilities.

Such judgments are not innocent, since they are implemented with very clear scopes. Within the market, such diagnoses allow economic agents distinct possibilities of action. Customer segmentation can define when to charge different prices — according to certain criteria or the propensity of each one to acquire a product or service or even exploring their vulnerabilities — either to deny access to certain services or specific conditions.

The issue is not restricted to the market, although this aspect, by itself, is already quite worrying. It also involves relevant dimensions of citizens’ private autonomy. There are algorithmic systems that decide who will enter universities or have access to jobs, positions, and professional opportunities, among countless other situations in which people’s life opportunities depend on these judgments.

Sometimes, it is human life itself that will be the object of the algorithmic decision. Autonomous cars may have to decide, in extreme situations, which lives should be spared or prioritized in the face of an imminent accident.

In the same way, the problem also enters into the realm of public autonomy and democracy. The state can also use algorithmic systems to, for example, classify citizens according to their political convictions and their level of support or not to a given government. This information can be used for all sorts of discrimination and persecution.

When dealing with the democratic issue, an important issue is also highlighted, which concerns the use of algorithmic systems for political purposes, to explore the knowledge they have of people, including about their weaknesses, to manipulate them to change the outcome of electoral processes.

The debate on algorithms must start from the premise that the problem is not just a reflection of a data-driven economy. A society that starts to be data-driven ties the future of its people to algorithmic judgments.

While we understand the growing importance of algorithmic qualification today, it also leads us to a question: given the source of big data, is it possible to use algorithms for any analysis? Is it enough for an economic agent or the state to desire to label their consumers or citizens for them to do so? Or are there limits to such judgments, as with the General Data Protection Regulation principle and goal?

We are so used to algorithmic judgments that we do not always reflect on questions that should be basic: does a platform have the right to measure the consumer and give him a grade? Does the state have the right to grade its citizens based on their political preferences or any other sensitive data? Based on what criteria can economic agents create consumer profiles? Is it possible to make a comprehensive analysis of citizens similar to a social scoring model?

If there are many issues concerning algorithmic judgments, they are necessary. They can help to circumvent many of the shortcomings of human judgments. The issue is that we cannot stop questioning these parameters and accept that algorithms dominate the decisions of our societies.