Nicolas Kayser-Bril (Algorithm Watch, Berlin) is journalist, author and trainer and will speak at the “Super-Scoring?” conference on 11 October about “Personal Scoring in the EU: Not quite Black Mirror yet, at least if you‘re rich” (here his essay as PDF). We asked him 7 questions about “super-scoring”. We take “super-scoring” to mean any accumulation and combination of personal data taken from different contexts and sources and used in an algorithmic production of one functional value that is intended to evaluate human behaviour.
1. What is your evaluation of super-scoring as a tool for social control?
Nicolas Kayser-Bril: “Superscoring” is a tool like any other. What it achieves is entirely up to what people in power decide.
Whether a Superscoring system is used to ensure that drivers are automatically fined when they drive over the speed limits (most new cars are GPS-tracked and connected) or to ensure that asylum-seekers are automatically punished when they use their government-issued debit card to make a purchase outside of the region where they are supposed to stay (or both) is a political issue.
The specificity of Superscoring or automated decision-making lies in its potential for opacity. Unless great care is taken to make such systems transparent and accountable, their decisions can be arbitrary and leave people responsible for or subject to a Superscoring system without any recourse. When a policeperson pepper-sprays a demonstrator, the facts can be analyzed by all parties based on tangible evidence. When an automated program denies social welfare to a citizen, it can be impossible to obtain any tangible item on which to base a claim.
2. Where and for whom do you expect to see the greatest opportunities and risks?
Nicolas Kayser-Bril: Superscoring systems could be used to restrain the power of the most powerful just like they can be used to oppress those who are already oppressed. The way a technology is used is never determined in advance.
3. Do you anticipate upcoming conflicts between super-scoring systems and established values and norms?
Nicolas Kayser-Bril: Norms and values can be attacked regardless of the technology. Of course, the opacity in which most government and corporate software is shrouded makes it hard to uphold some of the foundations of a fair society, such as the possibility to see and understand the evidence someone holds against you. However, while it might be true that Superscoring helps in carrying out actions that curtail the rights of citizens, it could just as well be used to increase fundamental rights, e.g. by helping institutions root out bias in their decision-making.
4. How do you expect super-scoring processes to shift with the implementation of certain digital technologies (such as smartphone tracking, face and voice recognition etc.)?
Nicolas Kayser-Bril: The ability to collect, store and compute large amounts of data, made possible by the parallelization and distribution of computing in the 2000s, helped create more Superscoring processes, and to make them cheaper. However, because most of them remain opaque, it’s hard to say if this trend made Superscoring processes less or more efficient. However, even if they might not change much themselves, they become more numerous and, because of their interconnection, more complex.
5. What kind of interplay can we expect between the digital economy and politics if super-scoring systems are implemented – on a regional, national or international level?
Nicolas Kayser-Bril: I’m not sure that making a clear-cut distinction between the political and corporate sphere helps us grasp the issue. Because Superscoring is defined as “influencing collective behavior”, it must be all-encompassing and has to involve both spheres. Do social networks play with our emotions to make us dependent? Of course, but that is not only a corporate issue. If they influence collective behavior, it is because governments refused to regulate social networks in the first place, as they did with other addictive products (alcohol, tobacco etc.). Governments conduct mass surveillance and treat their population as suspects? Of course, but they can only do it by leveraging tools and databases that were built by corporations. Influencing collective behavior requires that both spheres be intertwined both personally and ideologically.
6. Which aspects of super-scoring should concrete educational measures address? Where and how would you begin to do so?
Nicolas Kayser-Bril: Superscoring is a tool that helps wielding power like any other. I believe that a first step to help those interested in the topic would be to insist on its similarities with other technologies and help people see who is empowered or disempowered by Superscoring (and this, in turn, might require an understanding of some technical concepts, but is certainly not reserved to an elite of developers).
7. What aspects of super-scoring do you believe to be under-represented in current public discourse? What other questions should be raised?
Nicolas Kayser-Bril: It all depends on what you mean by “public debate”: discussions in Parliament? Coverage by the news outlets that are read by members of Parliament? The most-shared items on Whatsapp? From what I follow, I feel that, more than any specific aspect, the framing of some conversations is unhelpful. Seeing Superscoring as a purely government-driven, dystopian nightmare or as a benevolent and neutral tool that eliminates human error can only lead to misunderstandings. Instead, a systematic and holistic view of Superscoring systems would help us better assess their impact and their potential.
Thank you so much!