The Legitimacy of Global Social Indicators : Reconfiguring Authority, Accountability and Accuracy La légitimité des indicateurs sociaux globaux : repenser l’autorité, la responsabilisation et l’exactitude La legitimidad de los indicadores sociales mundiales : reconfigurando la autoridad, la responsabilidad y la exactitud

Abstract

This paper was written in response to an invitation by the organisers of the 2016 Brussels “colloquium on inter-normativity” to address “law-like” or “law related” transnational normative phenomena so as to assess inter alia, “the validity and legitimacy of social indicators such as university rankings, credit rating agencies indicators, rankings of legal regimes and other attempts to provide indicators in areas of social law, human rights law and business law”. The first part discusses the definition of global social indicators, how they relate to other normative materials — from law to algorithms — and what could be meant by asking for them to be legitimate. The second part then examines the implications of debates about and descriptions of indicators for three key aspects of legitimacy : authority, accountability and accuracy. It suggests that indicators are indeed assessed in terms of these criteria but that they also serve to reconfigure and transform them. As noted in the concluding section the paper argues that the relationship between the three aspects of legitimacy identified needs further examination, but that the challenge they pose is more fundamental. Objections to indicators turn on the ideas of the good society they embody, the part they play in constructing the relationship between more or less economically developed societies, and the kind of governance model they reflect — for example their tendency to substitute technicalities for political participation and their link to audit culture. On the other hand, even if caution is needed in endorsing the use of indicators in projects of governance, they do often have advantages of publicness, contestability and openness to violation when compared to the spread of algorithmic regulation.

Résumé

Le présent article a été écrit à la suite d’une invitation lancée par les organisateurs du colloque sur l’internormativité, tenu à Bruxelles en 2016, pour analyser les phénomènes normatifs transnationaux qui s’apparentent à la norme juridique afin d’évaluer, entre autres, « la validité et la légitimité des indicateurs sociaux tels que les classements des universités, les indicateurs des agences de notation de crédit, le classement de régimes juridiques et autres tentatives de fournir des indicateurs dans le domaine du droit social, des droits fondamentaux et du droit des affaires ». La première partie du texte se penche sur la définition d’indicateurs sociaux globaux, leurs interactions avec d’autres normes — des normes juridiques aux algorithmes — et le sens de leur légitimité. Dans la seconde partie, j’analyse la description d’indicateurs ainsi que les conséquences des débats les entourant au sujet de trois critères fondamentaux de la légitimité : l’autorité, la responsabilisation et l’exactitude. Cette seconde partie suggère que les indicateurs sont effectivement évalués par rapport à ces critères, mais qu’ils servent aussi à les reconfigurer et transformer. En conclusion, je propose que la relation entre les trois critères de la légitimité mentionnés doit être examinée plus en profondeur, mais que le défi qu’ils posent est plus fondamental. Les critiques contre les indicateurs portent sur les idées de la bonne société qu’ils incarnent, le rôle qu’ils jouent dans la construction de la relation entre les sociétés plus ou moins développées, et la sorte de modèle de gouvernance qu’ils reflètent, par exemple la tendance à substituer des normes techniques à des participations politiques et leur lien avec la culture de l’audit. Par ailleurs, même si la prudence est de mise dans l’utilisation d’indicateurs dans des projets de gouvernance, ils présentent l’avantage d’être publics, contestables et ouverts au non-respect lorsqu’ils sont comparés à l’étendue de la régulation par les algorithmes.

Resumen

Este artículo fue redactado tras la invitación realizada por los organizadores del colloque sur l’internormativité llevado a cabo en Bruselas en el año 2016, con el fin de analizar los fenómenos normativos trasnacionales que se relacionan con la norma jurídica y con el fin de evaluar, entre otros aspectos, « la validez y la legitimidad de los indicadores sociales tales como la clasificación de las universidades, los indicadores de las agencias de calificación crediticia, la clasificación de los regímenes jurídicos y otros tipos de tentativas para así  suministrar los indicadores en el campo del derecho social, de los derechos fundamentales y del derecho comercial ». En la primera parte del texto se realiza una reflexión sobre la definición de los indicadores sociales mundiales, sus interacciones con otras normas — reglas jurídicas y de algoritmos — y el sentido de su legitimidad. En la segunda parte, se analiza la descripción de los indicadores, así como los resultados de los debates vinculados con tres criterios fundamentales de la legitimidad : autoridad, rendición de cuentas y exactitud.  En esta segunda parte se plantea también el hecho que los indicadores se evalúan de acuerdo con estos criterios, sin embargo, sirven del mismo modo para reconfigurarlos y transformarlos.  Finalmente, se expone que la relación entre los tres mencionados criterios de la legitimidad debe ser examinada detalladamente, no obstante, el desafío propuesto es aún más fundamental. Las críticas formuladas en contra de los indicadores tratan sobre las ideas de la buena sociedad que representan, del rol que juegan en la construcción de la relación entre las sociedades más o menos desarrolladas, y del tipo de modelo de gobernanza que reflejan, por ejemplo, la tendencia que existe para sustituir normas técnicas por la participación política, y su relación con la cultura de la auditoría. Por otra parte, se requiere prudencia al emplear los indicadores en los proyectos de gobernanza, ya que poseen la prerrogativa de ser públicos, cuestionables y susceptibles a no ser respetados cuando se les compara con la amplitud de la normativa algorítmica.

Niewiedza tutaj jest zapracowana, ciagle coś liczy, porównuje, mierzy, wyciaga z tego wnioski i pierwiastki.

Wislawa Szymborska[1]

In what ways is the growth of the transnational or so-called “global” dimension of social interactions reliant on and productive of new forms of normativity ? From the perspective of the social theory of law it cannot be sufficient to limit discussion to the question whether or not given examples of normative ordering count as law as defined conventionally. Advancing such a debate requires closer examination of their forms and functions. The 2016 Brussels “colloquium on inter-normativity” therefore invited participants to address “law-like” or “law related” transnational normative phenomena so as to assess inter alia, “the validity and legitimacy of social indicators such as university rankings, credit rating agencies indicators, rankings of legal regimes and other attempts to provide indicators in areas of social law, human rights law and business law”.

This paper focuses on the issues that need to be addressed in thinking about the legitimacy of such indicators. The first part discusses the definition of global social indicators, how they relate to other normative materials and what could be meant by asking for them to be legitimate. The second part then examines the implications of debates about and descriptions of indicators for three key aspects of legitimacy : authority, accountability and accuracy. It suggests that indicators are indeed assessed in terms of these criteria but that they also serve to transform them. The concluding section develops earlier points so as to explain why it is that authors disagree about whether indicators should be (further) legitimated.

The point of this essay is to see what can be learned concerning the legitimacy of global social indicators by bringing together more abstract discussions of the topic with the substantive insights made possible by descriptive case-studies of different kinds of indicators[2]. It shows that moving between the general and the particular is important not only because of the need to incorporate the sometimes unexpected findings of empirical research, but also because what is true of some indicators — such as those concerning human rights — may not necessarily be generalizable to others.

1 Global Social Indicators and their Legitimacy

Before we can address the issue of their legitimacy we must first define what we mean by global social indicators. What are global social indicators ? How do they relate to other forms of normativity ? What does it mean to talk of them being legitimate ?

1.1 Defining Global Social Indicators

The term social indicators can be used in at least three rather different ways[3].

1. We can speak of explanatory indicators, meaning by this the indices, proxies of variables that social scientists use when seeking to construct explanations of variance in furtherance of theoretical claims about connections between variables — for example between corruption and economic development. Such correlations can then be tested for significance by statistical means.

2. Alternatively, we could be discussing policy indicators, i.e. the targets or benchmarks used in setting and assessing the results of social interventions. Such indicators help us establish whether given social interventions have achieved their intended outcomes[4].

3. Finally, and most relevantly for our purposes, we could be referring to ranking indicators, that is to say measures or standards[5] formulated as part of large-scale scaling exercises that stipulate and compare the performance of units on a range of matters ranging from better business opportunities to happiness.

As Shore and Wright write,

[v]irtually every aspect of contemporary professional life and organisational behaviour is now subject to elaborate systems of audit and inspection : everything from the provision of public services, education, policing and security, to health care, safety, energy conservation, information systems and the performance of individuals. Even intangible phenomena such as levels of “trust”, “perceptions of corruption”, “quality of life years” and “gross national happiness” are now routinely quantified, measured and ranked in competitive national and international league tables that purport to offer at-a-glance comparisons[6].

Many of these standards may be intended to be global in their reach even if, as we shall see, they may often represent particular and particularistic interests and values.

It is of course difficult to generalise about what makes for legitimation when indicators are used in many different contexts and for a wide variety of purposes. The collections by Merry and her colleagues include studies of indicators as different as those concerned with human rights and financial ratings, the Programme for International Student Assessment (PISA) for comparing educational achievement, university ranking, European union accession, human trafficking, corruption, better business, rule of law, law and development, millennium development goals, corporate responsibility and financial rating, settlement statistics and projects of land restitution. The range of purposes for which indicators may be used is similarly amorphous. There are as yet no comprehensive mappings of the full variety of indicators or shared criteria for classifying them. Some distinguish indicators that play a role in (1) statecraft, e.g. human trafficking indicators, (2) international governance, e.g. The World Bank worldwide governance indicator, (3) private market governance, e.g. rating agencies, and (4) transnational advocacy[7]. For their part, Siems and Nelken distinguish indicators that relate to law and development, good governance and rule of law ; personal rights and economic freedom ; human development and political stability ; the performance of educational and commercial entities indicators[8].

With such a range of their applications it is important to try to identify what ranking social indicators have in common. Sally Merry and her colleagues at New York University (who can be described as the NYU school and have been in the forefront of those studying the politics of global indicators), offer the following definition in their first collection of case studies. For them, social indicators are :

a named collection of rank-ordered data that purports to represent the past or projected performance of different units. The data are generated through a process that simplifies raw data about a complex social phenomenon. The data, in this simplified and processed form, are capable of being used to compare particular units of analysis (such as countries or institutions or corporations), synchronically or over time, and to evaluate their performance by reference to one or more standards[9].

They go on later to offer an extended description of their essential features. Firstly, they note “the significance of the name of the indicator and the associated assertion of its power to define and represent a phenomenon such as ‘the rule of law’[10]”. Secondly, they point to “the ordinal structure enabling comparison and ranking and exerting pressure for ‘improvement’ as measured by the indicator[11]”. Global social indicators are fundamentally comparative, and “[i]ndicators usually enable comparison of different units, but in a few cases only permit comparison of the same unit at different times[12]”. Thirdly, they tell us that “[s]implification (or reductionism) is central to the appeal (and probably the impact) of indicators[13]”. “They take flawed and incomplete data that may have been collected for other purposes, and merge them together to produce an apparently coherent and complete picture[14].” Finally, again, they underline the fact that they are capable of being used for the purpose of evaluation.

Starting points are never neutral, and can be especially problematic if they prejudge the phenomena they seek to delineate. So it is worth saying a little more about some of the implications of this influential approach. The NYU school mainly discusses indicators as quantitative instruments, even if the Organisation for Economic Co-operation and Development (OECD) includes in its definition of indicators not just quantitative but also qualitative ones, and some of the case studies in their own collections deal with qualitative indicators[15]. For both quantitative and qualitative methodologies it is important to know whether we have got things right. But some have argued that critics of indicators tend to confuse the question of measurement with that of quantification[16]. It is also not always clear how and how far their approach distinguishes global social indicators from the sort of economic and social statistics routinely gathered by national governments, and more recently, also by international organisations. Merry’s definition includes the telling qualification “are capable of being used[17]”. So even basic GNP data, regional health statistics or comparisons of national prison rates would fit their criterion. But while there are undoubtedly legitimacy issues raised in collecting any statistics[18], to include all such processes takes us some way from the world-making comparative exercises that are being described in most of the case studies in the collective volumes on global social indicators.

A continuing challenge is that of pinning down the exact role indicators play in building connections between the “normal” and the “normative”. In the introduction to a more recent collection of case studies co-edited by Merry with other anthropologists, the illustration of a quantitative indicator provided is the measurement of the mid-upper arm circumference so as to see if a child is undernourished (and the converse measurement to see if someone is obese). Admittedly, such measurement is always made in relation to what is “normal” according to standard distribution[19]. But, again, arguing that all statistics have standard distributions is different from the idea of the universalisability of standards such as those concerning the rule of law or corruption. Some of the chapters in these collections of case studies also include discussions of efforts to use indicators for the purpose of explanation or policy interventions as part of experimental trials[20]. This reminds us that, in practice, ranking indicators often overlap with or rely on the other two kinds of indicators we have distinguished.

Ranking indicators may institutionalise social science explanations, linking, for example, legal regimes and economic success, or they may try to engage in the proactive evaluation of policies and projects. Conversely, social science writers sometimes take indicators that were first developed for comparative evaluation as proxies in empirical studies geared to providing explanation of variation, as when Karstedt tries to correlate the evaluative category of “failing states” developed in the USA[21] with international levels of corruption[22]. But what may make sense when indicators are used for explanatory purposes may be less appropriate when indicators are used for policy interventions or evaluative schemes — and vice versa. As will be seen, these different uses of the term can lead to blurring between their scientific, pragmatic and programmatic justifications.

1.2 Internormativity : Law, Indicators/ Benchmarking and Algorithms

Discussions of the legitimacy of indicators and related kinds of regulation may be better understood by placing various kinds of regulatory modality on a continuum that runs from the most to the least counter-factual of interventions. At the more counter-factual end of the continuum we can locate those standards (as with many religious, moral and ethical statements) that represent appeals to people to behave in certain ways without this necessarily being accompanied by formal efforts to enforce them. At the other, more instrumental end, we find efforts to engineer changes in social behaviour without those subject to this being always made aware that they are being manipulated (for example, when the shape of seats in train stations are designed so as to prevent people sleeping on them, or the architecture of public spaces is constructed to make people hurry through them). Other kinds of normative material such as soft law, global social indicators, codes of conduct, professional, technical standards, and hyper- nudges, can then be placed at intermediate places along the continuum.

Figure 1

A Continuum of Normative Regulation

We shall concentrate here on seeing how global social indicators (and global benchmarking) are categorised in relation to law, at one end of the continuum, and to algorithms, at the other. For the NYU school, indicators have much in common with the law. According to Davis, Kingsbury and Merry, “[i]ndicators and law both embody standards, and for this reason both can operate as an expression of values and political commitments[23]”. For them, “the motivations for producing law and indicators may not be that different[24]”. But there are also differences : as compared to law, the production process for indicators is subjected to less limitation and regulation from that of a formal legal standard. Participation in the law making process, committees or treaties, and even regarding “soft law” such as domestic policies, usually requires slow and complex negotiation and adaptation, involving a large number of participants, and can be more influenced by, or targeted to political agendas, than the process of making an indicator[25].

Indicators (like law itself) certainly have instrumental uses. They may incorporate systems of incentives and sanctions as, for example, when the USA seeks to outlaw human trafficking[26] or when the European Union uses indicators to assess when candidate countries are ready to join. But (for those who would distinguish it from indicators[27]) the practice of benchmarking can be placed a little further towards the instrumental side. According to Broome and Quirk,

[b]enchmarking chiefly involves the development of comparative metrics of performance, which typically take the form of highly stylised comparisons which are generated by translating complex phenomena into numerical values via simplification and extrapolation, commensuration, reification, and symbolic judgements. This process of translation takes what might otherwise be highly contentious normative agendas and converts them into formats that gain credibility through rhetorical claims to neutral and technocratic assessment[28].

Benchmarks may be used in policy interventions regarding state failure, global supply chains, disaster management, economic governance, corporate social responsibility, and human development. Likewise they “serve to both ‘neutralise’ and ‘universalise’ a range of overlapping normative values and agendas, including freedom of speech, democracy, human development, environmental protection, poverty alleviation, ‘modern’ statehood, and ‘free’ markets[29]”.

Moving more towards the other end of the continuum, special interest attaches to the relationship between indicators and algorithms[30]. The importance of these latter techniques has been recognized by Davis, Kingsbury and Merry in their second collection of case studies. As they put it,

analysis should be extended beyond indicators and law to other practices that might have similar effects on both knowledge and governance. The leading example is “big data,” the practice of compiling large amounts of data on behavior such as movements, consumption patterns, and social interactions. […] The collection of such data is a way of both shaping knowledge and exercising power. Studies of law and indicators can and should be expanded to include big data[31].

When they are employed for “algorithmic regulation” such programmes collect data impersonally, they intervene in entire populations of users over widely dispersed geographic areas, are dynamic and seek to shape conduct and produce given outcomes[32]. But, rather than seeking to spread universal standards, they seek to adjust to the behaviour of those being monitored. In the public sphere they are legitimated as serving the public interest ; in the private sphere they are presented as being in the interest of those being monitored, though the extent to which this is subordinated to the economic or other agendas of the companies organising the monitoring is often not made clear.

Most writers see indicators as having rather more in common with law than with algorithms. Indicators have been described as a form of “quiet power” (to quote the title of Merry, Davis and Kingsbury’s 2015 volume[33]), but they are certainly more “noisy” than algorithms. Publicity is essential to their role in naming and shaming and this involves making known the standards and evaluations of performance of those being assessed. By contrast, the role of algorithms requires them to be not transparent, and any demand for publicity may also be blocked by claims of property rights. The workings of algorithms have thus been described as producing the “black box society[34]”.

On the other hand, indicators (and benchmarks) are not entirely distinguishable from algorithmic regulation. Algorithms are built into the operations of the indicators such as those used by financial rating agencies. And some authors argue that with indicators too. “[I]ndividuals are interpellated as ‘auditees’ whose behavior is expected to align with the rationality of audit[35]”. Scholars who study benchmarks see them as having similar tendencies. They involve “recursive implementation”, although they distinguish between “performance measurement” and “true benchmarking”, which includes “a plan that is implemented and continuously studied for ongoing improvement[36]”. Thus, for Restrepo-Amariles,

benchmarking significantly differs from the way rules set standards of behaviour and assesses compliance.

On the one hand, since a benchmark is drawn from comparison, its content is relative to the entities compared, realistic because is grounded in actual practice and dynamic because it changes overtime depending on real performance. In contrast to an intrinsic concept of outcome quality, benchmarking defines the concept of quality in a pragmatic manner, for instance, by focusing on the satisfaction of users of legal systems. On the other hand, benchmarking also introduces an alternative way of assessing social facts against norms, not in the binary terms legal/illegal – as legal reasoning does with rules, but in terms of degree, i.e. it determines where a given conduct is located in a performance scale[37].

As valuable as it may be to see how scholars place indicators along a normative continuum with other forms of regulation, it can therefore be even more important to examine the way different forms of normativity overlap — or may potentially clash. As the organisers of the colloquium said in their call for papers, their goal was to learn more about the way “standardization and management mechanisms are intended to compete, complement, and sometimes replace conventional legal mechanisms, […] the mechanisms upon which they rely, the practices they generate or upon which they are based, and the results they produce[38].”

Most obviously, law is often involved in requiring the establishment of an indicator, or the need to meet its prescriptions. According to Davis, Kingsbury and Merry, “[t]he system of letter-graded credit ratings for sovereign bonds is so embedded in legal rules and market practices (including requirements that some entities invest only in bonds rated as ‘investment grade’) that network effects result, making major changes to the system more costly and less likely[39].” Human rights indicators too are obviously deeply embedded in legal frameworks and are meant to monitor and promote compliance with human rights commitments.

But it is the differences between law and indicators that explains why indicators may be used as a supplement to it. In their careful overview of the intersection of law and indicators, Davis, Kingsbury and Merry tell us that :

[s]ome of the pathways through which indicators can influence expectations, social categories, status relations, behavior, and even identities are similar to those applicable to international legal instruments. These include naming, shaming, praising/blaming, emulation, acculturation, information changing beliefs or strategic calculations, behavioral or identity adaptations of the indicated, empowerment or redefinition of the indicators promulgator, altering external power structures, altering distributions of power within an entity, and building or intensifying networks[40].

At the same time, they add, “[i]ndicators are distinctive, however, in that (typically) the indicator simultaneously articulates the standard and applies it. The flow of costs and benefits as incentives to perform better on an indicator may be greater or less than the flows attached to legally binding instruments[41]”. For them,

indicators can be used to measure, evaluate, and seek to influence behavior of private entities. […] Second, indicators can be used to measure and report the nature and levels of the private conduct that an international legal regime seeks to curb or promote. Third, indicators can be used to assess the degree to which a state is in fact successfully controlling private actors, or to monitor the level of effort a state is making to regulate private conduct. […] Fourth, indicators can be used to help define and quantify a state’s duties in relation to private persons or conduct in markets[42].

Can different kinds of normativity also conflict ? This is of special significance because many scholars think that indicators need to gain legitimacy by being held accountable through law yet, at the same time, indicators are themselves used as part of evaluations of legal regimes, seeking to assess the quality of governance through law, so as to measure practices or perceptions of good governance, rule of law, corruption, regulatory quality, and related matters. For Frydman, this situation should be seen as one of legal or normative pluralism[43]. Davis, Kingsbury and Merry too argue that “indicators may serve either to promote or to undercut the objectives of existing law[44]”. Using the example of domestic and international tribunals that use indicators to assess compliance with social or economic rights, they suggest that “[t]he potential for complementarity is especially clear when indicators are incorporated by reference into a legal rule, or are used to extend, elaborate on the scope of, or give added specificity to a legal rule[45]”. By contrast, they tell us that “indicators that provide a competing standard and bypass existing law can create a more pluralistic legal environment and undercut the standards embodied in the law. […] Over time, these kinds of tensions between law and indicators may create pressure for law reform[46]”.

But not everyone agrees. Bogdandy and Goldmann, for example, in their analysis of what they describe as a “clutter of actors, competencies and programs” that “recalls the debate about the fragmentation of international law[47]”, claim that :

[i]n the case of governance by information, normative collisions are not possible for lack of legal norms. […] [W]hat might occur are “collisions of information” ; i.e. situations in which information produced within one governance mechanism might compromise the functioning of another governance mechanism[48].

Collision, for them, “does not refer to the utterance of contradictory information, but to contradictions in the treatment of information[49]”.

As we shall see, those most critical of regulatory modalities that compete with law worry that they are not sufficiently counter factual and respectful of the people whom they regulate. Although these are mostly legal scholars, the distinction between counterfactual and factual also has heuristic value for at least some social theorists. Luhmann, for example, claims that law counter factually stabilises normative expectations and in a sense is about “not learning” from violation[50]. On the other hand, any sharp contrast between factual and counterfactual is problematised by many other social theorists, such as Foucault, Bourdieu and Latour[51]. Both law and indicators shape knowledge, and this essay will confirm Foucault’s insight that thinking about law and sovereignty under current conditions often fails to recognise how new ways of governing through power-knowledge have “cut off the head of the king[52]”. Recent commentators on indicators have noted that what may seem merely technical standards can indeed often be shown to be the outcome of hard fought, value-laden battles[53]. Social indicators (and benchmarks) can be a way of passing off standards as facts — or vice versa. As Restrepo-Amariles puts it, “[b]enchmarking exemplifies as much as it contributes to collapse the sociological and normative dimensions of legitimacy of global legal indicators[54]”.

1.3 Of Legitimacy and Legitimation

Do indicators need to be legitimate ? What are the obligations binding on those who make and use them ? Who, if anyone, is under an obligation to comply with the standards built into them ? Are the answers to these questions the same for all indicators ? It is clear that indicators are often used with the intention of legitimising decision-making, even if few decisions rely entirely or mechanically on them[55]. According to Davis, Kingsbury and Merry :

[i]n the practice of global governance, many decisions by governing entities are in some way influenced by indicators […]. In the most straightforward case, an indicator promulgated by an extra-national entity is then used by that entity in generating or allocating resources or in influencing behavior. This is, for example, what the World Bank does in promulgating “good governance” indicators that are used by the World Bank itself in deciding how to allocate aid[56].

Much the same applies to benchmarks[57].

Given this, some writers claim that indicators are legitimate by definition. Davies, for example, characterises an indicator as an instrument that “successfully combines scientific objectivity with normative authority of some sort, winning agreement from both experts and non- expert publics[58]”. As Strathern also noted, audits have come to embody a new form of ethics : they are where “the financial and the moral meet[59]”. But we should not be too quick to accept a definitional stop that would make otiose the question to which this whole paper is addressed.

Frydman, playing devil’s advocate, poses the question squarely, suggesting that we might just want to discuss whether indicators are useful or accurate rather than whether they are legitimate[60]. As he puts it :

indicators do not resemble the classic ways in which power or law are expressed and, second, indicators do not appear as compulsory commands the violation of which would be subject to a sanction. Indicators convey facts, but do not prescribe rules. They do not give orders or take decisions. At most those who exercise power or work within administrations may rely on information contained in indicators to make decisions. Therefore, according to this point of view, the validity of indicators should not be assessed against criteria of political and legal legitimacy, but instead against the relevance and validity of the scientific method of which they are the result[61].

Nonetheless, for him, as for many other commentators, indicators can and must be put to the test of legitimacy. This is because, “social indicators not only convey information, but are genuine tools of global governance and […], for this reason, their legitimacy depends not only on their accuracy, but also on their accountability[62]”. Indeed, as Davis, Kingsbury and Merry predicted :

[a]s the awareness or the significance of indicators as standards rises, indicator design and production is likely to become increasingly subject to demands made of other standard-setting processes, including demands for transparency, participation, reason-giving, and review. […] This has the potential to intensify demands for “due process”, especially within inter-governmental bodies, as each specific ranked entity has a direct focused interest, going beyond the general interest in good standards, which it may regard as conferring “standing” to raise a challenge[63].

For Frydman too, “indicators go beyond the communication of information. They are part of a governance system that pressures behaviours of persons or groups that are subject to it[64]”. Certainly, the kind of information retailed by global social indicators can have and is often intended to have social consequences. Indicators influence can thus be compared to other social pressures and informal social control more generally. As Davis, Kingsbury and Merry put it, “[t]he majority of prominent indicators appear to operate in global governance […] by influencing professional, public, and political opinion to craft new approaches or take different policy orientations[65]”. But what follows from this ? We do not ask that all social forces and influences that shape behaviour pass a test of legitimacy. What if indicators are no more than a form of global gossip ? In the USA, financial rating agencies have successfully defended their mistaken assessments as merely “opinions”.

Much depends on what we mean by legitimacy — and this is something that varies according to the disciplinary affiliations of those scholars writing about indicators. Different understandings may be found in the varied branches of legal scholarship, in international relations, economics, or anthropology. Although much of the running in the study of indicators has come from the collaboration of scholars working at NYU, it is important to note that those who espouse a more legal approach (known as GAL i.e. Global Administrative Law) start from very different assumptions and propose different solutions from their more anthropological colleagues. The concept can also be used in more narrow or broader senses, as in speaking of an “illegitimate” deduction — and this too can be relevant to understanding how far global social indicators are found credible. But the term has a more relatively circumscribed sense in legal and political science discussions where most authors agree that it points to among other things, staying within boundaries, procedure, and effectiveness. Some scholars also distinguish input, throughput, and output legitimation. “Input legitimacy refers to aspects such as the expertise of the law-making authority and sufficient forms of accountability ; output legitimacy may be assessed by theories of justice as well as criteria of efficiency ; and throughput legitimacy refers to fair and transparent procedures[66]”.

A question of special relevance for us is how far legitimacy should be considered a legal requirement and how far just a matter of fact-the outcome of successful legitimation. For legal scholars the first conception prevails. Thus, for Cassese and Casini, indicators need to be made accountable when they form part of the exercise of public authority[67]. On the other hand, at the level of lived experience, the line between binding orders and voluntary acceptance can be hard to draw given that oligopolistic private companies often oblige those who use their essential services to waive their rights. And private orders can be experienced as illegitimate even if they are under no requirement to be legitimate. Arguably, legitimacy is even more important for pure private forms of governance, since — due to the lack of state enforcement — the success of such a private scheme crucially depends on its perception as legitimate[68].

In many empirical social science discussions of indicators, on the other hand, normative questions about their legitimacy are collapsed into descriptive ones concerned with whether they are in practice felt to be legitimate[69]. Beetham suggests that the “legitimation of power” requires that (i) the power confirms to established rules, (ii) that these rules need to be based on shared beliefs by both parties and (iii) that the subordinate party has consented to this power relationship[70]. But, depending of course on our theory of legitimacy, a given indicator could be legitimate even if it does not have popular support and is out of touch with what people feel to be legitimate. On the other hand, an indicator could be illegitimate even if most of those people affected by it felt it to be legitimate. Put differently, must indicators allow themselves to be challenged ? Should we say that a legitimate indicator is one that could overcome hypothetical challenge ? Or is it necessary that they actually invite contestation ?

Some commentators argue that an indicator’s validity is secured by having a range of competing indicators and seeing which gains adherence[71]. A well-functioning market for indicators is more likely to produce legitimacy than “monopolistic indicators” since their drafters face pressure to fix possible technical flaws. But those relying on indicators are often unlikely to be able to bring such pressure. On the basis of his case study, for example, Dutta claims “indicators users and targets will rarely demand either accuracy or accountability from generators, even when important decisions are based on indicators[72]”. Conversely, just because many people question the legitimacy of an indicator does not necessarily lead to its collapse. It is difficult otherwise to explain the survival of rating agencies despite so many people seeing their misleading indicators as one of the main factors in the 2008 financial crash[73]. In any case, a plurality of indicators does not of itself guarantee that any of them is right or, if accuracy or outcomes is what is at stake, that they have actually got things right.

Distinguishing between legitimacy and legitimation can be especially important if we want to examine whether and when the one leads to the other. Sociological (and other empirical) perspectives pose the question of the difference actually made by the belief in legitimacy. Max Weber argued that “social action […] may be guided by the belief in the existence of a legitimate order[74]”. Having laws which the public believe to be legitimate is important since the state may not be able or willing to enforce those by force, and other reasons for compliance (such as habit and expediency) may not always be sufficient. Some reject the possibility of finding empirical proof that legitimation is the basis for action by makers, users or targets of legal orders given the many other potential reasons for acting[75]. But psychological research suggests that “perceptions of system legitimacy do shape everyday compliance with the law, which is a conclusion that is also supported by more recent studies […]. Furthermore, perceived legitimacy seems to have more influence on compliance than do subjective assessments of the likely risk of punishment. When people perceive the system as legitimate, they feel an intrinsic moral obligation to comply with its demands[76]”.

Admittedly, most of these studies have more to do with the behaviour of actors and institutions such as police and courts rather than with the making and implementation of standards. But many of those who have provided case studies of indicators also focus on the degree to which people comply as a measure of something being legitimate. Certainly that is so when thinking about the usefulness of indicators. Restrepo-Amariles tells us that “legitimacy is defined as the quality of an indicator by means of which organisations and individuals subject to measurement — namely the addressees — feel compelled to acknowledge its results and to use them as a basis for taking action[77]”. He admits, however, that compliance may often be more instrumental than heartfelt, pointing out that, “organisations feel compelled to get in line with the precepts underlying indicators because they are believed to convey certain societal expectations[78]”.

Legitimacy, for most social scientists therefore is the social achievement of becoming accepted. And the various collections of case-studies provide considerable detail on how this is done.

According to Davis, Kingsbury and Merry what emerges as :

[c]entral to their power is their promise of objectivity and impartiality. Their credibility depends on being able to deliver on these promises. Some are seen as more vulnerable to politics and pressure, while others seem more based on “hard data” and more credible in the national sphere, where there is considerable competition among indicators promoted by very different actors and organizations[79].

For an indicator to be accepted it must first establish itself against competition from other indicators and get used by those for whom it is intended. Thus the problem of legitimacy tends to be most salient at the early stage of trying to get an indicator accepted[80]. The level of use by those to whom the indicator is made available then becomes the key marker of legitimacy. As Siems and Nelken explain, indicators that are reused, endorsed, or in any case become successfully entrenched in the institutional practices of organisations delivering incentives are not only most likely to exhibit higher degrees of legitimacy, but also to impose their values and views[81]. For example, the indicators used by the World Bank benefit from their institutional embeddedness and thus have an institutional advantage since they are directly used by a big international actor, whereas the indicators created by non-governmental organizations depend more on their use by other actors, international as well as domestic[82].

2 Claiming Legitimacy : Authority, Accountability and Accuracy

We have seen so far that the answer to what makes an indicator count as legitimate is not necessarily the same as whether or not this actually translates into felt legitimacy. But the claim to legitimacy still has to be formulated in ideal terms which make reference to why obedience should be given. Hence even social scientific discussions give considerable attention to the claims indicators make for their authority, accountability and accuracy. Our analysis will be focused, in addition, on showing how far research reveals that the growing resort to indicators is itself helping to reconfigure (and re con-figure) these criteria of legitimacy.

2.1 Authority and Changing Forms of Governance

Global social indicators, either explicitly or implicitly, claim to have authority. As Guimezanes argues, “the making of indicators thus constitutes a political act. Their use also constitutes an act of authority. They can be used to assess the effectiveness of a framework and take corrective action as part of an experimental scheme[83]”. But what is the nature of this authority ? How much does it rely on conventional ways of legitimating power, how far is it innovative ? Figure 2 provides a snapshot summary of the debate over this question.

Figure 2

The Authority of Global Social Indicators

In Weberian terms, indicators may sometimes be able to satisfy the requirements of legal rational authority. Even if global social indicators cannot and usually do not claim authority on the basis of a pedigree of norms derived from the executive, parliament or courts, there may still be some reliance on law or law like authority. Infantino tells us that the very binding nature of the establishment, preparation or use of an indicator comes in many shades of grey. It may for instance stem from traditional “hard” legal obligations, it may derive from institutional inter-organisational arrangements. In addition, authority may be expressed by contractual or quasi contractual obligation (as it is when funding is distributed on the condition that its beneficiary produces human rights indicators on issues determined by the fund provider) … or it may take the form of invitations and “soft” commands, whose binding force derives from good practices and institutionalised patterns of behaviour[84].

In many more cases, indicators rely on the charismatic authority of experts. Broome and Quirk tell us that “[t]he power of the symbolic judgements rating agencies issue comes primarily from their role as ‘reputational intermediaries’. This is based on their public image as independent, authoritative actors that are capable of making accurate expert assessments of creditworthiness, despite this image being subjected to stringent criticisms in recent years[85]”. Linked to this, indicators often make a claim to substantive rationality, asking to be judged by their effectiveness on producing given outcomes. Less obviously, they may also benefit from “traditional authority” insofar as they enact what Power has called a “ritual” of verification where few know what actually lies behind their claims or are in any position to verify them[86]. Strathern, for example, suggests that “an aura has come to surround numbers and, despite the caveats of professional auditors, it is those unfamiliar with financial auditing who tend to sanctify them[87]”.

As compared to nation state based sovereignty, indicators are more a medium of governmentality. The growth of regulation via indicators is linked to changes in the role and authority of the nation state that have been shaped on the one hand by globalisation and the other by the spread of neo-liberal ideology.

Indicators help fill the vacuum of authority at the global level, playing an especially important role in helping to overcome a lack of information about the risks posed by what can be called transnational social problems. Davis, Kingsbury and Merry tell us that human rights indicators act as a “technology of global governance”, that is to say, they secretly embody and implement policy visions about existing problems and about how these problems have to be overcome. These technologies of governance can be expected to affect : where, by whom, and in relation to whom governance takes place (the “topology of governance”) ; the processes through which standards are set ; the processes through which decisions are made about the application of standards to particular cases ; and the means and the dynamics of contesting and regulating exercises of power[88].

The use of social indicators takes a different role in historically different kinds of state- such as the liberal, welfare and neo-liberal states, even if it is most closely associated with the neo-liberal state and its methods of “governing at a distance[89]”. Davis, Kingsbury and Merry tell us that “[t]he burgeoning production and use of indicators in global governance has the potential to alter the forms, the exercise, and perhaps even the distributions of power in certain spheres of global governance[90]”. They must be seen as part of a sociotechnical assemblage “allowing certain actors to exercise influence over the conduct of large numbers of geographically dispersed actors, that are readily adapted to forms of governance outside or reaching across distances beyond the state[91]”. For Restrepo-Amariles, too, “indicators are better understood as part of a system of management control and distributed governance[92]”.

Benchmarks have also been linked to the rise of what has been called “governing at a distance”. Broome and Quirk, in a paper with that title, tell us that “benchmarks primarily operate by quantifying and projecting normative criteria regarding the parameters of appropriate conduct and performance. The literature on governmentality is especially useful for the insight that benchmarking functions to make diverse forms of behaviour legible and amenable to intervention[93]”. As they note :

global benchmarks usually operate by orienting how specific actors : (1) conceptualise their options, obligations, and opportunities ; and (2) seek to legitimate and justify their performance and perceived relative standing. It is within this context that benchmarking practices can be regarded as an exercise in governance at a distance, which combines indirect power, expert authority, and transnational governmentality. This also means that the political effects of benchmarking tend to be cumulative and subtle, rather than overt and immediate[94].

For Von Bogdandy and Goldmann, “[a]s states are less and less able to meet the needs of a globalized world, and as their citizens and economies must compete in worldwide markets, the legitimacy of the state-based structure of the international legal order might fade if states do not continue to perform at a high level[95]”. According to Restrepo-Amariles, indicators are needed because of the limits of international authority. As he explains, while international institutions often lack mechanisms for enforcing their orders, indicators provide a tool for them to communicate lack of compliance with a particular norm, even if the consequences remain unclear[96]. But some scepticism may be warranted about assuming that global indicators reflect the decline of the nation state. For Shore and Wright, the increasing use of indicators at the transnational level is not zero sum, “[w]hereas the calculative practices of measuring, ranking and auditing were once largely confined to the nation state, today they increasingly operate beyond — and below — the level of the state[97]”.

If indicators can involve the exercise of authority at different levels and in different roles we need to ask who[98] are “the new rulers[99]” ? Who is doing the categorization ? What are their convictions and interests ? Who is consulted ? Who sits round the planning table ? Who has the power to say what counts ? For whom exactly must indicators be legitimate ? A large variety of actors may be involved in making and implementing indicators. Some are promulgated by nation states such as the USA, as with the efforts to sanction human trafficking through the human trafficking act which are more influential than those of the United Nations. Others may be created by regional groupings such as the European Union. Non-Governmental and Inter-Governmental organisations are especially prominent. These organisations can have different degrees of independence from state bodies (contrast Freedom House with its strong financial dependence on the USA government[100] with Transparency International, Reporters without Borders, Walk Free Foundation, World Justice Project or Human Rights Watch[101]). Academics also play an important role in many indicators. We also should not forget the influence of some media organisations, as with the pioneering university ranking tables produced for USA News or for the Times Newspaper.

Economic motivations help to multiply the use of indicators. For Shore and Wright, “the shift from ‘government to governance’[102]” means that “[a]n entire global industry of measuring, ranking and auditing organisations and individuals has arisen based around ideas of enhancing ‘quality’, ‘efficiency’ and ‘transparency’, and as technologies for producing calculative, responsibilised, self-managed subjects[103]”. They go on to comment on :

the extraordinary breadth, scope and proliferation of these new industries of calculation and enumeration and their increasing involvement in new forms of governance, particularly of a managerial kind. If rankings and league tables have become a “populist project” […], they have also become a business and major source of private revenue[104].

But not all of those involved with indicators or benchmarks share the same interests or values. As Seabrooke and Wigan remark, “NGOs and expert-activists operate in a highly contested field of power where there are frictions between themselves and other types of organisations and groups, such as firms, states, and international organisations[105]”. Obviously, some organizations carry more clout than others as they compete for credibility and acceptance. Public bodies can have an advantage as far as their structures reflect the input legitimacy of their members. “[P]rivate bodies may also benefit from their high reputation (e.g. the ISO). It can also be the case that the independence and neutrality of a private body foster its credibility and therefore the input and output legitimacy of the indicator, (e.g. this may be the case for the Press Freedom Index by Reporters without Borders)[106]”.

In the competition for authority, resort to indicators can lead to a shift in authority from some professionals to others. For Davis, Kingsbury and Merry,

the transformation of particularistic knowledge into numerical representations that are readily comparable strips meaning and context from the phenomenon. In this numerical form, such knowledge carries a distinctive authority that shifts configurations and uses of power and of counter-power. This transformation reflects, but also contributes to, changes in decision-making structures and processes[107].

With reference to human rights indicators they suggest that they “may entail a silent but effective transfer of power from political elites to (officially) de-politicised communities of bureaucrats[108]”.

Depending on what we want to investigate, we may need to distinguish, amongst others, the roles of sponsors, makers, users — and targets of indicators. The addressees of indicators may include countries, donors, investors, citizens, workers in Non-Governmental Organisations, lobbies, corporations, academics, investors. It is important to note that what the users do with indicators is often not under the control of those who sponsor and make them (for example, investment agencies following the lead of rating agencies, or parents choosing universities). Davis, Kingsbury and Merry argue that all those who help shape indicators exercise authority (though it is hard to see how this can be constrained). As they say, “[r]ecognizing indicators as a technology of global governance implies that actors who promulgate indicators ought to be counted among the governors, even if they otherwise would not be recognized as wielders of power in global governance, or would be only to a lesser extent[109]”.

Indicators, as we have noted, call upon science as a source of authority to supplement their weak legitimation by legal or political pedigree. Davis, Kingsbury and Merry tell that :

[w]hereas political efforts to formulate norms and standards, for example in multilateral inter-governmental negotiations conducted by diplomats, tend to involve processes such as voting or interest-group bargaining or the exercise of material power, the processes in specialist agencies and expert meetings where the standards embedded in indicators are produced, accepted, and supported tend to involve derivation of power from scientific knowledge[110].

Like other forms of scientific knowledge production, an indicator builds on existing concepts, techniques, and categories of understanding that are taken for granted as correct, and on networks of experts, human rights workers, accountants, statisticians, as well as computer experts — in the case of algorithms[111].

Modernity is inseparable from the creation of “the modern ‘fact’[112]”. What has come to be described as “governance by information” or “governing by numbers” involves reducing complex processes to simple numerical indicators and rankings for purposes of management and control. Numbers exert a curiously seductive power ; indicators are assumed to be objective and unambiguous because of their association with science and the “invariable rules of mathematics[113]”. Indicators and benchmarks move us from the fact of authority to the authority of the fact — which purports to be self-evident. For the critics, such claims to authority should be treated with more suspicion. For Goodwin, these methods prioritize the numerical over the subjective, the universal over the particular, the scientific over the interpretative. Yet, the methods by which indicators are conceptualized and calculated are not politically neutral and are deeply rooted in the values of the enlightenment and modern capitalism. That numerical measures are naturalized and largely unquestioned in our knowledge system does not mean that they are devoid of ideological and cultural bias[114].

One promising approach to grasping this form of authority is Krisch’s ongoing project on “liquid authority[115]”. The concept of liquidity aims to characterise the dimensions of “informality, ideationality, dynamism, and multiplicity[116]”, in particular referring to forms of governance “without a clear centre, dispersed over a multitude of actors and institutions (public and private) without a formal ascription of authority[117]”. Interestingly, some of the multinational organizations who wield most power deny even the need to seek legitimacy. Major multinationals such as Google and Facebook have almost a monopoly of on line advertising market between them, and the order in which searches appear on search engines have a range of naming and shaming effects. But they present these as commercial decisions (or non-decisions) for which they accept no political responsibility.

The ultimate source of the authority of indicators is the spread of so called “audit culture[118]”. This involves, according to Shore and Wright,

a relentless pursuit of economic efficiency ; deregulation, outsourcing and privatization ; marketization and the privileging of competition over cooperation ; increasing separation between an empowered managerial elite and a deprofessionalized workforce ; the objectification of human labour – combined with increasing emphasis on calculative practices aimed at promoting individualization and responsibilization[119].

As they argue,

[w]hat is new about audit culture is the extent to which these calculative practices of measurement and ranking have become institutionalised, extended and above all, financialised. This is particularly evident in the way that the language of financial accounting is increasingly used in the governance of organisations and their employees. Keywords from audit and accountancy (“assets”, “income”, “liability”, “opportunity costs”, “products”, “output”, “overheads”, “revenue”, “return on investment”, “transparency”, “value for money”) have been combined to form new semantic clusters that have become the bedrock of a new ideology of calculating and monetarised self-governance[120].

2.2 Accountability and The Limits of Contestability

The rise of global social indicators is inextricably linked to the demand for more accountability ; they have even been described as “an information based accountability tool[121]”. As Riegner shows us, “[i]ndicator-based performance measurement, disclosure regimes and transparency requirements[122]” play a central role in making targets of governance accountable both in the sense of supplying information about their behaviour which allows it to be measured and assessed, and in the sense of those being assessed being held responsible for their behavior. Some “independent” tool to measure outcomes is required by sponsors, donors and those running programmes, because of the danger of misrepresentation by targets, and collusion by those working with them. At the same time, by providing evidence of self-regulation, they also help create space for manoeuvre for non-governmental organizations and inter-governmental organizations.

The use of indicators can lead — and is certainly intended to lead — to a reduction in discretion of those being monitored. Crucially, however, it is not so clear whether this applies to those doing the monitoring. Do they also ensure the accountability of those who make and use indicators ? More, given the transnational context, to whom should the makers of indicators be accountable ? For Goodwin, for example, “[p]ower should be accountable but to individuals and communities, not to the World Bank or agencies of donor governments[123]”. Implicit in this criticism is the assumption that this is currently often not the case. The state of the debate is summarised in Figure 3.

Figure 3

Accountability and Global Social Indicators

Indicators often make promises of bottom up accountability. “Basing governance decisions solely on publicly disseminated indicators excludes the possibility of basing them on subjective considerations of, or private data known only to, the particular decision-maker[124]”. Even those otherwise critical of some aspects of benchmarks acknowledge that they help produce more “publicness[125]”. On the other hand, as Davis, Kingsbury and Merry argued, “[t]his impartiality is limited […]. The reasons for simplifying raw data in one way instead of another, or choosing to rely upon one indicator rather than another, may be highly subjective[126].”

One problem arises from the potential conflicts of interest that affect many of those who make and use indicators. The USA sometimes allows political or economic motives to influence how it classifies other countries when ranking them for example as “fragile states” or for their performance in fighting human trafficking. For some of those conducting university rankings, a useful second income stream is generated by universities commissioning detailed analyses of their performance and consultancy services on how to improve it. Most famously, allegations have been made against some accountancy companies in making financial indicators, who were accused of being involved in the Enron scandal in the USA or behind a high proportion of known tax avoidance schemes. The worry that financial rating companies may be influenced by those that pay them is not altogether assuaged by their defence that it is not in their interest to the risk their reputation by making wrong predictions.

Whether indicators and benchmarks actually promote the accountability of those who make and use them will also vary by the kind of indicator and the circumstances of their use. In his case study of two mechanisms for assessing country performance, one based on quantitative data, the Millennium Challenge Corporation (MCC), and one on qualitative data, the EU accession process, Dutta showed that even though qualitative processes appeared to be more participatory than quantitative ones, in this case, given the sharp imbalance of power between the EU and states that wish to join, the qualitative process lacks significant opportunities for flexibility and participation[127]. Legibility will usually be a pipe dream without effective participation of all those affected. What is needed is access to the debates behind the scenes where decisions are made how to define and measure what is to be evaluated. The same applies even more to information about outcomes. Case studies of the measurement of corporate social responsibility, for example, show that some of the leading efforts to inject accountability here are not followed up in practice and it is left to companies themselves to mark their own achievements[128].

For many of those concerned about the deficit in accountability in the making of indicators the answer lies in ratcheting up the extent of legal regulation. Riegner argues that :

a careful analysis reveals the many legal forms, rules and principles that already shape global information governance in many explicit or implicit ways. For law does not simply “regulate” pre-defined informational “objects”, it co-constitutes these very objects. Law represents an underlying deep structure that shapes the surface of information that becomes visible in individual decisions, reports, statistics and indicators[129].

According to him, the challenge remaining is to develop more systematically what he calls “international information law”, something which applies especially to the early stage of making indicators[130].

Other steps can also be taken. Von Bogdandy and Goldmann, writing about the PISA educational rankings, propose what they call a “standard instrument” to add to the legitimacy, efficiency and effectiveness of the public authority exercised through them[131]. Davis, Kingsbury and Merry argue that “producers of indicators could be subject to scrutiny (although not necessarily legal obligations) with reference to human rights standards, domestic constitutional norms, and principles of global administrative law. Others may be regulated in the same ways as private actors such as multinational corporations or networks of firms linked by transnational supply chains[132]”. For their part, the proponents of Global Administrative Law argue that indicators must offer all the guarantees that the public expects from legal authorities, both in terms of their composition and their operation, notably those of transparency, impartiality and accountability. Second, it must be ascertained that the impacted persons benefit from a procedure complying with due process of law[133].

Law plays an increasingly comprehensive role in politically sensitive areas, for example, in the requirements placed on those who issue financial indicators[134]. On the other hand, some commentators warn that over-regulation could undermine the beneficial effects of indicators. Thus Cassese and Casini set out a taxonomy of different types, uses, and conditions of use of indicators in order to distinguish cases in which indicators as accountability-enhancers require protection from regulation from situations in which some public regulation may be required[135].

If the creative use of law could do a lot to improve the procedural regulatory of indicators it can do little about checking the purposes to which indicators are put. For most commentators the route to substantive accountability involves instead trying to re-politicise what Davis, Kingsbury and Merry call the governance “effects” of indicators, so as to find ways to increase their contestability. As Davis, Kingsbury and Merry explain :

[contestation can involve] long-established forms such as lobbying and litigation, as well as distinctive forms that are especially suited to changing or resisting governance through indicators, such as refusal to participate in data collection, challenges to scientific validity, or creation of alternative indicators. [It] can take the form of debates about the data used and not used in indicators, the weightings criteria for the indicators, or about the embedded social and political theory of the indicator. Contestation strategies can include the creation of new indicators, and resistance to or discrediting of existing indicators and their producers or users. This may in turn prompt modifications to the indicator, or counter-strategies by producers and users[136].

Building in contestation could be especially important for indicators where differences in cultural starting points are most relevant, as in the case of those dealing with human rights[137]. Rosga and Satterthwaite conclude an early, but penetrating, analysis of the strengths and weaknesses of indicators as social tools by affirming that “[r]ather than trusting in numbers alone, those using human rights compliance indicators should embrace the opportunities presented by this new project, finding ways to utilize human rights indicators as a tool of global governance that allow the governed to form strategic political alliances with global bodies in the task of holding their governors to account[138]”. But, for many other kinds of indicators, it can be more difficult for stakeholders to challenge the power of experts and their methodologies. And sometimes, as in the case of indicators about European Union accession or financial ratings, contestability is ruled out because those being assessed are obliged to give prior consent to the process. Alternatively there may be no need for contestation where indicators are controlled by those being regulated.

Other questions can be raised about the meaning and point of contestation. Certainly it is more common for indicators to be gamed than for them to be changed through contestation[139]. It can also be difficult to draw a line between contestation, gaming, and resistance. Perhaps gaming could be seen as a deliberate attempt to get round the indicator so as to pretend to comply with it rather than launch a challenge ? Often what matters is less what the indicator asks for but who is doing the asking. As some contributions to collected works by Davis, Kingsbury and Merry tell us, relatively powerless countries when subjected to measurement and management by more powerful ones also find ways of limiting their effects[140]. As Davis, Kingsbury and Merry note :

[r]ule of law, political freedom, corruption, and good governance are central public issues in struggles for reform in many developing countries. Global indicators produced by extranational institutions are used in local political struggles by reformers […] A major complication for such reformers is that invoking a global indicator tends to cast them as aligned with an extranational institution that may be viewed with skepticism by local political actors, and that may be pursuing, including through the indicator, a different set of political interests[141].

How far should we go so as to make contestation possible ? Siems and Nelken suggest that “having a wide scope can be beneficial to the procedural and substantive legitimacy of an indicator as far as it manages to balance all of the relevant interests. However, there is also a trade-off since an indicator that tries to please everyone may become too diffuse to be of use for any of the beneficiaries[142]”. Arguably, too, if an indicator is to fulfil its functions, especially if these are conceived in law — like terms, indicators cannot be considered merely an opportunity for debate. To provide certainty and a basis of reliance, some level of certainty and closure is essential[143]. It is also important to note that contesting an indicator plays a different role in politics and science. Politically, allowing contestation is a way of giving those affected a say, whereas for science it is a way of getting things right. When people question their place in a ranking they indirectly endorse it. Even when they disagree with a given indicator (for example the definition of what should count as an environment conducive to better business) this does not necessarily put into doubt the overall legitimacy of using indicators for such projects. All the more, the need to be contested is a key aspect of scientific progress ; it is only challenges to existing findings that can make it possible for them to be revised. Evidence of contestation does not necessarily undermine the indicator as such, it only lead to calls for better indicators[144].

Legal regulation, political challenge and rethinking what counts as consent do not exhaust the potential ways of gaining more accountability. As Davis, Kingsbury and Merry tell us, “for example, public bodies might support or subsidize the production of competing indicators, or certain organizations already exercising other substantial powers as governors might be encouraged to refrain from promulgating indicators[145]”, there could also be efforts “to educate users of indicators, and the members of the public who confer authority upon them, about both the costs and benefits associated with using indicators[146]”.

Many critics nonetheless remain suspicious that reliance on indicators will change the very meaning of accountability through the co-optation of professional values and their conversion into managerial indicators and instruments. As Shore and Wright put it, “[a]fter all, accountability, responsibility, quality, self-management and transparency are all values that professionals would normally espouse[147]”. The Janus headed character of indicators, both legitimating and needing legitimation, also create problems for its role in providing accountability. Too often accountability in practice is one-sided, one-directional and top-down, as those in positions of power seek to make legible the performance of those below them without being accountable to those they govern[148]. In the case of law and development indicators, for example, it can lead to “blame games[149]”, putting responsibility on underdeveloped countries for their underdevelopment. In neo-liberal regimes, where the modalities of normativity tend to focus exclusively on choices and rights of individuals, a stress on accountability also risks being a distraction from mobilisation in favour of a more equitable global and social redistribution of wealth.

More generally, while they may sometimes render individuals and organisations more “legible” to external experts, we are warned that there is a coercive dimension to that accountability : organisations must represent themselves in terms of the narrow, predetermined script of expert assessors, in what Strathern calls the “tyranny of transparence[150]”. Moreover, those being assessed often find themselves trapped into providing certain types of information in order to receive aid and support without the means to gather it. On the other hand, as Ruppert shows, efforts to get those behind indicators to reveal more may end up being misleading if it hides the selectivity that remains. Transparency can even become little more than a spectacle or performance, where it is well understood that consumers of such transparency will be unable to make use of it[151].

2.3 Accuracy and Effectiveness : Truth as a Currency

A final set of issues raised by commentators on indicators have to do less with accountability in the process of making indicators and more with what it takes to produce “legitimacy through outcomes”. How far do indicators accurately represent “the real” ? How well do they compare or make commensurable different realities ? Do they actually succeed in indicating what has or has not changed as the result of a given intervention ? The promise of reliable indicators is that they can replace supposedly subjective judgements based on values or politics with rational decision making on the basis of statistical information. They thus turn the exercise of judgement into one of technical measurement and, as noted, this tends to reduce the question of legitimacy to that of legitimation. Are indicators able to claim this kind of legitimacy or do they (only) re-define it performatively ? The competing positions in the debate are summarised in Figure 4.

Figure 4

The Accuracy and Effectiveness of Global Social Indicators

Accuracy certainly matters where indicators are concerned. Some of the most far reaching recent business scandals have turned on their manipulation, as with collusion over Inter Bank Lending Rates (Libor) or falsification of vehicle pollution emissions (Dieselgate). But even where those reporting indicators are in good faith they may not always get it right. The process of producing quantifiable indicators is one subject to numerous pitfalls. “Simplification (or reductionism) is central to the appeal (and probably the impact) of indicators. They are often numerical representations of complex phenomena intended to render these simpler and more comparable with other complex phenomena which have also been represented numerically[152]”.

Data may suffer from simple arithmetic flaws or problems of measurement and aggregation. Critics of the Doing Business Indicators, for example, calculated that when these were aggregated into rankings the ranking of the top four countries as shown in the 2008 Human Development Index has less than 1 percent probability of being the true top rank[153]. A given indicator may end up measuring the wrong things because it can only tell us about the things that can be counted. Indicators may mask large areas of missing or incomplete data, demonstrate inability to draw significant distinctions between entities that are nonetheless hierarchically ranked, conceal much higher levels of underlying uncertainty than the indicator depicts, and make questionable choices about the weighting of different components of composite indicators. Proxies can be very misleading — as where more people dying can improve the average levels of poverty or health[154]. The information that indicators include can be made to appear more robust than it actually is. “‘Raw’ information typically is collected and compiled by workers near the bottom of organizational hierarchies ; but as it is manipulated, parsed, and moved upward, it is transformed so as to make it accessible and amenable for those near the top, who make the big decision. […] The authority of the information parallels the authority of its handlers in the hierarchy[155]”.

What may seem to be merely technical issues often point to underlying theoretical challenges. The supposed facts about transnational social problems such as human trafficking are highly disputed[156]. When indicators are used to measure the seriousness of such problems and the success of efforts to deal with them they are bound to become part of the debate rather than put an end to it. As Davis, Kingsbury and Merry explain, “[i]ndicators often have embedded within them, or are placeholders for, a much more far-reaching theory — which some might call an ‘ideology’ — of what a good society is, or how governance should ideally be conducted to achieve the best possible approximation of a good society or good policy[157]”. Similarly, in her study of the benchmarking of human security, Homolar writes :

the process of translating the concept of human security into a series of tangible, measurable objects (“reification”) relies on operationalising normative assumptions about what constitutes “liveable” human existence into observable and measurable categories. The indicators chosen to make judgements about the state of human security thereby function to both concretise and reproduce abstract ideas about what constitutes a “secure” human life[158].

In practice, indicators (and benchmarks) often rely on “existing concepts, techniques, and categories of understanding that are taken for granted as correct[159]”. For example, a ranking of universities that did not place Harvard near the top would be neither credible nor influential. These problems are compounded because of the questionable comparative methods that lie behind ranking schemes. Indicators deliberately aim to produce “thin descriptions”, their task that of creating equivalence across different cases, developing categories and taxonomies, coding individual phenomena into the categories created[160]. Likewise with benchmarks, “[c]hallenging and contested concepts, such as slavery, acquire fixed and unproblematic meanings, which are presumed to be universally applicable irrespective of cultural context[161]”. As Homolar puts it, “[i]n this regard, the goal of international comparability is prioritised above contextual validity and accuracy[162]”.

On the other hand, while many of these criticisms of indicators have merit, their implications are not always obvious[163]. On the one hand, claims that indicators are measuring the wrong things presuppose that there must be right things to measure and better ways of doing it. Sometimes the answer that is proposed is to look for more impersonal input. Critics of Transparency International’s Corruption Perception Index, for instance, complain that the indicator is based too much on subjective data. Merry, by contrast, sees the problem as over-reliance of quantitative methods. This is seen most clearly in her recent The Seductions of Quantification[164] which discusses indicators in the realm of human rights, gender violence and human trafficking where she calls for the use of more qualitative or narrative data. But can qualitative data be used for ranking exercises without changing their whole nature ? Those who struggle against gender violence or human trafficking do indeed measure it without taking into account the local context. But this could be because, rightly or wrongly, they are trying to spread common standards, not because they are unaware that these differ profoundly. Nor is there any reason to think that qualitative data is necessarily more emancipatory given that anthropologists were often an integral part of colonial missions.

Be that as it may, some criticisms of the accuracy of indicators go beyond pointing to matters that could, even in theory, be remedied. Those who focus on the “knowledge effects” of indicators, for example, tell us that they are “constitutive”, meaning by this that they are not neutral representations of the world but rather that they bring into being the social categories that they then rank[165]. Davis, Kingsbury and Merry explain that :

[t]he name itself is usually a simplification of what the index purports to measure or rank. The name’s constancy may mask changes over time in the indicator itself. Calling an indicator a measure of “transparency” or “human development” asserts a claim that there is such a phenomenon and that the numerical representation measures it. […] [T]he indicator represents an assertion of power to produce knowledge and to define or shape the way the world is understood[166].

Or, as Urueña puts it, “[i]ndicators do not (indeed, cannot) measure reality objectively, in the way a thermometer would measure temperature. Rather, they inject, […], a normative analysis to the reality they measure, and constitute it[167]”.

Looking at existing case studies of indicators we find some cases, as with “failed states”, where indicators constitute a category that did not exist beforehand ; in others, such as the “rule of law” or “corporate responsibility”, the definition marshalled by a given indicator is only one of several possible ones. But even where we cannot get behind the constitutive character of indicators does this mean that the question of their accuracy becomes moot ? What is undeniable is that accuracy rarely matters for its own sake. Its relevance has to do with the role of indicators in testifying to differences or the outcomes of interventions, such as when they are involved in persuading states to pass laws or reform their institutions, or encouraging corporations or other organisations to behave more responsibly. It thus becomes hard to distinguish the success of an indicator from the “successful” realisation of wider goals — such as the reduction of human trafficking or improvement in legal processes.

Do indicators actually fulfil their promise ? Some certainly do. The Corruption Perceptions Index has had an impact in the policy of governments as diverse as Cameroon, Papua New Guinea, and Bosnia Herzegovina. Poland, Romania, and the Baltic states took very seriously their score in the CPI, as they saw that a low ranking would be a blow to their aspirations of joining the European Union[168]. Further studies could help establish the likely conditions of success of given indicators. Preliminary research in central Europe, for example, suggests that indicators of democracy or rule of law or good governance have the most demonstrable influence where an authoritarian regime is weakening or where a new democratic regime has just been established. But it will often be more difficult to show the exact contribution of indicators to the outcome of a given project. As Davis, Kingsbury and Merry tell us, “[t]he theory or idea embedded in an indicator may be developed or reframed by its users or by other actors in ways that differ from anything intended by the producers[169]”.

Admittedly, any social intervention (including legislative measures, judicial or administrative measures) may have unexpected and unintended effects. Indicators are as good a way as any for seeking to test for anomalous outcomes. But the charge against indicators is that they routinely produce the impression that a project is successful when what has really happened is that the target has focused only on the measure rather than the ultimate goal. Indicators’ emphasis on measuring performances has the effect of boosting “rank-seeking” strategies by indicators targets, inducing them to pursue policies designed to improve their rank, rather than their actual behaviour[170]. In the case of university rankings, for example, we learn about “salami slicing” (cutting research results into small chunks, each published as a separate journal article), “rushing to press” (publishing partial results as soon as they are available rather than making a mature and considered analysis) and the “star player” syndrome (hiring high profile researchers just before a research assessment exercise). An early judgement on the research assessment exercise in the UK described it as a “morass of fiddling, finagling and horse trading” that was “starting to lack credibility[171]”.

By contrast, some indicators do not even seriously try to monitor outcomes closely. Sarfaty’s study of indicators for corporate responsibility shows that their goal seems to be more about raising awareness of standards rather than actually applying them. She tells us that “[t]he GRI [Global Reporting Initiative] guidelines are voluntary and incremental, and include a high level of flexibility, allowing companies to decide which principles and indicators to adopt[172]”. For Davis, Kingsbury and Merry, “[i]n a sense, the participation of corporate actors in the regular reformulation of the GRI is a form of contestation from within : they shape what gets measured and what does not and therefore are able to channel the way they are regulated[173]”.

What do we mean by effectiveness ? It has been argued that indicators attempt the impossible when they offer to both measure and improve at the same time. What is known as Goodhart’s law claims that once a social or economic indicator or other surrogate measure is made a target for the purpose of conducting social or economic policy, then it will lose the information content that would qualify it to play that role[174]. On the other hand, some indicators, as with financial rating agencies, are almost self-executing. Once the credit status of a stock is downgraded it is bound to lose value as a result of selling. They work not because they are true, they are true because they work. Some practitioners even seem comfortable with such a post-positivist approach to indicators. This is often the case for those in Inter-governmental and Non-Governmental Organisations engaged in negotiating proxies for the standards they define and implement. Sometimes they knowingly adopt different concepts of the same indicator to use in the different contexts in which they move[175]. Because truth and the effectiveness of indicators are so deeply interconnected truth often becomes no more than a convenient currency. Take university rankings as an example. Once a university receives and maintains a high ranking — parents who want their children to benefit from the prestige will encourage their children to apply there without worrying about what lies behind the evaluation.

But self fulfilling indicators can be dangerous. In the case of algorithms, such as those that direct lenders to those who are more likely to repay their debts or tell the police where to look for crime, there has been much criticism of the way they produce false positives, with the result that the poor get poorer, and offenders drawn from predictable backgrounds are found to commit more crimes[176]. For Hildebrandt, even if algorithmic regulation is justified as a form of decision-making that corrects for human bias and subjectivity, it in fact reinforces the discrimination built into the axioms of the coding[177]. At least some of these worries may apply to indicators ; again taking the case of university indicators, we could think of the way top universities gain more resources and so are more able to stay ahead.

“Governing by numbers”, to it is claimed, however, is most problematic because of its wider effects[178]. Shore and Wright identify five of these that are of particular importance : “domaining effects” ; “classificatory effects” ; “individualising and totalizing effects” ; “governance effects”and “perverse effects”. For them, “governing by numbers, when taken to extremes or misapplied — such as in the Vietnam war — fails to deliver what it promises and, whether one counts beans or bodies, may result in decision making that is amoral or outcomes that are immoral[179]”. Linking indicators to audit culture, they warn of the perverse effects of using aggregated measures and rankings punitively to name and shame.

Conclusion : Should Global Social Indicators be made more Legitimate ?

In one of their earliest discussions of indicators, Davis, Kingsbury and Merry emphasised their law-like features, explaining that “[i]ndicators are attractive to decision makers and designers of decision making processes because processes that rely on indicators can be presented as efficient, consistent, transparent, scientific, and impartial[180]”. More recently, in the introduction to the collection of case studies on global social indicators published in 2015, Davis, Kingsbury and Merry queried how far this process had gone, asking : “Will indicators become law-like, in terms of their influence and perceived legitimacy ? Will law become more indicator-like, with more legal instruments specifying their goals in numerical terms ?[181]” Implicit in these kinds of questions is the implication that we could be witnessing a process by which indicators and law are coming to resemble each other.

In this paper we have reviewed what relevant analyses and case studies can tell us about these possibilities. Rather than serving as the basis for an exercise in futurology, however, we argued that what this literature can help us understand is the potential for overlaps between different kinds of normativity and the relevance this has for the legitimacy of different forms of governance. We discussed first the definition of indicators and suggested that indicators (and benchmarks) can be placed on a normative continuum and compared and contrasted with law on the one hand and algorithmic regulation on the other. We then noted the complex overlap between legitimacy and legitimation. In the second part of the essay we focused on claims concerning the authority, accountability and accuracy — or otherwise — of global social indicators. We also sought to show why it would be quixotic to try to make indicators legitimate according to conventional criteria if their rise reflects and furthers new forms of authority, accountability and performative legitimacy.

These closing remarks will seek to build on the arguments that have been canvassed so far so as to suggest areas where further reflection and research is needed if we are to answer the question when and whether indicators should be made more legitimate. Although it may be helpful for analytic purposes to distinguish the three aspects of legitimacy that we have singled out, the relationship between them undoubtedly needs further examination. How exactly do they go together ? When authors counterpose them, arguing, for example, that it is not enough for indicators to have accuracy because they also need legitimacy, are they really suggesting that accuracy as such is not part of their legitimacy ?[182] Conversely, ensuring good procedure in making and applying indicators will not necessarily produce accuracy or allow us to know whether a standard has been met. Can the criterion of accuracy be based on the views of those being evaluated ? Should it be ? This is likely to work better for some indicators than others, for example the definition of poverty, rather than corruption. Nor will it guarantee the success of interventions according to any external criteria not shared by those whose situation or behaviour is being measured.

In practice the three aspects of legitimacy are usually interdependent, and the lack of one aspect of legitimacy can reinforce problems in the others — for example conflicts of interest are also likely to produce inaccuracy. Less obviously, however, the achievement of one type of legitimacy can sometimes be counter-productive for another. For example if greater transparency draws attention to high levels of corruption more rather than fewer people may feel that they would be foolish not to take that rather than the more legal route to their goals. More commonly, there may be questions about how to satisfy the different requirements of legitimacy, as with the current debate over whether to trust experts and when to increase public participation. But such dilemmas should not be exaggerated — people can mobilise in favour of giving experts the say and against what they see as the rise of populism. Case studies also illustrate how those who make and use indicators manage to reconcile potential conflicts. Thus Von Bogdandy and Goldmann inform us that “the two-staged process in which the mandate of PISA was adopted, composed of a binding OECD decision preceded by exchanges of notes, has had the function of making PISA both more legitimate, as prior consultations of domestic stakeholders might reduce the effects of two-level games on national power balances[183]” and go on to say :

The drafters of PISA recognized that scientific expertise is not always culturally neutral, but that expertise is often influenced by the experiences made by an expert in his or her country of origin or residence. Accordingly, it appears necessary to ensure a geographically balanced selection of experts in order to avoid national or regional biases[184].

But finding ways to improve indicators’ legitimacy may not always be so straightforward. In our survey of the literature we have seen that critics of indicators do not always make it clear whether the drawbacks they identify can or should be remedied. Some scholars go to great lengths to show how indicators are getting things wrong in what they claim to be measuring, which (often) implies that they think the process could be improved. Others (sometimes even the same authors) insist that what indicators (or some indicators ?) measure is actually “constitutive” of the matters that they claim to be measuring. Merry, for example, has repeatedly warned that indicators have what she calls “knowledge” and “governance” “effects[185]”. By this she means that indicators construct or even constitute what they claim to be measuring, and that they deny that they are engaged in political choices pretending rather to be merely technical. But she does not say whether these characteristics are contingent and remediable or should rather be considered part of the very definition of all indicators[186]. The same applies to criticisms of “thin comparisons”. Certainly, a price is paid when complex matters are simplified and standards are made commensurable across very different contexts. But we also need to ask how far these limits can be accepted if the alternative is just relying on biased news sources or prejudices.

The difficulty in deciding whether and how to increase the legitimacy of indicators can also be seen by returning to our discussion of the role of contestation in their making and application. Contestation is already an intrinsic part of the life of most indicators, both in their formulation and consolidation. It occurs at the level of conceptualization, as countries, organisations, groups or individual policy makers, scholars or activists argue about what is the best measure for poverty, development, or violence against women. But it also happens at the level of data as countries or organizations choose what to measure and what to ignore. In meetings of the UN statistical commission, representatives of developing countries regularly insist that if the developed world wants them to measure something, that they will need more resources to do so. Developing countries sometimes object to international data that contradicts national data and demand that their national data should be taken more seriously. But at some point contestation has to be stopped if indicators are to do their job. Who should decide when this should be ?

The politics of indicators goes beyond deciding how to build in contestation. In considering which indicators to strengthen, if any, scholars can be found discussing, on the one hand, objections to the ideas of the good society they embody, and the part they play in constructing the relationship between more or less economically developed societies, or, on the other hand, the kind of governance model they reflect — for example their tendency to substitute technicalities for political participation or their link to audit culture. As Table 1 shows, debates over the legitimation of indicators can thus be categorised along two axis. The first is concerned with whether indicators and benchmarks are likely or not to reinforce the hegemonic power, for example, of states or corporations, or whether they could (also) contribute to counter-hegemonic movements. The second is concerned with whether indicators as a tool can make governance more legitimate or are rather reconfiguring it in new and questionable directions.

Table 1

Global Social Indicators, Hegemony and Audit Culture

Take first the issue of the goals of indicators (or, better, of the projects to which they belong). Many commentators distinguish between indicators on the basis of the goals they help to pursue. For example, those who have reservations about the neo-liberal market maximising goals that are built into the “Better Business” indicator may be better disposed towards indicators designed for purposes such as measuring the levels of human rights. The development of the human capabilities indicator was explicitly intended to provide an alternative yardstick to neo- liberal economic goal[187]. Some, however, question how far even the latter goals can be described as counter-hegemonic[188], and we should not forget that hegemony itself changes over time, as well illustrated by the way the World Bank has recently changed its views about the economic benefits of austerity programmes[189]. Likewise, some link global indicators to neo-colonial or post-colonial goals[190]. But reluctance to spread “our” standards can also mean abandoning ideals that cannot be confined geographically if they are to work at all, or, worse, lead to us colluding in allowing other places to serve our interests by applying what are for us lower standards.

Whatever the ostensible goal of an indicator, much will depend on how it is actually deployed and the field of social forces it encounters. This is well seen in a recent stimulating study by Seabrooke and Wigan of the way benchmarks can play both a hegemonic and counter-hegemonic role. Although their focus is on the benchmarks used in the transnational campaign against political corruption as contrasted with the less successful battle for global tax justice, their analysis can be generalised to other benchmarks, and indeed also to global indicators. According to them, “(a)s symbolic weapons benchmarks are useful for both reformist and revolutionary agendas[191]”. As they put it, “[t]he first is a reformist benchmarking cycle where organisations defer to experts to create a benchmark that conforms with the broader system of politico-economic norms. The second is a revolutionary benchmarking cycle driven by expert-activists that seek to contest strong vested interests and challenge established politico-economic norms[192]”. The authors argue that benchmarks or indicators that try to go against hegemonic ideas and groups will face considerable difficulties but that they also hold out the promise that sometime and somehow success is just possible.

For those involved in the debate captured in our other axis the issue is rather whether there is something wrong with indicators as an instrument, whatever the goal, hegemonic or otherwise, for which they are used. On the one hand, Rosga and Satterthwaite seek to distinguish goals and means. They tell us that “the human rights community’s efforts to use statistics as part of a larger project of holding governments accountable to their populations, while it partakes of the same technologies of governmentality, can arguably be said to aim at different ends[193]”. They conclude :

[t]he value of indicators as a social technology cannot be determined in advance, nor on the basis of the fact that they are largely quantitative. While it may be true that quantitative methods, in their very abstraction and stripping away of contextualizing information have particular – and especially high – risks for misuse by those with the power to mobilize them, they are tools like any other. All tools can be misused ; all social actors with power can misuse that power[194].

For other commentators, however, indicators need to be resisted because they “tend to foreclose other modes of creating and expressing value,” in particular, “those forms of evaluation that are messier and more time consuming and that require deliberation or consensus[195]”. Some even argue that we are witnessing a “struggle between the rule of law and governance by indicators” and look forward to asserting the “supremacy of law or at least a certain degree of control over indicators, with a view to ensuring the survival of fundamental values conveyed by law, such as the rule of law and fundamental rights[196]”. They call for publicity, competence, independence, and integrity, justification and rectification, liability and judicial review[197]. Whatever other differences there may be, therefore, indicators are here seen as posing many of the same concerns as algorithmic regulation, which, according to Hildebrandt, “will drain the life from the law, turning it into a handmaiden of governance (that fashionable term meaning anything to anybody), devouring the procedural kernel of the Rule of Law that enables people to stand up for their rights against big players, whether governmental or corporate or otherwise[198]”.

The answer Hilderbrandt proposes, one that we have already seen in discussing the accountability of indicators, calls for some combination of legal oversight and public contestation. She tells us that “to the extent that such determinations are not contestable by those subjected to their implications, we will have to reinvent such contestation to uphold the Rule of Law[199]”. But she goes further telling us that “[a] world focused on regulating, influencing or even enforcing behaviours has little to do with legal normativity. If one cannot disobey a law it is not but discipline[200]”. She thus suggests that, paradoxically, unlike the indicators search for accuracy, law’s legitimacy comes in part from it not being able to guarantee results — as well as from its being “redundant, ambiguous and indistinct[201]”.

On the other hand, imagining that we face a life and death struggle between law and other kinds of normativity may over-idealise law, and exaggerate the contrast between “law” and “administration”. In actual use law has often been criticized for many of the same faults as indicators. It is seen as being too often used to promote ideological goals and agendas, whilst being subject to manipulation by those who frame its meaning in actual practice and subverted by those who game their responses[202]. Law too swings between rules and discretion, oversimplifies factual situations, invents and imposes categories, etc. And, in terms of Hilderbrant’s claims, it is far from obvious that law can interrogate its own biases better than more scientifically-oriented types of normativity. Nor is it always more open to mutual accountability than other kinds of normativity. It is in any case questionable how far there can be reciprocity when — as with law and indicators — the point is that one side is involved in imposing standards on others.

We also should not underestimate the extent to which law and other types of normativity can and do reshape each other. Police engage in “actuarial justice[203]”, and engage in forms of prevention described as “pre-crime[204]”. And Hildebrandt herself has argued that law should be seen as a highly successful early version of cybernetic regulation even if, according to her, it is now “on its last legs[205]”. It can therefore be misleading to exaggerate the incompatibility of law with indicators — or even with algorithms, and more instructive to seek to disentangle the ways law, indicators and algorithms interact so as to create new and unexpected hybrids[206]. As Restrepo Amariles shows us, in describing what is happening in the realm of financial indicators, “law, as a process, embraces part of the managerial toolbox in its everyday life, opening new horizons to think about the creation, implementation and enforcement of norms, and not only of rules, in the managerial terms of data-gathering, benchmarking and auditing[207]”.

The future is not yet written. Santos, in his manifesto for a “new common sense”, argues that we can and should challenge the use of science and precise indicators and invoke a sociology of absence and emergence and what he calls transforming impossible into possible objects[208]. A major obstacle to his new common sense, however, lies in the extent to which reconfigured forms of governance succeed in persuading people that they are getting what they want. As Shore and Wright put it, “[r]ankings and measures may appear as external, constraining and oppressive, but they are also empowering and self-affirming, or at least for some people[209]”. For them, “the political technologies of financial cost accounting wedded to the project of management have been highly effective in producing accountable transparent subjects that are simultaneously docile yet self-managed[210]”.

As this suggests, it is not just a question of learning how to build into transnational standardising indicators the constraints that legal institutions are asked to respect, but rather how far we should try to do this The supposed irresolvable tension between measurement and improvement, for example, can be overcome by assimilating indicators more to algorithms, given that these instruments revise their predictions in terms of feedback. But the more indicators are set up to work on a similarly experimental basis the less they will resemble the more counterfactual features of legal regulation. We should be wary of building a dystopia where indicators imitate the kind of algorithmic regulation that reconfigures the neo-liberal self as primarily a “consumer of digital services, rather than a politically active citizen engaged in processes of public deliberation[211]”. Currently, in contrast to algorithms, many indicators are still exposed to the possibility of violation, of gaming and of people negotiating their implementation in unpredictable ways. What we have tried to show in this paper is why the possibility of such outcomes should be considered an important sign of their legitimacy and not taken as evidence of their failed legitimation.

Notes

• [1] Wislawa Szymborska, Here, New York, Houghton Mifflin Harcourt, 2010, p. 3. English translation : “Ignorance works overtime here, something is always being counted, compared, measured, from which roots and conclusions are then drawn.”
• [2] I have relied in particular on Kevin E. Davis et al. (eds.), Governance by Indicators. Global Power through Quantification and Rankings, Oxford, Oxford University Press, 2012 ; Sally Engle Merry, Kevin E. Davis and Benedict Kingsbury (eds.), The Quiet Power of Indicators. Measuring Governance, Corruption, and Rule of Law, New York, Cambridge University Press, 2015 ; Richard Rottenburg et al. (eds.), The World of Indicators. The Making of Governmental Knowledge through Quantification, Cambridge, Cambridge University Press, 2015. The second of these edited volumes has most to say specifically about questions of legitimacy. Other recent interesting collections on indicators include Benoît Frydman and Arnaud Van Waeyenberge (eds.), Gouverner par les standards et les indicateurs. De Hume aux rankings, Bruxelles, Bruylant, 2014 ; and Benoît Frydman and William Twining, “A Symposium on Global Law, Legal Pluralism and Legal Indicators”, (2015) 47 The Journal of Legal Pluralism and Unofficial Law 1.
• [3] See also David Nelken, “The Changing Roles of Social Indicators : From Explanation to Governance”, in Valsamis Mitsilegas, Peter Alldridge and Leonidas Cheliotis (eds.), Globalisation, Criminal Law and Criminal Justice. Theoretical, Comparative and Transnational Perspectives, London, Hart Publishing, 2015, p. 25.
• [4] The faith in using social indicators for the purpose of policy evaluations declined in the 1970’s, but went up again in the 1990’s. See e.g. Matthias Schmelzer, The Hegemony of Growth. The OECD and the Making of the Economic Growth Paradigm, Cambridge, Cambridge University Press, 2016, p. 311, discussing the use of indicators by the OECD.
• [5] On which see Nils Brunsson and Bengt Jacobsson, A World of Standards, Oxford, Oxford University Press, 2002.
• [6] Cris Shore and Susan Wright, “Governing by Numbers : Audit Culture, Rankings and the New World Order”, Social Anthropology, vol. 23, n^o 1, 2015, p. 22, at page 22.
• [7] See André Broome and Joel Quirk, “Governing the World at a Distance : the Practice of Global Benchmarking”, Review of International Studies, vol. 41, 2015, p. 819.
• [8] Mathias Siems and David Nelken, “Global Social Indicators and the Concept of Legitimacy”, (2017) 13 International Journal of Law in Context 436.
• [9] Kevin E. Davis, Benedict Kingsbury and Sally Engle Merry, “Introduction : Global Governance by Indicators”, in K.E. Davis et al. (eds.), supra, note 2, p. 3, at page 6 (our emphasis).
• [10] Id., at page 7.
• [11] Id.
• [12] Id., at page 8.
• [13] Id.
• [14] Id., at pages 8 and 9.
• [15] See e.g. Nikhil K. Dutta, “Accountability in the Generation of Governance Indicators”, in K.E. Davis et al. (eds.), supra, note 2, p. 437.
• [16] Alain Desrosières, “Retroaction : How Indicators Feed Back onto Quantified Actors”, in R. Rottenburg et al. (eds.), supra, note 2, p. 329.
• [17] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 6.
• [18] See e.g. Michael Riegner, “Towards an International Institutional Law of Information”, (2015) 12 International Organizations Law Review 50.
• [19] Richard Rottenburg and Sally Engle Merry, “A World of Indicators : the Making of Governmental Knowledge through Quantification”, in R. Rottenburg et al. (eds.), supra, note 2, p. 1, at page 18.
• [20] See R. Rottenburg et al. (eds.), supra, note 2.
• [21] See Nehal Bhuta, “Governmentalizing Sovereignty : Indexes of State Fragility and the Calculability of Political Order”, in K.E. Davis et al. (eds.), supra, note 2, p. 132.
• [22] Susanne Karstedt, “Exit : The State. Globalization, State Failure and Crime”, in David Nelken (ed.), Comparative Criminal Justice and Globalization, Farnham, Ashgate Publishing, 2011, p. 107.
• [23] Kevin E. Davis, Benedict Kingsbury and Sally Engle Merry, “Introduction. The Local-Global Life of Indicators : Law, Power, and Resistance”, in S.E. Merry, K.E. Davis and B. Kingsbury (eds.), supra, note 2, p. 1, at page 18.
• [24] Id., at page 19.
• [25] Id., at page 20.
• [26] Judith G. Kelley and Beth A. Simmons, “Politics by Number : Indicators as Social Pressure in International Relations”, American Journal of Political Science, vol. 59, n^o 1, 2015, p. 55.
• [27] There is no consensus on how to distinguish indicators from benchmarks. For many students of indicators, benchmarking represents just one part of the way they seek to measure performance. But those who focus more on benchmarks treat indicators as one of their elements (A. Broome and J. Quirk, supra, note 7, at page 820).
• [28] André Broome and Joel Quirk, “The Politics of Numbers : The Normative Agendas of Global Benchmarking”, Review of International Studies, vol. 41, 2015, p. 813, at page 813.
• [29] A. Broome and J. Quirk, supra, note 7, at page 819.
• [30] Algorithms are defined as “[a] process or set of rules to be followed in calculations or other problem-solving operations, especially by a computer”. See English Oxford Dictionary, “algorithm”, [Online], [en.oxforddictionaries.com/definition/algorithm] (November 28^th, 2017).
• [31] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 23, at page 22.
• [32] Karen Yeung, Algorithmic Regulation, 2017, [Online] [www.nasonline.org/programs/sackler-forum/frontiers-of-machine-learning/yeung-ppt.pdf] (November 7^th, 2017).
• [33] S.E. Merry, K.E. Davis and B. Kingsbury (eds.), supra, note 2.
• [34] Frank Pasquale, The Black Box Society. The Secret Algorithms That Control Money and Information, Cambridge, Harvard University Press, 2015.
• [35] Cris Shore and Susan Wright, “Audit Culture Revisited. Rankings, Ratings, and the Reassembling of Society”, Current Anthropology, vol. 56, n^o 3, 2015, p. 421, at page 425.
• [36] Tony Porter, “Global Benchmarking Networks : the Cases of Disaster Risk Reduction and Supply Chains”, Review of International Studies, vol. 41, 2015, p. 865, at page 868, quoting Jeffrey Alstete.
• [37] David Restrepo Amariles, “Legal Indicators, Global Law and Legal Pluralism : an Introduction”, (2015) 47 The Journal of Legal Pluralism and Unofficial Law 9, 18.
• [38] Perelman Center for Legal Philosophy and International Association of Legal Methodology, Call for Papers – Colloquium on Internormativity, [Online] [www.philodroit.be/IMG/pdf/call_for_papers_-_ialm.pdf] (November 7^th, 2017). We should also not forget that there may be multiple and potentially conflicting indicators in play all dealing with the same matters, such as human trafficking, or the rule of law.
• [39] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 23, at page 15.
• [40] Id., at pages 18 and 19.
• [41] Id., at page 19.
• [42] Id., at pages 19 and 20.
• [43] Benoît Frydman, “From Accuracy to Accountability : Subjecting Global Indicators to the Rule of Law”, (2017) 13 International Journal of Law in Context 450.
• [44] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 23, at page 20.
• [45] Id.
• [46] Id.
• [47] Armin Von Bogdandy and Matthias Goldmann, “The Exercise of Public Authority through National Policy Assessment”, Institute for International Law and Justice Working Paper 2009/2, p. 42.
• [48] Id.
• [49] Id.
• [50] Niklas Luhmann, Law as a Social System, Oxford, Oxford University Press, 2004, p. 93.
• [51] The important Foucauldian analysis by Ewald uses research on insurance and industrial standardisation to chart “normalisation” as a process that collapses the “norm” and the “normal”. François Ewald, “Norms, Discipline, and the Law”, Representations, n^o 30, 1990, p. 138.
• [52] Michel Foucault, The History of Sexuality. An Introduction, vol. 1, London, Allen Lane, 1979, p. 89.
• [53] Amanda Perry-Kessaris, “The Re-co-construction of Legitimacy of/through the Doing Business Indicators”, (2017) 13 International Journal of Law in Context 498.
• [54] David Restrepo-Amariles, “Supping with the Devil ? Indicators and the Rise of Managerial Rationality in Law ?”, (2017) 13 International Journal of Law in Context 465, 472.
• [55] See e.g. Johanna Mugler, “By their own Account : (Quantitative) Accountability, Numerical Reflexivity and the National Prosecuting Authority in South Africa”, in R. Rottenburg et al. (eds.), supra, note 2, p. 76.
• [56] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 15.
• [57] “Much of the political power of human security indicators and benchmarks rests on their dual role in global governance discourses. On the one hand, they serve to underwrite legitimation claims by international political actors in pursuit of specific policy agendas to change state behaviour. On the other, they act as an ‘authorising force’ in the creation of international legal standards, such as the emerging norm of the ‘responsibility to protect’”. (Alexandra Homolar, “Human Security Benchmark : Governing Human Wellbeing at a Distance”, Review of International Studies, vol. 41, 2015, p. 843, at page 863).
• [58] William Davies, “Spirits of Neoliberalism : ‘Competitiveness’ and ‘Wellbeing’ Indicators as Rival Orders of Worth”, in R. Rottenburg et al. (eds.), supra, note 2, p. 283, at page 285.
• [59] Marilyn Strathern, Audit Cultures. Anthropological Studies in Accountability Ethics and Academy, London, Routledge, 2000, p. 1.
• [60] B. Frydman, supra, note 43.
• [61] Id., 450.
• [62] Id.
• [63] Kevin E. Davis, Benedict Kingsbury and Sally Engle Merry, “Indicators as a Technology of Global Governance”, Institute for International Law and Justice Working Papers 2010/2 Rev., p. 19.
• [64] B. Frydman, supra, note 43, 457.
• [65] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 16.
• [66] See M. Siems and D. Nelken, supra, note 8, 440.
• [67] Sabino Cassese and Lorenzo Casini, “Public Regulation of Global Indicators”, in K.E. Davis et al. (eds.), supra, note 2, p. 465.
• [68] Marianne Beisheim and Klaus Dingwerth, “Procedural Legitimacy and Private Transnational Governance. Are the Good Ones Doing Better ?”, SFB-Governance Working Paper Series, n^o 14, 2008, p. 3.
• [69] But, for a major and influential exception amongst social theorists see Jürgen Habermas, Between Facts and Norms. Contribution to a Discourse Theory of Law and Democracy, Cambridge, MIT Press, 1996.
• [70] David Beetham, The Legitimation of Power, 2^nd ed., London, Palgrave, 2013.
• [71] Likewise Yeung explains that “faith in the market as a vehicle for securing algorithmic accountability seems completely misplaced given the opacity of the underlying algorithms and the lack of awareness or understanding by many digital service users of their significance and operation) and the dominance of a handful of extraordinarily powerful transnational companies in a global networked market for digital services”. K. Yeung, supra, note 32.
• [72] N.K. Dutta, supra, note 15, at page 464.
• [73] Andrew Farlow, “Financial Indicators and the Global Financial Crash”, in R. Rottenburg et al. (eds.), supra, note 2, p. 220.
• [74] Max Weber, Economy and Society, Berkeley, University of California Press, 1968, p. 31.
• [75] Alan Hyde, “The Concept of Legitimation in the Sociology of Law”, (1983) Wis. L. Rev. 379.
• [76] Tom R. Tyler, “Psychology and the Law”, in Keith E. Whittington, R. Daniel Kelemen and Gregory A. Caldeira (eds.), The Oxford Handbook of Law and Politics, Oxford, Oxford University Press, 2008, p. 711, at page 717.
• [77] D. Restrepo-Amariles, supra, note 54, 466.
• [78] Id., 467.
• [79] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 23, at page 9.
• [80] Halliday and Shaffer likewise describe the way transnational legal orders are contested unless and until “settled”. Terence C. Halliday and Gregory Shaffer (eds.), Transnational Legal Orders, New York, Cambridge University Press, 2015.
• [81] M. Siems and D. Nelken, supra, note 8.
• [82] Id.
• [83] Marie Guimezanes, “An Inquiry into the Life and Death of Indicators. The Case of the Aid Effectiveness Regime”, (2015) 12 International Organizations Law Review 116, 125.
• [84] Marta Infantino, “The Law of Indicators on Women’s Human Rights : Unmet Promises and Global Challenges”, IRPA Working Paper – GAL Series n^o 9/2012, p. 21. See also Marta Infantino, “Human Rights Indicators across Institutional Regimes”, (2015) 12 International Organizations Law Review 146.
• [85] A. Broome and J. Quirk, supra, note 7, at page 838.
• [86] Michael Power, The Audit Society : Rituals of Verification, Oxford, Oxford University Press, 1997.
• [87] M. Strathern, supra, note 59, p. 8.
• [88] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 63, p. 15.
• [89] A. Desrosières, supra, note 16.
• [90] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 4.
• [91] Id., at page 13.
• [92] D. Restrepo-Amariles, supra, note 54, 465.
• [93] A. Broome and J. Quirk, supra, note 7, at page 826.
• [94] Id.
• [95] Armin Von Bogdandy and Matthias Goldmann, “Taming and Framing Indicators : A Legal Reconstruction of the OECD’s Programme for International Student Assessment (PISA)”, in S.E. Merry, K.E. Davis and B. Kingsbury (eds.), supra, note 2, p. 52, at page 85.
• [96] D. Restrepo-Amariles, supra, note 54.
• [97] C. Shore and S. Wright, supra, note 6, at page 23.
• [98] Those convinced by Latour’s ideas about what he calls “actants” (e.g. Bruno Latour, Politics of Nature. How to Bring Science Into Democracy, Cambridge, Harvard University Press, 2004) might prefer to ask what is the new ruler ?
• [99] Tim Buthe and Walter Mattli, The New Global Rulers. The Privatization of Regulation in the World Economy, Princeton, Princeton University Press, 2011.
• [100] Christopher G. Bradley, “International Organizations and the Production of Indicators. The Cases of Freedom House”, in S.E. Merry, K.E. Davis and B. Kingsbury (eds.), supra, note 2, p. 27.
• [101] On which see Bal Sokhi-Bulley, Governing (Through) Rights, Oxford, Hart Publishing, 2016.
• [102] C. Shore and S. Wright, supra, note 6, at page 24.
• [103] Id., at page 22.
• [104] Id., at page 24.
• [105] Leonard Seabrooke and Duncan Wigan, “How Activists Use Benchmarks : Reformist and Revolutionary Benchmarks for Global Economic Justice”, Review of International Studies, vol. 41, 2015, p. 887, at page 890.
• [106] M. Siems and D. Nelken, supra, note 8, 444.
• [107] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 63, p. 9.
• [108] M. Infantino, “Human Rights Indicators across Institutional Regime”, supra, note 84, 152.
• [109] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 63, p. 16.
• [110] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 15.
• [111] Again, much the same can be said of global benchmarking which rely “primarily on the imprimatur of scientific expertise for establishing authority” (A. Broome and J. Quirk, supra, note 28, at page 816). Writers on benchmarks also place the emphasis on the networks on which they are embedded rather than the single actors who make and use them (T. Porter, supra, note 36, at page 866.).
• [112] R. Rottenburg and S.E. Merry, supra, note 19, at page 20.
• [113] Sally Engle Merry, “Measuring the World. Indicators, Human Rights, and Global Governance”, Current Anthropology, vol. 52, n^o S3, 2011, p. S83, at page S89 ; and likewise, for benchmarks, see A. Broome and J. Quirk, supra, note 7, at pages 829 and 830.
• [114] Morag Goodwin, “The Poverty of Numbers : Reflections on the Legitimacy of Global Development Indicators”, (2017) 13 International Journal of Law in Context 485.
• [115] Nico Krisch, “Authority, Solid and Liquid, in Postnational Governance”, in Roger Cotterrell and Maksymilian Del Mar (eds.), Authority in Transnational Legal Theory. Theorising Across Disciplines, Cheltenham, Edward Elgar Publishing, 2016, p. 25.
• [116] M. Siems and D. Nelken, supra, note 8, 442.
• [117] Id.
• [118] M. Power, supra, note 86.
• [119] C. Shore and S. Wright, supra, note 35, at page 430.
• [120] C. Shore and S. Wright, supra, note 6, at page 24.
• [121] M. Riegner, supra, note 18, 51.
• [122] Id.
• [123] M. Goodwin, supra, note 114, 495.
• [124] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 18.
• [125] T. Porter, supra, note 36.
• [126] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 63, p. 24.
• [127] N.K. Dutta, supra, note 15.
• [128] Ronen Shamir and Dana Weiss, “Semiotics of Indicators : The Case of Corporate Human Rights Responsibility”, in K.E. Davis et al. (eds.), supra, note 2, p. 110 ; Galit A. Sarfaty, “Measuring Corporate Accountability through Global Indicators”, in S.E. Merry, K.E. Davis and B. Kingsbury (eds.), supra, note 2, p. 103.
• [129] M. Riegner, supra, note 18, 54.
• [130] Id.
• [131] A. Von Bogdandy and M. Goldmann, supra, note 47.
• [132] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at pages 20 and 21.
• [133] M. Siems and D. Nelken, supra, note 8.
• [134] For an up to date analysis see B. Frydman, supra, note 43.
• [135] S. Cassese and L. Casini, supra, note 67.
• [136] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 19.
• [137] But empirical research on the contestation of human rights standards by states from the Global South shows the (perhaps inevitable) difficulties of attempts at the same time to spread and “re-politicise” such indicators (See Lena J. Kruckenberg, The UNreal World of Human Rights. An Ethnography of the UN Committee on the Elimination of Racial Discrimination, Baden-Baden, Nomos, 2012).
• [138] AnnJanette Rosga and Margaret L. Satterthwaite, “The Trust in Indicators : Measuring Human Rights”, New York University Public Law and Legal Theory Working Paper n^o 91, 2008, p. 61. Some of those who study rule of law indicators agree (see e.g. René Urueña, “Indicators and the Law. A Case Study of the Rule of Law Index”, in S.E. Merry, K.E. Davis and B. Kingsbury (eds.), supra, note 2, p. 75).
• [139] For an exceptional case see Debbie Collier and Paul Benjamin, “Measuring Labor Market Efficiency. Indicators that Fuel an Ideological War and Undermine Social Concern and Trust in the South African Regulatory Process”, in S.E. Merry, K.E. Davis and B. Kingsbury (eds.), supra, note 2, p. 284, who describe the successful effort to resist the World Bank’s indicator that equated ease of doing business with a lack of protection for workers. But this was an exceptional case in which indicators were at odds with other international agreements.
• [140] See e.g. Migai Akech, “Evaluating the Impact of Corruption Indicators on Governance Discourses in Kenya”, in S.E. Merry, K.E. Davis and B. Kingsbury (eds.), supra, note 2, p. 248 and Mihaela Serban, “Rule of Law Indicators as a Technology of Power in Romania”, in S.E. Merry, K.E. Davis and B. Kingsbury (eds.), supra, note 2, p. 199.
• [141] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 23, at page 18.
• [142] M. Siems and D. Nelken, supra, note 8.
• [143] David Nelken, “Conclusion. Contesting Global Indicators”, in S.E. Merry, K.E. Davis and B. Kingsbury (eds.), supra, note 2, p. 317.
• [144] Conversely, showing consent – the absence of contestation – may not be enough to make indicators legitimate. As Yeung explains “the liberal focus on notice and consent fails to grapple with the particular way in which Big Data algorithmic techniques exert behavioural influence through the hyperpersonalisation of individuals’ informational choice environments” using our cognitive flaws to manipulate us (Karen Yeung, “‘Hypernudge’ : Big Data as a Mode of Regulation by Design”, Information, Communication & Society, vol. 20, n^o 1, 2017, p. 118, at page 131).
• [145] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 21.
• [146] Id.
• [147] C. Shore and S. Wright, supra, note 35, at page 431.
• [148] See M. Goodwin, supra, note 114.
• [149] Liam Clegg, “Benchmarking and Blame Games : Exploring the Contestation of the Millennium Development Goals”, Review of International Studies, vol. 41, 2015, p. 947.
• [150] M. Strathern, supra, note 59.
• [151] Evelyn Ruppert, “Doing the Transparent State : Open Government Data as Performance Indicators”, in R. Rottenburg et al. (eds.), supra, note 2, p. 127.
• [152] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 8.
• [153] M. Siems and D. Nelken, supra, note 8.
• [154] Thomas Pogge, “Fighting Global Poverty”, (2017) 13 International Journal of Law in Context 512.
• [155] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 9.
• [156] David Nelken, “Human Trafficking and Legal Culture”, (2010) 43 Israel Law Review 479.
• [157] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 9.
• [158] A. Homolar, supra, note 57, at page 846.
• [159] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 19.
• [160] R. Rottenburg and S.E. Merry, supra, note 19, at page 12.
• [161] A. Broome and J. Quirk, supra, note 28, at page 814.
• [162] A. Homolar, supra, note 57, at page 861.
• [163] Sital Kalantry, book review of The Quiet Power of Indicators : Measuring Governance, Corruption, and the Rule of Law by Sally Engle Merry, Kevin E. Davis and Benedict Kingsbury (eds.), Nordic Journal of Human Rights, vol. 34, n^o 3, 2016, p. 226.
• [164] Sally Engle Merry, The Seductions of Quantification. Measuring Human Rights, Gender Violence, and Sex Trafficking, Chicago, University of Chicago Press, 2016.
• [165] See R. Rottenburg and S.E. Merry, supra, note 19, at page 5. Arguably this can even be said of the status of the “units” being assessed, where supposedly similar nation-states are compared even though they have in fact very different characters and resources.
• [166] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 9, at page 8.
• [167] R. Urueña, supra, note 138, at page 89.
• [168] See Rene Urueña, « Indicators as Political Spaces », (2015) 12 International Organizations Law Review 1, 7.
• [169] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 63, p. 11.
• [170] For a good illustration in the area of human rights see Marina Zaloznaya and John Hagan, “Fighting Human Trafficking or Instituting Authoritarian Control ? The Political Co-optation of Human Rights Protection in Belarus”, in K.E. Davis et al. (eds.), supra, note 2, p. 344.
• [171] Cited in C. Shore and S. Wright, supra, note 35, at page 425 ; see also the thorough study of Law school rankings in the USA by Wendy Nelson Espeland and Michael Sauder, “Rankings and Reactivity : How Public Measures Recreate Social Worlds”, American Journal of Sociology, vol. 113, n^o 1, 2007, p. 1.
• [172] G.A. Sarfaty, supra, note 128, at page 109.
• [173] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 23, at page 17.
• [174] C.A.E. Goodhart, “Monetary Relationships : A View from Threadneedle Street”, in Papers in Monetary Economics, vol. 1, Sydney, Reserve Bank of Australia, 1975.
• [175] Richard Rottenburg, Far-Fetched Facts. A Parable of Development Aid, Cambridge, MIT Press, 2009.
• [176] Cathy O’Neil, Weapons of Math Destruction : How Big Data Increases Inequality and Threatens Democracy, New York, Crown, 2016.
• [177] Mireille Hildebrandt, Smart Technologies and the End(s) of Law. Novel Entanglements of Law and Technology, Cheltenham, Edward Elgar Publishing, 2015.
• [178] C. Shore and S. Wright, supra, note 6.
• [179] C. Shore and S. Wright, supra, note 35, at page 426.
• [180] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 63, p. 21.
• [181] K.E. Davis, B. Kingsbury and S.E. Merry, supra, note 23, at page 22.
• [182] See e.g. A. Von Bogdandy and M. Goldmann, supra, note 95 ; B. Frydman, supra, note 43.
• [183] A. Von Bogdandy and M. Goldmann, supra, note 47, p. 36.
• [184] Id., p. 37.
• [185] See e.g. S.E. Merry, supra, note 113, at page S84.
• [186] On the other hand, more recently, Pogge has written that “[t]wo kinds of legitimacy worries may arise about any […] indicator : one regarding its reliability as a measure of progress and another regarding the uses to which it is being put” (T. Pogge, supra, note 154, 512). These questions about indicators parallel Merry’s concerns with knowledge and governance effects. But Pogge does not treat these problems as intrinsic to indicators, and indeed he claims to have found a method of establishing poverty indicators which would largely overcome such worries.
• [187] See e.g. Martha C. Nussbaum, “Capabilities of Human Rights”, (1997) 66 Fordham L. Rev. 273 ; Amartya Sen, Development as Freedom, New York, Alfred A. Knopf, 1999.
• [188] For Davies, the pursuit of goals apparently incompatible with market maximising neoliberalism may only reflect its complex and its changing forms, because neoliberalism itself embraces, as his chapter title puts it “ ‘competitiveness’ and ‘wellbeing’ as rival orders of worth” (W. Davies, supra, note 58).
• [189] A. Perry-Kessaris, supra, note 53.
• [190] Amanda Perry-Kessaris, “Prepare your Indicators : Economics Imperialism on the Shores of Law and Development”, (2011) 7 International Journal of Law in Context 401 ; Eve Darian-Smith, “Mismeasuring Humanity : Examining Indicators Through a Critical Global Studies Perspective”, New Global Studies, vol. 10, n^o 1, 2016, p. 73.
• [191] L. Seabrooke and D. Wigan, supra, note 105, at page 887.
• [192] Id.
• [193] A. Rosga and M.L. Satterthwaite, supra, note 138, p. 57.
• [194] Id., p. 61.
• [195] Wendy Nelson Espeland and Michael Sauder, Engines of Anxiety. Academic Rankings, Reputation, and Accountability, New York, Russell Sage, 2016, p. 201. Though the difference of opinion here could perhaps also be linked to the extent the authors are sympathetic towards for the kind of goals pursued by the indicators they are discussing.
• [196] B. Frydman, supra, note 43, 461.
• [197] Id., 460.
• [198] Mireille Hildebrandt, “Law as Information in the Era of Data-Driven Agency”, (2016) 79 Mod. L. Rev. 1, 2.
• [199] Id., 26.
• [200] M. Hildebrandt, supra, note 177, p. 226.
• [201] Hilderbrandt is not of course suggesting that we should set out to get things wrong, but rather questioning particular ways of reaching certainty (M. Hilderbrandt, supra, note 198, 29).
• [202] See, most recently, David Kennedy, A World of Struggle. How Power, Law, and Expertise Shape Global Political Economy, Princeton, Princeton University Press, 2016.
• [203] Malcolm Feeley and Jonathan Simon, “Actuarial Justice : the Emerging New Criminal Law”, in David Nelken (ed.), The Futures of Criminology, London, Sage Publications, 1994, p. 173.
• [204] Lucia Zedner, “Pre-Crime and Post-Criminology ?”, Theoretical Criminology, vol. 11, n^o 2, 2007, p. 261.
• [205] M. Hildebrandt, supra, note 198, 7 ; see also Tim Murphy, The Oldest Social Science ? Configurations of Law and Modernity, Oxford, Clarendon Press, 1997.
• [206] Valverde argues that legal knowledge formation consists of a hybrid of scientific, legal and common sense based knowledges (Mariana Valverde, Law’s Dream of a Common Knowledge, Princeton, Princeton University Press, 2003).
• [207] D. Restrepo Amariles, supra, note 37, 17.
• [208] Boaventura de Sousa Santos, Toward a New Legal Common Sense : Law, Globalization, and Emancipation, 2^nd ed., London, Butterworths LexisNexis, 2002 ; Boaventura de Sousa Santos, Epistemologies of the South : Justice Against Epistemicide, Abingdon, Routledge, 2016 ; see also E. Darian-Smith, supra, note 190.
• [209] C. Shore and S. Wright, supra, note 6, at page 27.
• [210] C. Shore and S. Wright, supra, note 35, at page 430.
• [211] K. Yeung, supra, note 144, at page 132.