Broadening the perspective

Today, there are hundreds of approaches to rank and rate phenomena that are complex, dynamic and multi-dimensional, for instance, democracy, governance, freedom or corruption. In the process, “governance” with its sub-categories has become the prevailing label, sometimes merged into the term “democratic governance”, to use the language of the UNDP.

Various parties are compiling indices. They include international organisations (the World Bank, for instance), NGOs (like Freedom House), foundations (Bertelsmann Foundation and others), academic institutions and individual scholars (for example, Jaggers and Gurr). However, even though there is a large number of freely available indicators, only a handful are widely used. Attention is focussed on a few well-established names.

Conceptual challenges

In spite of the lack of basic data in many developing countries, there is a good supply of governance and democracy indicators. Some even cover every country in the world. They cannot only be based on data which are gathered objectively. For the most part, they are based on expert assessments and complemented by household surveys and other empirical research, which are transformed into figures. This aggregation process poses a host of conceptual and methodological challenges.

Given the complex and multi-dimensional nature of the assessed phenomena, it is challenging to design ratings or rankings, especially for cross-national comparisons. From the outset, countries differ in development paths and what outside influences they are exposed to. Attempts to squeeze all political systems of the world into one-size-fits-all rankings ranging from tyranny to democracy and from good to bad governance run the risk of oversimplification and have to dispense with nuance and detail. Highly complex, multidimensional issues are reduced to straight lines between opposing terms, with graded scales indicating differentiation, reducing them to their core elements.

Therefore, indices sometimes lack coherence and transparency in relation to the underlying definitions, which often remain blurred. To avoid pitfalls, indices tend to use very minimalist definitions. Another option, when compiling several studies into an overarching one, is to assume that all definitions used are legitimate and accept that the resulting “big picture” remains imperfect.

Labelling problem

Democracy and governance indicators conceptually set out from different levels. Some of the underlying concepts refer to the de jure state of affairs, whereas others focus on process and de facto issues and yet others serve to specify policy results. Typically, constitutional arrangements are easier to assess than processes, such as the free and fair conduct of elections.

Anyone who considers the disaggregated dimensions of indicators understands that “democracy”, “governance” or “freedom” often serve as catch-all labels. Composite indicators are based on a host of data that measure quite diverse sub-categories such as the “rule of law”, “elections”, “political stability”, “accountability” et cetera. Instead of speaking of “democracy measurement”, it would make more sense to speak of indicators for elections, the rule of law or political stability.

Validity and reliability pose challenges. For instance, indicators often cannot be compared over time. The reason is that different methods are used at different times to construct them. For similar reasons, cross-country comparisons often lack robustness: different kinds of sources and experts were consulted. In addition, ratings are often distorted due to inconsistent and complicated aggregation of data. Critics regularly point out biases of systematic, methodological, cultural or ideological nature.

Methods have nonetheless improved considerably in recent years. The internet, in particular, has led to greater transparency and easier application of indicators, even though difficulties remain. Today, all prominent index providers make disaggregate data available online. They also disclose information on authors, definitions, methods and sources.

Assessing freedom

In development cooperation, even experienced users tend to limit their attention to only a few governance and democracy indicators, and for good reason. Prominence has served to improve the quality of some widely-acknowledged indicators because these indicators are exposed to broad and informal peer review. Such public debate has boosted transparency, accessibility and coherence. Competition between index providers also serves as an incentive to compile indices annually.

One of the frequently quoted sources is the annual “Freedom in the World” survey. Since 1972, the US-based Freedom House, a not-for-profit organisation, has been conducting this annual rating. Today it covers 194 sovereign states and territories, attributing numerical ratings and different classifications of freedom (free, partly free, not free). The freedom ratings are divided into two indices for political rights and civil rights,respectively. Countries are rated on scales ranging from 1 (best) to 7 (worst). Many people use them as benchmarks for democracy and governance too. Raymond Gastil was the scholar who contributed most to getting this annual exercise started. From the outset, he focused more on tangible government action and actual operations of state authorities than on legal and institutional arrangements.

Apart from the freedom indices, Freedom House uses another classification to assess electoral democracies. This approach shows that a country can be quite unfree even if it is a democracy in formal terms.

Freedom House has repeatedly been accused of cultural bias. The organisation assigns experts to deliver ratings, and their judgments were not transparent in the past. Criticism has subsided to a somewhat limited extent in recent years, and Freedom House now publishes data for sub-categories regularly.

Transparency International and the World Bank

Since 1995, the non-governmental network Transparency International (TI) has been publishing its annual Corruption Perceptions Index (CPI). This index is popular with the media. It has no labelling problem. The CPI’s very name spells out precisely what it is measuring: subjective perceptions of corruption. This is often misunderstood, nonetheless. The CPI compiles surveys of domestic and foreign experts on how they see corruption in a given country and then translates the results into a trans-regional scale from 0 (absolutely corrupt) to 10 (totally clean).

The reliability of the country ratings is made evident through the definition of confidence intervals and the publication of the number of sources used. However, the CPI is constantly changing the number of surveys considered and the number of countries assessed. It is also keeps adjusting its methodology. Its ratings are therefore of little use for comparisons over time.

The World Bank Institute (WBI) has been publishing the Worldwide Governance Indicators since 1996, initially every two years and annually since 2002. Currently, 212 countries are covered. These indicators have become very important in the context of development cooperation, reflecting the emphasis put on good governance and related conditionalities. They define what is considered state of the art in regard to governance indicators.

The WBI conceptually disaggregates “governance” into six dimensions:
– voice and accountability,
– political stability and absence of violence/terrorism,
– government effectiveness,
– regulatory quality,
– rule of law and
– control of corruption.

As in the case of the CPI, trans-regional indicators are derived from various surveys. The WBI relies on 33 data sources from 30 organisations. It has resisted the temptation to aggregate all six dimensions into an overall “governance” index and only publishes results for the six individual dimensions. Naturally, the results of ratings will always depend on the choice of sources and the methods of aggregation.

The WBI relies on many studies that are designed for the private sector and thus emphasise the view of businesspeople. The CPI is equally business-biased. Over time and space, the WBI indicators can be used for comparisons, the advantage being that limitations become apparent to the user through the overlap of confidence intervals.

Rules of thumb

The increasing relevance of these measurements in development leads to misunderstandings that concern the scope and performance of democracy and governance indicators.

A few rules of thumb could serve to mitigate various weaknesses and help to get some well-founded snapshots of the situation in a country. First of all, the conceptual arrangement and the underlying definitions of the data deserve more attention. More than an indicator’s name, this kind of information provides clues for assessing its content, objectives and tendencies.

As one moves towards disaggregation of amalgamated indicators, one begins to understand underlying issues. Inter-temporal comparisons should always be taken with a grain of salt. To assess the quality of cross-country comparisons, moreover, it is important to know the number and nature of the sources included.

Disagreement between various indicators that relate to one and the same question can provide important insights. It makes sense to consider not only how various approaches correlate, but also to take account of measurements that were done with different methods in different places.

Latin America, for example

There are three trends that help to provide alternatives and additional information to prominent indicators:
– First of all, there are the perception-based studies which rely on household surveys and thus reflect the subjective views of citizens on a specific subject. Data of this kind are used for the World Bank’s governance indicators and TI’s CPI, but only to a limited extent. Instead of drawing solely on international indicators, it is worth looking at the regional surveys done by Latinobarómetro or the Latin American Public Opinion Project (LAPOP). The Latinobarómetro is a part of the Global Barometer Survey Network. It assigns local pollsters with more than 20,000 interviews in 18 Latin American countries every year, generating information on people’s satisfaction with their political and economic system. The results of these surveys can differ considerably from current expert opinion.
– Second, there is the qualitative approach taken by IDEA, the International Institute for Democracy and Electoral Assistance, and Democratic Audit, the British research institute. This approach makes it possible to assess the state of democracy in a country thoroughly along vertical and horizontal lines, without claims of temporal or cross-national comparability. Instead of coming up with quantified numbers, local teams of experts use a pre-defined framework to assess levels of democracy in qualitative terms.
– Finally, many smaller initiatives make independent efforts to measure the quality of democracy and governance in their respective country or region. Examples from Latin America include the Índice de Desarrollo Democrático, the Índice de Participación Ciudadana and the Electoral Democracy Index. It is true that some of the problems mentioned above arise again – concerning, for instance, the digitised availability of data, the transparency of methods or their stringent application over time. Nonetheless, these studies provide valuable data and can complement more prominent indices.

Most important, however, local initiatives are in line with a key concept in development cooperation: they lead to greater local ownership in respect to the measurement of governance and democracy. German development cooperation is aware of this potential. Since 2003, InWEnt has been running an annual workshop on measuring democracy, governance and human rights, teaching people from developing countries about the options for assessing political and legal systems.

Informed use

Democracy and governance indicators have led to more information and standardisation. At the same time, critical questions have been raised concerning the use of rankings and ratings and their comparability across countries and over time. Governance and democracy indicators should be treated for what they are: powerful tools providing snapshots of a country’s situation through expert opinion.

In all cases, the measurements take account of issues from different angles. Cultural bias, methodological imperfection and the dominance of some approaches all play a role, though these limitations should not be overestimated. Problems arise whenever underlying definitions are not made transparent, when methods are not coherent and when users don’t refer to additional information.

One must always bear in mind that every indicator is just an instrument, one tool among others, none of which can claim exclusive validity. Users should consider what additional options they have to gain information on the topic of investigation. If we simply rely on indicators, we unwittingly accept to stay ignorant of all things that defy quantitative measurement.