“Interactive results”

Joseph Stiglitz, the Nobel laureate and former chief economist of the World Bank, once defined development as a “transformation of society”. If that is the case, can we really expect to accurately assess the contribution of any single donor agency?
Development is indeed the process of a society transforming itself. Anybody who wants to support that process will have to work through the decision makers in that country. Therefore, recent development discourse is right to stress local and national ownership of the decisions. Naturally, that has implications for evaluation. While our task is to unravel dynamic processes and to understand causes and effects, we’ve learnt just how difficult it is to attribute success to any single actor. Rather than of attribution, I would speak of contribution of individual actors.

To come up with quantified data on the performance of schools or the usefulness of a road is not difficult. You can assess enrolment and the level of literacy of those who leave the school. You can count vehicles and measure how long it takes to get from place A to place B. Does a successful project mean developmental progress?
Not always. We know that project results have improved systematically over the past 20 years. However, such success does not automatically translate into societal progress. The success of an education programme depends not just on school enrollment resulting from a project, but also on learning outcomes. More generally, four out of five projects achieved the stated objectives. Yet, in about a third of the cases, the country programmes perform at a lower level than the countries’ project portfolios. There are several plausible reasons. For instance, a good primary school established with donor support may be a worthy project, but its contribution to improved education depends on systemic improvements and whether its impact is scaled up across the country. Or, an additional road might be assessed to be a good project in itself, but if transport problems are not the binding constraint to economic growth, that project might not reflect progress in a key area.

If project success does not necessarily mean that funding should be continued, does failure at least indicate that support should be discontinued?
No, that conclusion would be too simplistic too. There are risks as well as failures the international community cannot simply walk away from. Consider the so-called fragile states: they need support, no doubt, even though a good deal of the aid they may have received in the past did not lead to the desired results. The environment is another area where development programmes have not been as successful as one would have hoped, as environmental problems are getting worse in spite of development agencies’ activities. But in both these instances, if there are turnaround situations or learning of lessons, continued aid will be necessary, even if it did not deliver the necessary results in the past.

As a global and novel phenomenon, climate change and its consequences are particularly difficult to assess. But other issues are of a national nature and are hardly quantifiable either – governance for instance.
Governance is crucial for explaining development, but very tough to measure in the aggregate. What we can do is to break the issue down into different areas. For example, public-sector management consists of various components. Of those components, financial management may be relatively easy to assess and to reform. It is possible to track the flow of funds, after all. Progress in civil-service reform or in the fight against corruption is much more difficult to measure as well as to reform. Especially in these more complex aspects, it may not be possible to rely on objective indicators alone but also complement them with perceptions of those directly affected.

Does that imply that it is better to focus on financial management and forget about civil-service reform or anti-corruption for the time being?
No, that would be a false choice to make. But it does make sense to look for approaches where you can work with quantifiable data and with opportunities even for partial reforms. For instance, if energy pricing is reformed in a country, that is relatively easy to track. Pricing and market reforms will also reduce room for corruption and have an impact on how the civil service works. The idea in these instances is to tackle difficult problems indirectly.

If you look at hard figures for financial management, however, you can tell whether the Bank has directly boosted development or not?
Yes, it has to a considerable extent. For instance, the World Bank has given loans to two thirds of the countries in Eastern Europe and Central Asia in the past decade, and governance scores, including for financial management, have improved in some 90% of those countries. That is a positive sign, no doubt. On the other hand, governance scores also improved in 86 % of the countries that did not get World Bank loans. So perhaps the links of some of the countries to Europe and the European Union are important in explaining these improvements. Even where we have data, we will always need an intelligent analysis of the broader context. One cannot expect quantitative indicators to tell the whole story. And that is all the more so as development does not depend only on aid, but also on other factors like trade or security.

I imagine it must be even more difficult to sort out the impact of a single agency in cases where several donors are active.
Well, if one agency is really the dominant player in any given developing country, it would be a safer bet to assign it the credit or the blame for most of the impact of aid. But that is less and less the case. In the 1960s, 12 agencies would, on average, be involved in a single country. The number has tripled since, and the financial sums have increased too. Moreover, we are now seeing much more private-sector investment.

How do governments and civil-society organisations in developing countries assess donor-agency performance? If we stress national ownership, we should take their views into account.
Yes, indeed, and just like donors they are showing a growing interest in joint evaluations. In Bangladesh, for instance, we are making the effort to jointly evaluate among the Asian Development Bank, Japan, the UK and the World Bank. There are coordination costs in such attempts, but it would still make sense to work jointly, because it makes it more possible to look at interplay of different interventions. Bangladesh’s development experience makes this point about interactive results. For example, an evaluation showed that better secondary schooling for girls coupled with the impact of nutritional and family planning efforts can multiply benefits. Of course it helps our partners to look at such data in context, it gives everyone involved a clearer understanding of what is going on. In that sense, it is encouraging that joint evaluations have shown that cooperation of the World Bank with German development agencies give country partners a richer set of options than each agency on its own. But that does not mean that we could precisely attribute success to any of the participants.

In my experience, evaluators tend to be technocrats who diligently fill in their forms. PhD students, on the other hand, seem to be much more enthusiastic about their work, generating new hypotheses all the time. Could it be that more independent academic research would help us understand development challenges better than trying to evaluate agencies’ performance?
It seems to me that you are moving back to our opening remarks on Stiglitz seeing development to be about societies transforming themselves. That’s always a complex process, and to understand it will take passion for change – which I have seen in researchers as well as in evaluators. It simply is not enough to have a technocratic assessment of the issues if we want to understand what works in what circumstances, and why it works. And it helps to have enthusiasm that is firmly planted in experience, rigor, and a keen interest to encourage changes in policies and behaviors.