Digital tools serve evaluation purposes

What are your innovations in regard to evaluations?
The way we work keeps evolving continuously. We focus on:

• ex-post assessments of completed development projects,
• data-based impact measurement of ongoing projects and
• institutional learning.

The first task has been our core duty for more than 20 years, and the other two tasks have been dealt with sporadically. We are now focusing more on them.

What exactly is data-based impact measurement of ongoing projects?
Well, a good example is a water pipeline currently being built in Tanzania. Within a range of 24 kilometres, villages can be connected to it. If they are farther away, that is not feasible. This setting makes it possible to compare villages and measure impacts. Our evaluators interview people who get water from the pipeline as well as people who do not, allowing us to measure precisely how the new infrastructure is changing lives. We collect quantitative data related to water consumption, regarding the prevalence of diarrhoea, for example. It is challenging to implement this kind of rigorous impact evaluation, but it helps improve the project, so it is worthwhile. However, we can only afford to support a limited number of projects this way. In the next few years, we plan to evaluate 10 to 20 projects in real time.

Is digitalisation changing how you work?
Yes, it is the key to state-of-the-art evaluations. Consider geodata on afforestation programmes, for example. In cooperation with the French development bank Agence Française de Développement, we have launched MapMe, an open source project designed to build databases concerning conservation areas. Moreover, mobile telephony is making it easier to collect data in developing countries and transfer the information internationally. These things help us in developing countries. At the same time, digital innovations contribute to more institutional learning within KfW development bank. For example, we have coded and programmed an app that gives users interactive access to the full contents of all evaluation reports in the bank’s archives. Relevant knowledge can thus be found fast. You enter a few search terms and immediately get the evaluation results that you need – ranging from general lessons learnt to detailed analyses. This tool is extremely useful. Creating the app was a major effort and took a year, and from October on, all KfW staff members can use it.

Will the public get access too?
Yes, we plan to make that happen in the not too distant future, but we want to make sure everyone in the bank has full access first. Of course, we use digital technology for our public outreach. Our most recent comprehensive evaluation report, which documents the results of the past two years, is available in a print edition as well as in a digital version. The latter includes videos and interactive maps.

What is the comprehensive report’s core insight?
It includes a host of details, but the first, overarching message is our success rate. Translated into school grades ranging from 1, that is, very good, to 6, 86 % of KfW projects would have earned a 3, which is the German equivalent of the American C. This success rate does not mean that there aren’t any weak points. The average grade was 2.8, and it varies from sector to sector. The average grade for water projects was 3.1.

In my eyes, your success rate is neither surprising nor convincing. International development agencies – including the World Bank, the GIZ and the KfW – typically report success rates of about 80 %, and that looks dubious to observers.
I can only speak for KfW. We have been applying the international evaluation standards defined by the OECD-DAC for two decades, we give assignments to independent experts, we are diligent in regard to methods and we publish all our results. The way we work is transparent and the methods are state-of-the-art.

And yet there is scepticism. A colleague who has spent many years working for the World Bank and the GIZ once told me that success rates of around 80 % are systemically important. They are high enough to confirm an agency’s legitimacy, but low enough to admit some cases of failure.
I am a scholar, and we scientists welcome criticism and scepticism. That said, it does not make sense to cast general doubt on results. For debate to be constructive, you would have to point out what is supposedly wrong with our methods and explain why you do not trust conclusions. We are applying methods that are well-defined, universally recognised and transparent. If these methods yield a particular success rate, that rate cannot be allegedly wrong only because it coincides with the success rates of other institutions.

You need independence to do evaluation work. How does the KfW ensure that you are independent?
First of all, the head of the evaluation department is recruited from academia, and that was true of my predecessors as well. Second, I report directly to the board of the KfW Banking Group, so I’m not involved in the immediate operations of the Development Bank, the work of which I assess. Finally, my department determines what questions we find important and what projects we evaluate. Our results can be quite uncomfortable for the Development Bank, which is obviously the case when we find fault with a project. After all, someone is responsible for that project. However, we are not primarily focusing on success or failure. The tangible results of every single evaluation are what makes our work relevant.

Please give some examples.
Well, results show that supporting the financial sector works well when loans from KfW enable banks in partner countries to hand out credit to small and medium-sized enterprises. They also tell us that the more professional our partners’ operations are, the more effective this approach becomes. The obvious implication is that our partner banks themselves require advice and support in places where the financial sector has not fully developed yet. In a similar way, we see that the KfW’s policy-based lending is generally quite successful. KfW gives loans to government authorities so they can implement policies they have agreed with us. Our data show that performance improves the more intensive the exchange with partners, the more precisely goals are defined and the more accurately spelled-out the criteria are on which the payment of the next tranche hinges. Findings of this kind are of great practical relevance for the bank.

Links
KfW Development Bank: Evaluation Report 2019/2020.
https://www.kfw-entwicklungsbank.de/Bilder/Evaluierungsbericht-2021/Startseite/KfW-Evaluation-report_2019_2020.pdf

MapMe
https://mapme-initiative.org

Jochen Kluve is professor of economics at Berlin’s Humboldt University and has been leading the evaluation department of KfW Development Bank since October 2019.
jochen.kluve@kfw.de
https://www.kfw-entwicklungsbank.de/International-financing/KfW-Entwicklungsbank/