Methods

Process of an ex-post evaluation

The ex-post evaluation takes place approximately three to five years after the completion of the projects. The evaluation assesses the entire project cycle, from planning and implementation to operational experience, in terms of the impacts achieved and their sustainability. Ideally, the question is answered as to which net effects can be attributed causally to a FC project. The ex-post evaluations use the Rapid Appraisal methodology, which combines qualitative and quantitative approaches and triangulates findings. To this end, the use of data is essential, and insights are methodically hardened by using modern technology, analysis tools and software. The OECD-DAC evaluation criteria provide the framework for the evaluation of projects. These are considered per criterion and finally assessed using a grade scale of 1 to 6.

Graphic: Process of an ex-post evaluation

The process is carried out in 5 steps:
1. Preparation/Conception (research, analysis of available information, evaluation concept and methodology, mission planning and coordination the the partners)
2. Mission on site or remote (discussions with project partner, target groups and stakeholders, field visits, collection of primary and secondary data, discussion of preliminary findings with the project partners)
3. Analysis (analysis and triangulation of data and findings)
4. Evaluation report (final assessment according to OECD-DAC-Criteria, quality assurance through internal peer review)
5. Institutional Learning (communication of reports internally and externally (BMZ, partners and public), systematic processing of the findings)

In attributing overarching developmental effects (impacts), the ex-post evaluation follows the approach of a contribution analysis based on plausibility considerations and a careful analysis of documents, data, facts and impressions. The evaluators use the available files and reports to examine the project, its content and background. They conduct interviews on site or remotely, analyze data and statistics, evaluate information from other donors and stakeholders and look for scientific evidence to prove the results chain.

Causes of any contradictory information are investigated, attempts are made to eliminate these contradictions and the evaluation is based on statements that are confirmed by several sources of information, if possible (triangulation). The analysis of the impacts is based on assumed impact relationships that are already developed during the appraisal and, if necessary, updated and documented in an impact matrix during the ex-post evaluation. The impact logic/impact chain (theory of change) is also presented in the report in a way that is comprehensible to outsiders and existing risks and assumptions are pointed out. The evaluation report sets out the influencing factors identified for the impacts and why the project under review presumably made which contribution. The results are weighed up against comparable projects. Where possible, the impact logic is underpinned with indicators at the level of outcomes and impacts.

The diagram illustrates the sources of information that are selected by the evaluator on a case-by-case basis.

Graphic: diagram illustrates the sources of information

Various sources of information are used for the assessment:

Analysis of available project data (including: appraisal, progress and final inspection reports, KfW project documentation, reporting by the project executing agency, consulting reports, previous evaluations and evaluations by other donors)

Collection of project-specific data (including: Field visits, interviews with stakeholders and target group, focus group discussions, monitoring data from the project executing organisation, photos)

Analysis of project-relevant primary data (including: Databases, open source/time series, GPS data, satellite images/nightlight data, mass media and big data)

Use of project-relevant secondary data (including: Evidence from rigorous impact evaluations, systematic reviews, context, country and sector analyses, surveys and census data, scientific literature)