Relating Outputs, Outcomes and Impact in the Evaluation of International Development Projects

Abstract

Background: The recent controversies on the pages of the JMDE regarding UNDP evaluations, as well as the DAC criteria, are discussed in the light of two UNDP/GEF evaluations in Latin America for which the author was primarily responsible.

Purpose: The author defends the utilization of all five evaluation criteria of the Development Assistance Committee of the OECD (the DAC criteria), and their integration via Theories of Change.

Setting: The pine/oak forest of Honduras and the mangrove regions along the coast of Brazil.

Intervention: Two projects of the Global Environment Facility (GEF) and the United Nations Development Programme (UNDP), with the respective governments.

Research Design: Mid-term evaluations with site visits and interviews.

Data Collection and Analysis: In the Brazilian case, longitudinal analysis of available data was conducted and related to the findings of the interviews and observation, as well as published reports and studies. A Theory of Change (action model) of the Honduran project was structured and graphically portrayed based on desk review of the project document and other documentation, adapted following initial interviews, and field tested.

Findings: In Brazil, preliminary evidence derived in part from Management Effectiveness Tracking Tool (METT) questions with reference to results indicates a relative lack of project effectiveness and of preliminary signs of impact. The state of Pará offers one possible exception, related in part to the fact that a certain momentum had already been built up in that state prior to project start-up and in part to the early adoption through the project of a new method of transportation of the fiddler crab (Ucides cordatus), in baskets covered with wet sponges rather than in sacks. The Honduran project, with the strong support of the national government and the UNDP field office as well as GEF, successfully adapted the provisions of the project document to pursue a more community- and community organization-centered approach, rather than relying on the outside consultants originally specified. Project-supported gathering of impact data raised environmental awareness, strengthened the local university and established a baseline for future ex-post impact evaluation.

Keywords: Theory of Change; DAC criteria; capacity development; displacement of goals; UNDP/GEF.