A generic theory of change for AfDB PBOs was constructed based on AfDB documents, consultations with internal stakeholders, and the methodology endorsed by the Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD-DAC) for evaluating budget support operations. The theory of change helped to refine the evaluation questions. Individual theories of change were also developed for each of the 10 in-depth country studies based on the generic version.
The evaluation had seven components and each used different sources of data and analytical techniques. In addition, the evaluation was based on a thorough inception phase which included two staff focus groups and interviews with 42 internal stakeholders, including eight executive directors and 12 members of senior management.
For the 10 in-depth studies, a cluster approach was used. This allowed strong evidence to be collected within specific areas but limited the extent to which the findings could be generalized across the full portfolio. The two focus areas of energy and the private sector enabling environment (PSE) were identified early in the design stage following analysis of: (i) the recent use of PBO funds, (ii) the availability of evidence, and (iii) the pertinence of these areas to future directions for AfDB, particularly in support of the High 5s. In addition to evaluating these two focus areas, the evaluation also examined the PFM components of the PBOs (present in nine of the 10 countries). It is important to note that, for the two general budget support (GBS) cases that included a broad range of targeted sectors in addition to energy, PSE, and PFM, those other sectors were not the focus of in-depth analysis, although the delivery of all reforms was assessed.
Within these two focus areas of energy and the private sector enabling environment, cases were selected according to the following considerations.
- Evaluability The sample included countries with PBOs that were at a reasonably mature stage of implementation (least 12 months since approval) so some influence could be expected on intermediate outcomes (known as “induced outputs,” see Table 2). All 2017 approvals were therefore excluded.
- Contemporary relevance The sample covered countries with relatively recent PBOs, whose design and implementation should reflect the 2012 policy, and where the process of implementation could still be recalled by those interviewed. This meant most of the cases came from the 2014–2016 period.
- Diversity in types of PBOs In selecting the cases, the goal was to include examples of SBS, GBS, single-standing operations, programmatic operations, and programmatic tranching.
- Diversity in country contexts The sample covered: (i) MICs, LICs, and transition countries; (ii) countries in at least four of the five sub-regions in which AfDB operates; and (iii) anglophone, francophone, and lusophone countries.
- Diversity in size of PBOs The sample included some of the largest and most important PBOs, intermediate PBOs, and small PBOs.
Ten countries and 16 operations were covered by the in-depth studies (Table 2.1). Collectively, they accounted for UA2,155,040 in approvals and 36% of PBO approvals by amount in 2012–2016. The assessments covered energy, PSE, and PFM29
. However, the sample was not designed to be generalizable across the full portfolio. In particular, PBOs with a focus on social sectors—which have also been an important part of the portfolio and are managed by a different department—were not covered.
Table 2.1: Program-Based Operations Covered by the In-Depth Assessments
|
Country |
PBO Operations |
Approval Date |
PBO Type |
Disbursement |
Net Loan Amount (UA million) |
|---|---|---|---|---|---|
|
Energy Cluster |
|||||
|
Angola MIC Lusophone Southern Africa |
Power Sector Reform Support Programme |
2014 |
SBS |
100% |
705 |
|
Comoros Transition Francophone East Africa |
Energy Sector Support Programme |
2014 |
SBS |
100% |
4 |
|
Energy Sector Reform and Financial Governance support Programme |
2012 |
GBS |
100% |
2 |
|
|
Burkina Faso LIC Francophone West Africa |
Energy Sector Budget Support Programme |
2015 |
SBS |
100% |
20 |
|
Nigeria MIC Anglophone West Africa |
Economic Governance, Diversification and Competitiveness Support Programme |
2016 |
GBS |
100% |
445.6 |
|
Tanzania LIC Anglophone East Africa |
Power Sector Reform and Governance Support program |
2016 |
SBS |
100% |
37.4 |
|
Power Sector Reform and Governance Support Program |
2015 |
SBS |
100% |
35.5 |
|
|
Private Sector Environment Cluster |
|||||
|
Egypt MIC Arabic and Anglophone North Africa |
Economic Governance and Energy Support Program Phase II |
2016 |
GBS |
100% |
371.3 |
|
Economic Governance and Energy Support Program Phase I |
2015 |
GBS |
100% |
371.3 |
|
|
Mali Transition Francophone West Africa |
Programme d'appui aux réformes de la gouvernance économique Phase II |
2016 |
GBS |
0% |
23.2 |
|
Programme d'appui aux réformes de la gouvernance économique Phase I |
2015 |
GBS |
100% |
15 |
|
|
Morocco MIC Francophone North Africa |
Morocco Economic Competitiveness Support Programme |
2015 |
GBS |
100% |
83.5 |
|
Ghana LIC Anglophone and Francophone West Africa |
Public Financial Management and Private Sector Competitiveness Support Programme Phase II |
2016 |
GBS |
100% |
35 |
|
Public Financial Management and Private Sector Competitiveness Support Programme Phase I |
2015 |
GBS |
100% |
40 |
|
|
Seychelles HIC Anglophone East Africa |
Inclusive Private Sector Development and Competitiveness Programme Phase II |
2015 |
GBS |
100% |
7.4 |
|
Inclusive Private Sector Development and Competitiveness Programme Phase I |
2013 |
GBS |
100% |
14.9 |
|
GBS = general budget support, HIC = high-income country, LIC = low-income country, MIC = middle-income country, PBO = program-based operations, SBS = sector budget support.
Source: AfDB
he evaluation was subject to a number of limitations. Each of these was taken into account in how the evaluation was designed and reported.
First, at the design stage, the evaluation team explicitly limited the extent to which the overarching question (ii), on results, would be addressed.30
This limitation related both to how far up the results chain the evaluation could go in assessing AfDB’s contribution to results. Additionally, the decision to focus primary data collection on performance within the two sectors and PFM covered by the two clusters of in-depth assessments allowed to give strong internal validity. Other data were used to establish whether the observed patterns had validity beyond those sectors.
Second, secondary data were not always available. For example, in some cases project completion reports, validations, and implementation progress and results reports (IPRs) were not available. In addition, AfDB does not systematically record its policy dialogue with countries. These constraints were mitigated by using other data sources where possible.
Third, given resource constraints, the balance between depth and breadth was based on stakeholder information needs. The 10 in-depth cases were chosen to maximize learning in the specific areas of energy and PSE, nine of the 10 cases also included PFM. The survey sought to bring in broader staff and RMC views, as did focus groups for staff. The project portfolio documentation review used a representative stratified sample. It did not include an assessment of the quality of analytical work.
Fourth, recent cases (from early 2018) were included to ensure that the evidence was contemporary, which also helped with the availability of informed staff and stakeholders. However, since many reforms are medium-term in nature, this meant that fewer results were available.
Fifth, to understand how PBOs had contributed to results, the in-depth assessments followed a uniform methodology designed to reflect the specificity of the PBO instrument and to maximize the learning potential for AfDB. Four different assessments were conducted (Table 2) to arrive at the overall assessment. The overall assessment was satisfactory, and the weakest area was AfDB’s contribution to the direction or the pace of achieving landmark policy changes. In order to mitigate risks when it came to evaluating the results of PBOs, the evaluation compared the results with those of other internal and external evaluations of AfDB operations to validate them.
Table 2.2: Assessing the Effectiveness of Program-Based Operations and Their Contribution to Landmark Policy Changes
| Country’s Achievement of RMF Indicators |
AfDB Contribution to Landmark Policy Changes a |
| Induced output bexecution ratio |
Achievement of landmark changes c |
| Overall assessment of effectiveness by country, including induced outputs and final outcomes |
Evidence of PBO influence on landmark changes |
AfDB = African Development Bank, PBO = program-based operations, RMF = results measurement framework.
Defined in Box 1.
The term induced outputs is aligned with international methodological standards on evaluating PBOs developed by OECD. However, induced results may also be considered to be an intermediate outcome. The theory of change in Annex 2 of the evaluation report outlines the kind of results anticipated at this level.
Landmark policy changes constitute (i) changes introduced as a result of decisions made at senior levels of government, and (ii) substantive changes, with a clear link to a desired final outcome. Thus, the mere adoption of a plan of action for reform would not be a landmark policy change, but the implementation of legislative or regulatory reforms as a result of that plan would constitute a landmark policy change.