The evaluation found that PBOs remained a relevant and useful instrument for AfDB and its clients, although they were challenging to design and manage effectively. The evaluation found the relevance of the PBOs in AfDB’s portfolio to be broadly satisfactory, based on their programming and design and their broad adherence to AfDB’s own policy and guidelines and to international good practice. With regard to the achievement of reform objectives, the overall picture was also satisfactory. However, it was much harder to find evidence of AfDB’s influence on reform direction and speed. Regarding sustainability, even in the presence of strong ownership, concerns about the institutional and financial dimensions of sustainability meant that the overall outlook for sustainability in the sectors examined was unsatisfactory.
Despite having deployed UA7.2 billion as PBOs 2012–2017, AfDB had failed to invest in its own institutional infrastructure to obtain maximum value for money from the instrument. As was well reflected in the 2012 policy, PBOs were expected to form part of a “package of support” in order to ensure that they influenced and supported reform agendas, while also providing important funding. This package included analytical work to inform technical input, policy dialogue and capacity support. In practice, while there was some variation across countries, overall AfDB had underperformed when it came to policy dialogue, despite its strong position as a trusted partner. This was partly due to its institutional arrangements; a lack of clarity about who was responsible for policy dialogue; the structure of how the dialogue should be conducted, reported, and coordinated; and a lack of investment in human resources to conduct it. In addition, AfDB had underperformed in providing timely and adequate capacity support and specialized technical advice, partly due to the limited menu of instruments available to do so. These shortcomings had implications for how well AfDB was able to influence or add value to country reform paths.
Programming issues. A range of programming issues were examined and, while the overall picture was assessed to be broadly satisfactory, the evaluation identified areas that could be strengthened. First, for the large majority of the PBOs reviewed (94%, excluding CRBS), their use was envisaged in either the relevant country strategy paper (CSP) or the midterm review (MTR), in line with the policy. However, in the majority of cases, the assessment against the eligibility criteria was made for the first time during the PBO preparation phase. In terms of the type of PBO, the justification for the type chosen could have been be stronger, especially when the PBO did not use the recommended programmatic approach. Second, in approximately two-thirds of the operations reviewed, the analytical underpinnings used were clearly listed and relatively complete. However, exactly how this work informed or underpinned the design of the operation was not clear. Third, while risk assessment was assessed as satisfactory in two-thirds of the operations reviewed, reputational risk was rarely explicitly considered. The risk mitigation measures, such as future capacity support to address current risks, were generally not convincing within the timeframe of a PBO.
Alignment with country and AfDB priorities. This was assessed positively on the basis of the document review and in terms of stakeholder perceptions. All the PBOs could be mapped to at least one of the High 5s or supported crucial governance issues which cut across them. AfDB had also succeeded in expanding use of the instrument to support sector reforms in addition to economic and financial governance. Nearly 80% of survey respondents had positive views on the alignment of PBOs with country policy frameworks.
Coordination. There were many good examples of how AfDB had coordinated with other development partners, notably during the identification and appraisal periods. AfDB staff had taken coordination seriously and had invested in upfront work with other development partners. However, the in-depth assessment illustrated how difficult AfDB had found it to sustain these initial high levels of coordination throughout the implementation phase. Moreover, following the adoption of the G20 Principles for Effective Coordination between the IMF and MDBs on Policy-Based Lending in 2017, in countries facing macroeconomic vulnerability MDBs needed to align behind the IMF.
Designing PBOs for results. The overall picture was satisfactory, although some shortcomings were identified. Although two-thirds of the PBO appraisal reports examined stated there was an important role to be played by complementary inputs, only a handful explained how this was to be achieved. All PBO results frameworks defined baselines, targets and means of verification, and integrated prior actions and triggers. However, over a third were less than satisfactory because of: (i) weaknesses in presenting a convincing results chain; (ii) high proportions of process- and action-based indicators; and (iii) a lack of realism, particularly for single-year operations. The use of conditions was suitably selective; in programmatic operations these linked from one phase to the next in order to plot a medium-term path, and they were linked to broader dialogue frameworks. However, weaknesses were noted when the number of prior actions was high, opportunities for identifying triggers were missed, or the level of ambition for prior actions was not appropriate.
Gender and environment. The evaluation found that AfDB had missed valuable opportunities provided by the PBOs to address gender equality and environmental reform issues at the policy level. Just over a third of the PBO project appraisal reports (PARs) that were assessed included gender-related indicators and 7% included environmental or climate-sensitive indicators. The opportunity to push gender equality and environmental concerns varied according to the type of PBO. However, particularly in the energy sector, PBOs can provide valuable opportunities to shift national policies in support of AfDB’s ambitions of inclusive and green growth.
Efficiency. Broadly speaking, PBOs were broadly disbursed and implemented in a timely way, although some receiving countries said that disbursement was unpredictable. In line with expectations for the PBO instrument, the evaluation found that AfDB had disbursed the funds fully and, compared with investment projects, quickly. In addition, implementation progress was very rarely identified as a cause for concern. Nine of the 10 in-depth assessments were efficient in terms of transaction costs and the time taken to disburse the funds. However, perceptions of timeliness and transaction costs varied among both staff and RMC officials.
Technical assistance. Perceptions of the efficiency and transaction costs of technical assistance or institutional support provided to support PBOs was negative. Such support, when it was provided, was slow and tended to arrive toward the end rather than beginning of a PBO series. This was partly because capacity support tended to be designed in parallel with PBOs rather than in advance, and partly because of the limited set of instruments AfDB had available to provide small items of technical assistance, all of which operated like full projects rather than as rapidly deployable expertise.
Policy dialogue. AfDB did not use policy dialogue sufficiently or make best use of its “African Voice” to ensure PBO results. This finding is not dissimilar to that of the 2011 evaluation which described AfDB as “punching below its weight” when it came to policy dialogue. Only three of the 10 in-depth assessments had satisfactory frameworks for policy dialogue in the targeted sectors. The deficiencies that emerged in relation to policy dialogue can be broadly categorized as: (i) lack of clarity over who is leading and responsible for policy dialogue, especially after approval;
(ii) limited capacity to engage in in-depth technical dialogue in some areas; (iii) lack of structured planning or reporting for policy dialogue efforts, including through AfDB’s normal supervision channels; and (iv) lack of a medium-term strategy to capitalize on doors that may be opened by a PBO after formal completion. In the survey, fewer than a third of respondents were clearly positive when asked about the extent to which AfDB mobilizes appropriate resources for policy dialogue. In the 10 countries investigated, only five had a satisfactory framework for policy dialogue.
Policy guidance. The existence of the 2012 policy helped AfDB to improve its approach to PBOs and to make it more consistent; however, there were areas where implementation was wanting. The policy provided clarity on the authorizing environment and on a range of important issues, including the type of instrument, when it should be employed, on what basis and with what objectives. The policy was broadly aligned with good practices. Although it clearly set out activities or changes that needed to take place in order to facilitate implementation, not all aspects of implementation have gone as planned. For example, there have been delays in producing the supporting guidelines, a glaring lack of training, and unfinished business in ensuring an enhanced role for country offices. The guidelines, which are described as a living document, have not been updated and there has been no additional guidance on new reform areas, such as energy. The guidelines have not been following the adoption of the G20 principles. The survey and focus groups both also revealed staff demand for more guidance in areas such as policy dialogue, working in post-conflict contexts, and results measurement.
Institutional arrangements. Some of AfDB’s practices were out of line with both its own policy and the practices at the World Bank and the European Union. First, PBO design and management remained somewhat centralized and led by either the Governance and Public Financial Management Coordination Office (ECGF) or by sector departments. The extent to which country offices had taken up ownership varied significantly. Second, in practice, there was no centralized unit that provided specialized support to PBO teams. ECGF staff had been task-managing most of AfDB’s GBS. This lack of a central support unit, and the limited guidance and training provided to staff, was in stark contrast to the support available at the World Bank and the European Union.
Effectiveness. Overall, the assessment of PBO effectiveness, which focused on energy, PSE, and PFM, was broadly satisfactory. The evaluation highlighted areas where AfDB could focus attention in order to strengthen results and specifically how it could contribute to the direction and pace of reforms. Data from project completion reports and country strategy and program evaluations by the Independent Development Evaluation indicated that the satisfactory assessment was likely to reflect the effectiveness of the broader portfolio.
- All of the 10 cases achieved or partially achieved all, or the majority, of the reform actions listed in the results framework. In only one case were 25% of outputs considered to have been not achieved. In all other cases, at least 75% of the reforms were assessed to have been either fully achieved, partially achieved, or achieved with significant delay. With regard to the achievement ratios by sector covered, no clear pattern emerged. No sector performed notably better than any other. At an aggregate level, in seven of the 10 countries covered by the in-depth studies the overall effectiveness in terms of the achievement of the objectives stated in the results measurement framework (RMF) was considered satisfactory.
- Across the 2131
individually assessed components, two-thirds were assessed satisfactory in terms of the achievement of “landmark policy changes.” Within a PBO RMF, some actions can be much more important than others, although an RMF can include a large number of “tick-box” type items alongside more fundamental issues that have the potential to drive change and contribute to transformative outcomes. Such indicators were identified as “landmark policy changes.” Where these were not achieved, it is worth noting that the sample included a transition state (Comoros); two cases where the principal focus of the PBO was in another area (Tanzania and Seychelles); and one where the second part of the planned series never took place (Nigeria). - AfDB’s influence on the achievement of landmark policy changes was not always evident. In one third of the components, AfDB’s influence on either the direction or pace of reforms was evident and was usually achieved through analytical work, technical inputs, and policy dialogue. AfDB staff respondents to the survey supported the view that AfDB’s influence was limited, and strongest at the appraisal stage.
Sustainability. The sustainability of PBOs in energy, PSE and PFM was assessed to have been unsatisfactory, particularly in relation to the institutional and financial dimensions of sustainability. Only four of the 10 in-depth assessments had good prospects for sustainability. Almost all of the 10 had laid strong foundations for sustainability in terms of government ownership and leadership, which should be at the core of the decision to proceed with a PBO. However, the weak institutional and financial sustainability undermined the positive assessments in terms of ownership. While this trend was clear for energy, PSE and PFM, it cannot be generalized across the whole PBO portfolio.
Contextual, design, and management factors. The evaluation evidence from AfDB and from other institutions providing budget support in Africa indicates that the most frequently identified factors relating to country context were: (i) ownership, country capacity, having a “champion” for reforms; (ii) the country’s socioeconomic status; and (iii) country systems. The most frequent factors relating to the budget support mechanism were: (i) quality of design, programming, development partner coordination; and (ii) quality of monitoring and choice of indicators. The single most frequently cited enabling factor was the quality of design. In terms of hindering performance, the most frequently highlighted were: insufficient policy dialogue, high efficiency and transaction costs, poor choice of indicators, weak monitoring, and poor predictability.
The most significant factors associated with achievement of landmark policy changes,32
stronger even than the country’s socioeconomic status, were: programming, design, and efficiency factors; technical assistance; the operation being part of a series; and the existence of a country office.
- 31
Separate assessments were made for each of the relevant sectors: PFM, energy, and PSE. The total number of assessments was 21. Where the PBO was part of a series, the assessment was for all parts of the series completed or well underway. - 32
Budgetary or institutional changes of substance and influence targeted by PBOs within the set of intermediate outcomes (induced outputs) identified in the theory of change.