Evaluations
Evaluations and reviews are important for learning and accountability of bilateral support initiatives. Larger bilateral support country projects need to be externally evaluated. This may include mid-term reviews if they are required by partners and donors. Findings and recommendations of all evaluations and mid-term reviews are presented below.
Term-end Evaluation of support to the SAI of South Sudan: NAC Strategic Change Project 2024 and IDI's Management Response
Term-end Evaluation of Support to the SAI of South Sudan: NAC Strategic Change Project 2020-2024
Oslo Economics has conducted a term-end evaluation of the Strategic Change Project (SCP) 2020–2024. The SCP is a partnership between IDI, OAG Kenya and AFROSAI-E aiming to help the National Audit Chamber of South Sudan (NAC) successfully implement its strategic plan.
Oslo Economics found that "the SCP has significantly contributed to capacity-building, increased audit outputs and progress across multiple focus areas in NAC South Sudan. Importantly, the SCP has helped NAC keep up with activities in a period of lacking financial and administrative autonomy, as well as absent salaries and funding. Oslo Economics concluded that the SCP has been both relevant, effective and efficient, since the support, with its flexible design and relatively low cost, has enabled the continuity of an institution that will be particularly important in a scenario where political conditions improve."
The Term-end Evaluation and IDI´s management response can be found here.
External Mid-term Evaluation of the Accelerated Peer-Support Partnership (PAP-APP) and IDI's Management Response
Mid-Term evaluation of the Accelerated Peer-Support Partnership - Phase 2: 2020 - 2024
Ernst & Young, Sweden conducted the mid-term evaluation of the Accelerated Peer-Support Partnership (PAP-APP) and concluded that "the programme design meets the requirement of achieving the programmes output and outcomes. The programme has increased peer-to-peer support to the participating SAIs. Thanks to the country projects the SAIs has developed working methods and tools that will assist the SAIs to fulfil their mandate and serve the citizens including the programme objectives.”
The mid-tern evaluation and IDI´s management response can be found here.
External Mid-term Review of IDI's Bilateral Support to SAI Madagascar and IDI's Management Response
Bilateral Support to SAI Madagascar 2020-2025
Ernst & Young, Sweden conducted the review and concluded that the project has contributed to improved performance of SAI Madagascar. The management of the SAI has shown ownership of the project and the everyday management of the project is functioning well.
The final review report and IDI´s management response to the recommendations can be found here.
Evaluation of Implementation of the GCP Tier 2 Strategy
Component 1 examined the nine SAIs selected for the Global Call for Proposals (GCP) Tier 2 , namely the Democratic Republic of the Congo (DRC), Madagascar, Guinea-Conakry, Togo and Niger (French-speaking, CREFIAF members), and Eritrea, Zimbabwe, Sierra Leone and The Gambia (English speaking, AFROSAI-E members). IDI, AFROSAI-E and CREFIAF have established a partnership to support the SAIs included in the GCP Tier 2.
The evaluation concluded that GCP Tier 2 was a relevant and effective response to kick-start capacity development support in SAIs in challenging environments, considered at risk of getting left behind. It could be replicated with two adjustments: (1) The process should aim at selecting 1-2 SAIs in each region and (2) the likelihood of SAI reform should be included among the criteria.
Additional recommendations for future GCP Tier 2 programmes included:
- emphasising with stakeholders the importance of SAI leadership in the reform process
- involving regional and sub-regional bodies from the beginning to improve communications
- using documentation from PAP-APP to give SAIs a better idea of what the reform process and the support means in practice, for future GCP Tier 2 programmes
Accelerated Peer-Support Partnership Programme (PAP-APP) Phase 1, 2018-2020
Based on relevant documents and interviews with representatives of all stakeholders in the February/March 2020 period, the evaluation concludes that PAP-APP was well designed and implemented, that SAIs responded above expectations, that the partnership with AFROSAI-E and CREFIAF on the whole worked well, and that most of the planned outcomes and outputs were likely to be achieved by the end of Phase 1.
Most SAIs in PAP-APP improved their strategic planning capacity and became more sustainable institutions compared with the situation before the Programme was initiated. The evaluation attributes these positive results to the high level of trust in IDI, AFROSAI-E and CREFIAF, as well as the approach of letting the SAIs lead their own needs and development with IDI and partner support and clear synergies with AFROSAI-E and CREFIAF training programmes. While there have been shortcomings in some respects, these are largely recognised and serve as lessons learnt.
Accelerated Peer-Support Partnership Programme (PAP-APP) Phase 1, 2018-2020
During the project period the NAC has been highly challenged. South Sudan is currently in a deep and longstanding economic, political, security and humanitarian crisis, and the Government of South Sudan (GOSS) has in many cases not met its commitments in the Revitalized Peace Agreement (R-ARCSS) of September 2018. Consequently, the NAC faces serious problems considering the difficult and challenging situation it is working in, and basic conditions for playing the role of a SAI are missing.
As a result, IDI classified this project as ‘lifeline support’, aiming to maintain the basic competency and skills of the SAI and its staff in the interim period till the situation stabilises and more long-term support could be provided.
Despite the label ‘lifeline support’, the evaluation found that the Project had an ambitious Results framework. Risks were underestimated in the project description, however, so many of the targets were not been fully achieved. That being said, even with a delay to many of the deliverables, the evaluation determined that the results are good considering the challenged situation of the NAC.
SAI South Sudan Strategic Change Project 2020 - 2025
Under the ongoing bilateral support arrangements with the NAC of South Sudan, the SAI is currently half-way through a five-year strategic change project. Full details of the project itself, the mid-term review and the management response to the review can be found here.
Synthesis of findings and lessons for implementation of IDI’s bilateral policy 2017-2020
Has IDI’s bilateral policy been applied as intended and implemented efficiently?
Has IDI’s bilateral policy been effective in contributing to SAIs enhancing their performance and capacity, relative to the resources available?
Is IDI’s bilateral policy in need of refinement?
The document includes a list of recommendations and a review of the bilateral policy's relevance, efficiency, effectiveness and sustainability.
IDI Management Response to Evaluations of Bilateral Support 2017-2020
The first was a broad evaluation of several initiatives focused on the period 2017-2020. This evaluation is broken down into four components covering GCP Tier 2, PAP-APP and support to SAI South Sudan, concluding with a sysnthesis of finds and lessons learnt. The second evaluation focused specificly on peer support to SAI Somalia.
Given the similarities of the evaluation objectives and findings, IDI prepared this consolidated response to both evaluations. The document reviews each recommendation from the evaluations, IDI's response to the recommendations, and, if required, proposed action.
Evaluation of IDI's Support to SAI Somalia
This evaluation report covers the project “Peer Support to the OAGS 2018-20” that provided bilateral assistance by IDI in partnership with AFROSAI-E to the Office of the Auditor General of Somalia (OAGS).
The OAGS Project was among the first projects under IDI’s Bilateral Policy adopted in March 2017, to ensure that the most challenged SAIs with substantial need for capacity development are assisted and are improving their performance. During project implementation, this support was broadened to also include the other strategic goals of the OAGS relating to communication, human resource management and IT infrastructure. This ensured a holistic approach to the development of the OAGS.
In summary, the Project’s objectives were mostly met and scored reasonably well on each of the six OECD-DAC evaluation criteria. Performance improvement was evidenced by OAGS ability to audit the Government’s annual budget execution report in a timely manner for the first time in 2019, and its willingness to report openly on challenging topics involving corruption and malpractice with regard to donor funds.