Measuring social protection gains: Monitor, evaluate, strengthen

In the past 200 years, economic growth bolstered by scientific and industrial developments has lifted more people out of poverty than one could have imagined in the 19th century. Reconstruction of data by Bourguignon and Morrison in 2002[i] shows that the proportion of world citizens living in extreme poverty has fallen from a whopping 84% in the 1820s to about 16% in the current decade.

 

Reduction in absolute poverty has accentuated global focus on inequalities and vulnerabilities faced by those left behind. Organized, state-led welfare systems have been around since the late 19th century, when Prussian-era Germany introduced state support programs under Bismarck’s reform agenda. In the past few decades, social protection systems have been adopted around the world in some form or another, to ensure certain minimum standards of living and dignity to all citizens.  

However, notwithstanding increasing amounts of public spending by governments, extreme poverty and deprivation are a long way from eradication. One reason being, despite a shift towards evidence-based development policy in recent times, many national programs remain a product of populist concerns, virtue signalling, or mere theorising (well-intentioned but untested), rather than being based on needs-assessed and evidence-based inputs. This is especially true for big-ticket, one-size-fits-all programs which are often inefficient and ineffective in terms of both costs incurred and results achieved.

This highlights the role that results-based program management, performance monitoring and impact evaluation – followed by regular course-corrections – must play in social programs funded by public expenditure.

Some countries provide successful examples of social programs being facilitated and strengthened by rigorous impact evaluation, such as Mexico’s PROGRESA (where the findings of a commissioned evaluation significantly influenced the decision to extend the programme nationwide in spite of initial opposition and budgetary constraints[ii]) and Colombia’s SINERGIA (M&E in the public sector has been written into the country’s constitution and several laws).

Monitoring & evaluation (M&E) provides policymakers with the ability to draw causal connections between policy priorities, the resourcing of policy objectives, the programmes designed to implement them, the outputs achieved, and their impact on communities[iii]. M&E also creates an evidence base for decisions regarding public resources which not only brings in transparency but also provides objective (rather than emotional or moral) grounds to engage with public/political opposition to such expenditure.

Performance monitoring is a continuous process of collecting and analyzing data for performance indicators, to compare how well an intervention or policy reform is being implemented against expected results. (Results Based Management Handbook, United Nations Development Group, 2011)

Evaluation is a systematic and objective examination of the relevance and fulfilment of objectives, efficiency, effectiveness, impact and sustainability of an ongoing or complete project, program or policy. (Evaluation Policy, United Nations Evaluations Group )

M&E is critical in the national social protection context where the presence of several actors can dilute the accountability of each; for instance, a policy may be formulated by the central government, based on which programs may be designed by the local/municipal government and implemented by field officers or NGOs. If this program shows poor results, it would be difficult to understand where the problem lies, or how it could be corrected, in the absence of proper data collection, measurement of indicators at various program phases, and evaluation of results.

As important as it is, performance measurement comes with its own set of challenges, especially in low-income countries. To name a few:

  • Availability of data is often a problem, and establishing a system (e.g. MIS) for capturing information can be a complex task, especially considering the technological requirements.
  • Financial and human resources would need to be allocated for creating the requisite capacity for data collection and analysis.
  • There would have to be a protocol of coordination between various wings/agencies to collect and share data.
  • Crucially, not all aspects of performance are quantifiable or easily measured. Developing suitable indicators for qualitative aspects, and collecting data on them through means such as interview responses, can be time-consuming, costly, unreliable or simply impossible if the scale is huge.

Identifying the appropriate indicators for performance measurement is a critical task. It requires sound formulation of the objectives the indicators are attempting to measure, and conceptual thinking around the performance-related model[iv]. Indicators can be of several kinds: stand-alone (e.g. number of beneficiaries reached, or average amount received per pensioner), benchmarks (e.g. over a period of time, or national or international comparisons), composite (individual sub-indicators ‘weighted’ by their relative importance), etc.

As with other aspects of program design, the specifics of the intervention (local context, incentive-disincentive structures of concerned stakeholders, etc.) are all-important for designing the M&E framework: what works in one case might not work in another similar situation. To give an example, using camera-feeds to monitor teachers’ attendance in rural Indian schools succeeded in improving student learning outcomes, but the same when applied to government doctors at rural clinics failed to achieve results. (Why?)

Moving to an implementation framework where policies and projects are subjected to rigorous M&E and impact assessment can also set the stage for governments to start looking at private-sector partnership models like Social Impact Bonds (SIBs) or Development Bonds – at present an idea still in nascent stages – for funding development projects. SIBs are financial contracts between private investors in a social project and the government/donor, where the latter's repayment to investors is contingent upon specified social outcomes being achieved, and financial returns to investors are commensurate with the level of success. While SIBs are not suitable for all social provision and do not expect to replace public services, 60+ pilots across 15 countries (mostly developed, with the exception of India) are exploring niche areas where private investor-led initiatives can bring quantifiable change. There are also other efforts to crowd-in private investment in the social sector, but these are currently focused on finding innovative solutions to development issues, rather than reaching transformative scale.

Coming back to the existing model of social programs, international donors consider M&E a key funding requisite, but donors typically fund only a few and certain kinds of programs – short-term, distinct, prototype projects, often dealing with infrastructure creation. Ultimately, it is national governments that are responsible for ensuring a social protection floor, that have a duty to ensure that their programs are effective and evidence-backed, have established processes to minimise leakages, and can incorporate lessons from previous implementation efforts.

No country has unlimited public resources; for developed as well as developing countries to increase their social protection spending, they must assure the cost & impact effectiveness of their programs.

 

This blog post is published as part of the Ambassador Series, which presents insights into social protection around the world from the viewpoint of our Ambassadors, a group of international online United Nations Volunteers who support the online knowledge exchange activities, networking and promotion of socialprotection.org.

[Banner image: toolkit.communitymonitoring.org]

References:

[i] Bourguignon, François and Christian Morrisson. 2002. “Inequality Among World Citizens: 1820-1992.” American Economic Review, 92(4): 727-744.

[ii] Barrientos, Armando and Juan M.Villa. 2013. “Evaluating antipoverty transfer programmes in Latin America and sub-Saharan Africa.” WIDER Working Paper No. 2013/009.

[iii] ESL. 2013. “End-of-Term Evaluation of the Joint Programme Support for Strengthening the Monitoring and Evaluation Systems in Malawi”.

[iv] Delorme, P. and O. Chatelain. 2011. “The role and use of performance measurement indicators.” Aid Delivery Methods Programme. Brussels: EC and EuropeAid.

Directorate-General for International Cooperation and Development. 2017. “Indicators to measure social protection performance: Implications for EC Programming.” Brussels: EC.

United Nations Development Group. 2011. “Results Based Management Handbook.”