Measuring gender outcomes in social protection programmes: Why is it important? How best to do it?
The webinar, titled Measuring gender outcomes in social protection programmes: Why is it important? How best to do it?, took place on 22 March 2018. The event was organised by the Food and Agriculture Organisation of the United Nations (FAO) and the International Policy Centre for Inclusive Growth (IPC-IG).
The event highlighted how social protection programmes increasingly mainstream gender components in their design, however very few monitor and assess gender-related outcomes. With this gap in mind, it was discussed how gender results, including women’s empowerment, can be measured in social protection programmes.
The event brought together three experts: Elena Bardasi (Senior Economist, Independent Evaluation Group, the World Bank), Susana Martinez-Restrepo (Director of Research and Development, CoreWoman) and Tara Cookson (Director, Ladysmith and research fellow, Social Sciences and Humanities Research Council of Canada). Maja Gavrilovic (Social Protection Specialist, FAO) moderated the webinar.
The recording is available here and the presentation here.
Key messages
The World Bank’s Social Safety Nets and Gender Review
The webinar began with Elena Bardasi’s presentation of the key findings of the IEG World Bank’s study, ‘Social Safety Nets and Gender: Learning from Impact Evaluations and World Bank Projects’. This systematic review assessed the evidence from 145 impact evaluations on the effects of social safety nets (SSNs) interventions on gender-related outcomes, such as women’s bargaining power and decision-making, education outcomes of boys and girls, and maternal and child health.
The review also analysed the extent of gender integration in the World Bank’s portfolio of 112 World Bank supported investment projects (between 2003 and 2013), including conditional and unconditional cash transfers, non-contributory pensions, in-kind (food) transfers, and public work programmes.
The findings
Elena presented some of the report’s main findings regarding the attention to gender relevant impacts:
- 58% of studies assess gender-relevant impacts of safety nets programs;
- Only 45% of World Bank projects integrate gender elements into design;
- Only 39% of results frameworks include gender related indicators – which is a relatively low percentage;
- Rich and increasing evidence exists on the importance of unintended gender outcomes. Attention to gender impacts in World Bank’s projects is also increasing over time;
- Evidence is not evenly distributed across outcomes and interventions;
- The most frequently evaluated outcomes include education, health and employment (which are the projects’ intended outcomes), compared to less evaluated issues such as fertility, participation in representative bodies or domestic violence (which are unintended outcomes).
Challenges
- Discrepancies between the prevalence of SSNs and available evidence (conditional cash transfers/CCTs, especially in Latin America, have been widely impact evaluated, while public works programmes – one of the most frequent World Bank safety nets interventions – have received much lower attention);
- Difficulties in measuring some outcomes, such as empowerment. Some impact evaluations look at consumption decisions, other studies consider a broader set of indicators. This raises problems of interpretation and comparability;
- Limited evidence of impacts over a long period (most impact evaluations are carried out one or two years after the start of the programme);
- Limited integration of gender in SSNs World Bank projects;
- Limited use of sex-disaggregated indicators;
- Limited reporting.
Lessons learned
Interventions may have and very often have distributional impacts within the household even when this aspect is not explicitly recognised. Acknowledging these impacts is the first step to identify the project’s potential to address gender gaps or the existence of positive or negative unintended impacts.
The World Bank institutional environment stresses the importance of meaningful integration of gender in project design, implementation and evaluation. This involves the needs to identify gender gaps, take action (address those gaps in projects), and measure gender-disaggregated project results etc. Finally, Elena stressed the importance of informing impact evaluation frameworks with a robust theory of change and thinking of how to document unintended gender impacts.
Measuring women’s empowerment in South America
Susana then presented the new book, entitled ‘Measuring women’s Economic Empowerment: Lessons from South America’, which she co-edited with Laura Ramos-Jaimes. It features the evaluation of several Social Protection Programs in South America: Red Unidos Strategy in Colombia, Juntos in Perú among others. During the field research, research team realised that the data collection instruments to measure Women’s Economic Empowerment, were not working well when applied in the context of the lives of low-income women in South America.
With this in mind, she then presented five big ideas about measurement of women’s empowerment designed for South America:
- Empowerment is a process: It requires us to consider resources as pre-conditions (education, access to land), that in many cases are considered rather as proxies to economic empowerment. These resources, need to enable agency, negotiations and strategic decision making to act upon goals, which feed into well-being achievements. Very often we include in our measurements, decisions that are rather disempowering because they affect our freedom to act upon goals that could actually improve our wellbeing;
- Conceptualise Women’s Economic Empowerment (WEE) according to what is empowering in a given context: A decision that might be considered as strategic in South Asia (deciding over groceries) might not be strategic in the South American context, because it is a decision that is already traditionally taken by women anyways. These decisions and the household case and chores burden, instead affect the time they could use to do paid work;
- Integrating subjective dimensions of WEE is critical: However, this requires applying abstract concepts to real-life scenarios relevant to the lives of women interviewed. This demands more pilots, comprehensive survey training, new ways of asking questions, and using mixed methods;
- Measurements of WEE must consider subjective dimensions or work: This includes labour decisions, the status of work, and the constraints that limit women’s employment choices (child care and transportation constraints, street safety, and gang violence). We found that women under moderate and extreme poverty do not consider necessarily work as empowering but rather disempowering. Understanding these meanings can help to better design programs aiming at increasing women’s labor participation or entrepreneurship;
- Researchers must share their experience more: They should talk about their challenges they face in the field and be fully aware of how researcher’s positionality can bias the way we measure WEE.
Documenting the ’hidden’ gender impacts of conditionalities in Peru’s CCT programme
Tara Cookson, the final speaker, conducted an institutional ethnography of the gender impacts of Peru’s CCT programme, Juntos, which will be published in her forthcoming book, Unjust Conditions: Women’s Work and the Hidden Cost of Cash Transfer Programs. After realising that women were reluctant to talk about their experiences of participating in the programme because they were afraid of losing access to benefits, Tara partnered with a local women’s organisation that was trusted by the CCT recipients as an impartial actor.
In addition to interviewing CCT recipients and observing what participation in the CCT involved, Tara’s institutional ethnography included interviewing CCT staff and observing programme implementation and the functioning of the schools, clinics and banks linked to it, and understanding how women’s experiences of the programme were overlooked by routine programme monitoring and evaluation.
The findings:
- Walking and waiting: To meet the conditions, women must overcome geographical barriers to accessing health, education, and financial services in rural places. Routine programme evaluations do not capture the time that women spend accessing or waiting for services that are located far from their homes, and are frequently closed or short-staffed due to underfunding. This finding suggests that CCTs are not efficient from women’s perspective.
- Shadow conditions: Local CCT staff, clinic and school staff and other local authorities used threats of programme suspension to get CCT recipients to perform additional tasks, including having hospital births, participating in parades, and cooking unpaid for the school lunch programme, etc. These “shadow conditions” were not part of official policy but were a widespread practice.
Recommendations
Based on the rapid policy research Tara conducts at Ladysmith, a feminist venture for gender equality, Tara shared key recommendations for gender-sensitive monitoring and evaluation, which Ladysmith uses in their Feminist Data Collection Toolkit:
- When you don’t have time to do slow research, make use of the existing slow research. Not everyone has the time or resources to do an ethnographic study. But there are decades of feminist research on the gender impacts of social policy and development, including women’s time use, experiences of violence, and transformative definitions of empowerment. Rapid M&E can make use of this research;
- Involve women in the research process. Everywhere in the world there are local women’s movements. We should be asking them “what do you think about these programmes?”;
- Think about intersectionality. How do geography, social class, ethnicity, and age (among other factors, depending on the context), influence women’s experiences of programmes? For instance, how does women’s participation in a CCT in a rural place differ from an urban place? What difference does it make if the CCT recipient speaks Quechua, but the nurse in the clinic does not?;
- There’s no substitute for fieldwork research. Qualitative research isn’t an alternative to quantitative evaluations. It’s a great way to evaluate the quality of statistics and show where quantitative research could be applied to capture important trends. Going out to where people live and understanding how they access services is important;
- Document practices and processes through ethnographic research. Fieldwork is a good way to document institutional practices and processes that depart from the books. Understanding and mapping out how policies are implemented - how people go about their work on a regular basis - is one of the main ways qualitative research can feed into programme iteration.
Link to Tara Cookson book: https://www.ucpress.edu/book.php?isbn=9780520296992
This blog post is part of the Gender-Sensitive Social Protection Series, which brings together the summaries of webinars organised by IPC-IG and FAO on the topic. Please join the Gender-Sensitive Social Protection online community if you are interested in following the most recent discussions on the topic. If you have any thoughts on this webinar summary, we would love to hear from you. Please add your comments below!