What works?

After 1 year of implementation in the two different cohorts, HC improved some aspects of children's perceptions of their school ecologies. For the designers and implementers of HC (the IRC), these changes were what they considered to be goal #1 in education programming for children affected by armed conflicts and protracted crises. They reasoned that more safe/supportive and predictable/cooperative school ecologies were necessary to free children cognitively and emotionally to grow academically in school.

As hypothesized by the HC reduced-form TOC, in Cohort 1 and again replicated in Cohort 2, children improved in basic literacy skills (by an average of 3–4 months of additional growth per year compared with the rate of learning in the control group in eastern DRC) and numeracy skills (by an average of 9–10 months of growth per year) (Aber, Torrente, et al., Reference Aber, Torrente, Starkey, Johnston, Seidman, Halpin and Wolf2017; Torrente et al., Reference Torrente, Aber, Starkey, Johnston, Shivshanker, Weisenhorn and Dolan2019; Torrente, Johnston, et al., Reference Torrente, Johnston, Starkey, Seidman, Shivshanker, Weisenhorn and Aber2015). These are promising improvements under such difficult conditions, but it must be noted that the annual rates of improvement in these skills in control schools in the eastern DRC are very slow indeed. It is thus likely that even if children experienced high-quality HC every year of primary school – and thus achieved this enhanced rate of improvement in basic skills – most children would nonetheless leave primary school functionally illiterate and innumerate.

What doesn't work (yet)?

Neither in Cohort 1 nor in Cohort 2 did HC significantly improve teachers’ job-related wellbeing (motivation, job satisfaction, burnout) or children's social–emotional outcomes (mental health symptoms or experiences of peer victimization). It thus appears that, while HC is sufficient to effect positive changes in children's perceptions of their school ecologies and their basic academic skills, it is not sufficient to move the needle on the social–emotional outcomes for teachers or their students. In retrospect, perhaps this should not be surprising. While HC aimed to improve the quality of the classroom environment, it did not explicitly or directly target the key physiological, cognitive, social, and emotional mechanisms known to predict children's or teachers’ mental health and wellbeing. In crisis-affected contexts, where children and their teachers face extreme adversities that can overwhelm fundamental human adaptive systems, a focus on the safety and supportiveness of the classroom environment may be necessary but insufficient to improve children's and teachers’ wellbeing without attention to other powerful causal mechanisms (Tubbs Dolan & Weiss-Yagoda, Reference Tubbs Dolan and Weiss-Yagoda2017).

When the program was extended for an additional year in Cohort 1, the initial positive impacts of HC on reading, math, and school supportiveness faded out (Torrente et al., Reference Torrente, Aber, Starkey, Johnston, Shivshanker, Weisenhorn and Dolan2019). Retrospective qualitative data collection suggested that the quality of implementation probably suffered in the second year due to the addition of new treatment schools from the waitlist condition at the same time as the number of IRC staff dedicated to teacher training and monitoring declined. Unfortunately, we were unable to collect the prospective implementation data that would help evaluate this issue.

By what mechanisms?

The HC impacts we observed on literacy and numeracy skills after 1 year were plausibly and partially mediated by changes in children's perceptions of their school ecologies (Aber, Tubbs, et al., Reference Aber, Tubbs, Torrente, Halpin, Johnston, Starkey and Wolf2017). We say “plausibly” because with this TOC, research design, and the set of statistical assumptions required, we could not formally test true causal mediation. While our estimates of the impact of HC on change in the mediators and on change in the outcomes are causal, our estimates of the impact of the change in mediators on the change in outcomes are correlational. This is an unfortunate yet unavoidable limitation that can only be addressed via replication and accumulation of research. We say “partially” because there is still an unexplained direct effect of HC on outcomes even after accounting for the indirect effect via children's perceptions of their school ecologies. This means that there are likely other mechanisms that helped children in HC to learn better, which the current TOC and the data were not able to capture – suggesting the current TOC is incomplete.

For whom and under what conditions?

The main take-home message is that HC had equally positive effects across important subgroups of children (age, grade, gender) and in two independent cohorts of school districts in the country. However, there were two exceptions to this pattern.

First, when we tested gender differences in impacts of HC on teacher wellbeing, we discovered that the null average impacts were obscuring opposite effects for males and females (Wolf et al., Reference Wigelsworth, Lendrum, Oldfield, Scott, Bokkel, Tate and Emery2015). As hoped, HC increased teacher wellbeing among the male teachers; unexpectedly, HC decreased wellbeing among the female teachers. Although we did not have adequate data to investigate why this may be so, anecdotal evidence from IRC field staff generated a hypothesis. Because the teacher corps and consequently the teacher learning circles (the process for peer coaching and support) were 75% male on average, participation in the teacher learning circles may have been more stressful and less supportive for female teachers who were in the minority.

Second, HC had positive impacts (a) for both minority-language and majority-language children and (b) for schools with higher and lower percentages of minority-language children. Nonetheless, minority-language children and children in schools with higher percentages of minority-language children benefitted more than the majority-language children and schools. It is important to note here that the DRC is an enormously linguistically diverse nation with over 215 living mother tongues and four indigenous national languages. This is one reason why French has been and remains the national language of instruction in the DRC. Thus, among refugee and forcibly displaced children, a significant proportion of children did not speak the same mother tongue as their teachers and were only beginning to learn French.

In conclusion, the results of the impact evaluation were generative. We learned that the positive impacts of HC in two independent cohorts of Congolese schools on children's perceptions of their school ecologies and on their literacy and numeracy skills were consistent with two central elements of the co-constructed TOC that guided the impact evaluation. However, the findings about the lack of impact on teachers’ and children's social–emotional wellbeing and the fade-out effects after 1 year were not consistent with the TOC. Moreover, despite our adherence to the child development and social policy approach, we lacked the ability to adequately explain why and what to do about it – key impediments to developing a science for action. We now turn to our reflections on what succeeded and what we needed to improve as we continued on our path towards a science for action.

Partnerships

The results of the evaluation provided an existence proof that research–practice partnerships can be an effective and ethical strategy for generating rigorous evidence at scale under some of the most volatile and dangerous conditions in the world. Humanitarian organizations like the IRC have decades of experience operating and reaching children in crisis contexts. They are also attuned to the ethical principles underlying the provision of services to and collection of information from children, families, teachers, and schools in such contexts. Researchers bring to bear theories and evidence of how children develop, experience leveraging the realities of limited program and research funding to create rigorous research designs, and expertise developing new or testing existing measurement tools to assess program outcomes reliably and validly. The resulting evidence would not be possible without either partner. However, such partnerships require tremendous perspective taking, given that the imperatives of practitioners and researchers are often at odds. Humanitarian actors must respond to the needs of children in crisis as quickly and as widely as possible, while researchers strive to attain unbiased and precise estimates of intervention impacts – a traditionally long and slow process. Balancing practical feasibility and scientific rigor requires a shared understanding of priorities, a commitment to learning and listening, and a deep respect for respective strengths. In the DRC, we were able to embark on such a relationship with the IRC, but we also recognized two limitations to our partnership model at the time.

First, both the IRC and NYU-TIES (Transforming Intervention Effectiveness and Scale) are organizations grounded in western, educated, industrialized, rich, democratic (WEIRD) traditions and contexts (Henrich, Heine, & Norenzayan, Reference Henrich, Heine and Norenzayan2010; Wuermli, Tubbs, Petersen, & Aber, 2015). Western researchers and practitioners tend to call non-academic skills “SEL,” but countries from the global south tend to frame the non-academic features of education in quite different ways, such as peace education, moral education, or citizenship education (Torrente, Alimchandani, et al., Reference Torrente, Alimchandani, Aber, Durlak, Domitrovich, Weissberg and Gullotta2015). In efforts to rapidly intervene and conduct research in crisis contexts, we risk insufficiently grounding programmatic and research approaches in local cultures, frames, priorities, norms, and values while reifying existing inequities in who has access to research and research capacity.

Second, and relatedly, we learned that such research–practice partnerships are only one small part of the larger political economy of countries in conflicts and crises. Other actors such as policymakers and donors are able to enhance or constrain the effectiveness of the partnership and the research (e.g., by cutting key funding part way through the project). This suggested to us that a diversity of partnerships and networks of partnerships – engaging both local and regional actors – may be necessary to move as a field towards a science for action.

Conceptual frameworks

As described above, we co-constructed with the IRC a reduced-form TOC that allowed us to specify the core or essential elements of HC and create a research design that was both scientifically rigorous and practically feasible. While we did the very best we could at the time, in retrospect we did not ground the work sufficiently in theories and methods from developmental psychopathology. The field of developmental psychopathology had learned a lot about risk and protective factors facing children and their families at multiple levels of the human ecology. The field also had learned a lot about the many cognitive, social, and emotional developmental processes that mediate the influence of risk and protective factors on child outcomes, including children's academic outcomes and social and emotional skills. The inclusion of such risk and protective factors and developmental processes in our TOC and measurement approach could have helped more fully and precisely test the impact of the intervention on children's learning outcomes in our mediation models. It also could have provided key insights into the lack of impact on social and emotional outcomes. Our failure to do so was in part a reflection of logistical and budgetary constraints in this particular study, but it is also a reflection of existing inequities in where, with whom, and for whom developmental science research has taken place: 95% of psychological research has occurred with just 5% of the world's population (Arnett, Reference Arnett2008). Therefore, we could not draw much on prior theoretical, methodological, and empirical research on typical and atypical child development in the global south, including in linguistically and culturally diverse countries affected by armed conflicts and protracted crises, to inform the design and conduct of this important impact evaluation. This is a shortcoming of our study and a shortcoming of the developmental science field at large – it is not yet a global developmental science.

Methods and measures

Regarding research design, methods, and measures, we learned from this evaluation what it actually takes to conduct high-quality research in crisis-affected contexts – and we learned we needed to do significantly better. First and most foundationally, administrative data systems with unique identifiers that enable tracking of children over time were not available in schools in the DRC (or in the majority of crisis-affected contexts due to a confluence of lack of data infrastructure and migration). Investing in such systems under the evaluation budget was not feasible and thus we were unable to reliably and feasibly track individual children over the 3 years of the study. This prevented us from examining individual child-level dynamics nested within dynamic school contexts, one of the central foci of the developmental psychopathology in context perspective. Second, due to budgetary and logistical constraints, we were only able to collect annual waves of data on representative samples of children in second to fifth grade. This meant we could not evaluate within-year change – only between-year change – thereby obscuring key fluctuations in outcomes within a school year that could shed light on the ultimate positive or null impact findings.

Another serious shortcoming was our inability to conceptualize and confidently measure children's holistic learning and development (CHILD) outcomes and the quantity and quality of implementation of HC. The former concern stems, in part, from our use of CHILD measures such as the Early Grade Reading Assessment (Gove & Wetterberg, Reference Gove and Wetterberg2011) and the Strengths and Difficulties Questionnaire (Goodman, Reference Goodman1997) – measures that were developed for use based on, respectively, theories of early literacy development and social–emotional wellbeing in western contexts. We elected to use these measures (which had been used previously in some low-income and conflict-affected countries) given the lack of existing validated measures in the Congolese context and the need to rapidly mount the impact evaluation with inadequate funding for prospective measure development. While the measures were adapted for use in the Congolese contexts, the resulting data did not meet the assumptions (e.g., normal distributions) necessary for traditional factor analytic and scoring approaches (in the case of the Early Grade Reading Assessment, Halpin & Torrente, Reference Halpin and Torrente2014) and did not adhere to previously hypothesized factor structures (in the case of the Strengths and Difficulties Questionnaire, Torrente, Johnston, et al., Reference Torrente, Johnston, Starkey, Seidman, Shivshanker, Weisenhorn and Aber2015). To address such concerns, we applied post-hoc state-of-the-art psychometric methods that enabled us to build confidence in the reliability and validity of the data. Nonetheless, it is possible that we did not see an impact of the program on social and emotional wellbeing, for example, because we did not measure it in a contextually or developmentally valid way. In addition, we were unable to collect any information on the dosage and quality of implementation of HC. Thus, we don't know (a) whether variation in the size of impact of HC is explained in part by variation in its implementation or (b) whether the fade out of positive effects after 2 years of the HC intervention was due to a decline in the quantity or quality of implementation. Such experiences clearly indicated that more investment was needed to develop and adapt field-feasible measures of program implementation and quality and CHILD with evidence of reliability and validity in crisis contexts (Tubbs Dolan, Reference Tubbs Dolan2018).

Joint reflection

Traditional models of partnership are often transactional and communications one-sided: funders and/or practitioners commission an evaluation, and researchers report results at the end of project for “take-up.” For both tangible and intangible reasons, our partnership with the IRC evolved as one that strove to be more deeply collaborative. Our joint reflection on both the positive and negative results from the DRC work and the strengths and shortcomings of the partnership showed us how much we were able to achieve together – and how much more work can be done as a partnership and as a field. What came next?

Theory(ies) of change

In order to build a comprehensive understanding of the skills that these innovative programs were expected to improve, and by what mechanisms and with what assumptions, we identified a program-specific TOC for each of the programs evaluated. Specifically, we first identified the most proximal targets that each program was designed to improve. We did so through a collective review and discussion of the key elements of each program with the program developers and implementation partners of the programs (the IRC and Harvard University). As illustrated in Table 3, these immediate, proximal target outcomes were considered primary outcomes and key mechanisms of change for each program. For example, based on the findings from the evaluation of HC in the DRC, we identified children's perceptions of their school environments and their literacy and numeracy skills as the primary outcomes of the SEL-infused remedial education program. In contrast, stress response and emotional regulation were the primary outcomes of the mindfulness activities, while executive functioning skills were the primary outcomes of the brain games program.

Table 3. 3EA Lebanon program primary and secondary outcomes.

Notes: ST-SEL = skill-targeted social–emotional learning. HC = Healing Classrooms. Primary outcomes are outcomes directly targeted by the program key elements and key mechanisms of change for each program. Secondary outcomes are distal outcomes that are hypothesized to be indirectly impacted by the change of the primary outcomes.

We then identified more distal target outcomes that were hypothesized to be indirectly impacted by the change of the primary outcomes. We did so by reviewing existing theories and empirical evidence on development across interconnected domains of children's skills and across contexts. We also explicitly prioritized distal outcomes that have policy implications, such as academic and mental health outcomes. These distal outcomes – along with other outcomes that we did not hypothesize to be impacted by that specific program but were primary or distal outcomes of other programs that were concurrently being evaluated – were included in our TOC and data collection as secondary outcomes.

This process resulted in four program-specific TOCs: the HCB remedial education program, two skill-targeted activities (mindfulness, brain games), and the SEL curriculum targeting five distinct SEL skills. The co-construction of these four comprehensive TOCs by the researchers and the practitioners had two main implications for the impact evaluation studies in Lebanon and Niger.

First, these TOCs included a wide range of cognitive, emotional, social, and behavioral skills and experiences. This provided a much more comprehensive picture of intra- and interpersonal developmental processes compared with the reduced-form TOC than we were able to test in the DRC. At the same time, due to the complex research design that evaluated multiple programs concurrently, these partially overlapping – and somewhat disjointed – TOCs included multiple developmental processes about which we did not have a priori hypotheses. This provided a unique opportunity to explore relationships between key developmental processes beyond the hypothesized mechanisms of impacts with underrepresented populations in Lebanon and Niger.

Second, testing these comprehensive TOCs required developing, adapting, and validating context-relevant measures of developmental processes and outcomes that are feasible and appropriate to use in low-resourced settings. As already noted, there is a paucity of measures – particularly social–emotional skills measures – tested for use in non-WEIRD contexts. This is particularly important given how social and emotional skills are defined, prioritized, and manifested varies greatly within and between cultures and contexts (Torrente, Alimchandani, et al., Reference Torrente, Alimchandani, Aber, Durlak, Domitrovich, Weissberg and Gullotta2015). Therefore, measure development, adaptation, and validation that are fit for the context and purpose – the primary foci of the 3EA measurement work – were critical for meaningful interpretation of the findings from the study (Tubbs Dolan & Caires, Reference Tubbs Dolan and Caires2020).

Research design

The impact evaluations in Lebanon and Niger had two main design features.

The first main design feature was that our two distinct programmatic foci (access to/dosage of remedial education; value added of skill-targeted SEL activities) required a complex experimental design. Testing the impact of access to SEL-infused remedial programming (compared with formal school only) required a business-as-usual control group, while testing the impact of added skill-targeted SEL strategies required at least two groups – those who had access to HCB and those with access to HCP. We achieved this with different designs in Lebanon and Niger given the unique contexts and operational constraints. In Lebanon, in the first year of 3EA programming, we employed a three-arm site-randomized trial, with one arm being a business-as-usual waitlist control group (Tubbs Dolan, Kim, Brown, Gjicali & Aber, Reference Tubbs Dolan, Kim, Brown, Gjicali and & Aber2020). As in the evaluation in the DRC, we leveraged the IRC's lack of financial and human resources to provide services to all beneficiaries at the start of the intervention to create a scientifically and ethically sound research design. However, achieving statistical power at the site level to field a three-arm trial required the IRC to rapidly scale up their remedial programming in Lebanon, creating implementation challenges and field staff fatigue.

In Niger, the research design was constrained by both the number of schools in which the IRC had funding to operate, as well as the very real possibility that Boko Haram (which literally translates to “western education is forbidden”) could attack schools. In order to attain sufficient power and to ensure the internal validity of our design if treatment schools differentially dropped out, we fielded a design in which pairs of schools were matched on a set of baseline characteristics (Brown, Kim, Tubbs Dolan, & Aber, Reference Brown, Kim, Tubbs Dolan and Aber2020). We randomized within pairs to implement either HCB or HCP remedial programming. Then, within schools, over 8,000 students were assessed each year to determine their eligibility for remedial programming. Given that the majority of students met the criteria but there were limited spaces available in remedial classrooms, we created a lottery and randomized eligible individual students to receive remedial programming (or not). Given the size of the student body, however, we were only able to collect the most basic information (grades, attendance, literacy and numeracy skill levels) to test the impact of access to remedial programming.

The second main design feature was that, where possible, we collected longitudinal data about a robust set of developmental processes and holistic learning outcomes, demographic and ecological risk factors of children and teachers, and (some) program implementation data. We followed the same group of children within years, collecting two to three waves of data in each of the two school years 3EA operated. Because some children remained in the program across years, we also had up to five waves of data for a subset of children.

Taken together, these two design features had several implications.

First, by aligning data collection timing, program implementation cycles, and (in Lebanon) mid-year re-randomization, we were able to experimentally test 10 different research contrasts over 2 years in two countries. These research contrasts enabled us to ask the classic child development and social policy questions about our two programmatic foci. For example, in terms of “what works?” we were able to ask the following questions. (a) Does access to SEL-infused remedial education programming improve children's school ecologies, academic, social, and emotional processes and outcomes, compared with access to formal school only? (b) Does access to additional skill-targeted SEL activities and curriculum improve children's targeted social, emotional, and cognitive processes (compared with access to HCB remedial programming only)? Because we were able to examine some of these questions over multiple years and in different countries, we were also able to build confidence in the findings through a form of replication.

Second, by focusing on longitudinal data collection within the experimental trials, we had large samples with which to pose and answer questions about child development in crisis-affected settings from the perspective of developmental psychopathology in context. Such research can inform the design of more contextually appropriate programming and generate evidence to contribute to the global knowledge base for a more representative science of child development. For example, how do experiences and conditions of crisis-affected settings affect the basic processes of typical development? To what extent do children's cultures, communities, and atypical experiences lead to important similarities and differences in their developmental processes and trajectories? In the next part of this paper, we summarize the answers we have arrived at to date and discuss our partnership's collective interpretation of the results.

What works?

Overall, the findings from the DRC and early results from Lebanon and Niger provide mounting evidence that the HC approach – whether embedded in formal school curricula (as in the DRC) or as remedial education programming (as in Lebanon and Niger) – can work to improve children's perceptions of their schools and their basic academic skills.

Firstly, recall that, in the DRC, access to HC improved children's perceptions of the safety and supportiveness of their schools after 1 year of programming, but not after 2 years. We found similar results in Lebanon. A half-year of access to the SEL-infused remedial programming – including both HCB and HCP – increased children's perceptions of the safety and supportiveness of their Lebanese public school environment. A full-year of access to the same remedial programming, however, did not further improve children's perceptions (Tubbs Dolan et al., Reference Tubbs Dolan, Kim, Brown, Gjicali and & Aber2020).

Secondly, recall from the evaluation in the DRC that access to HC programming significantly improved children's literacy and numeracy skills, but that the impacts were not large enough to put children on the path to literacy and numeracy by the end of primary school. In the first year of 3EA in Niger, access to a full year of SEL-infused remedial education programming – including both HCB and HCP – improved both Nigerian refugee and Nigerien local students’ literacy and numeracy scores (effect size = .22 to .28). These positive impacts were consistent across children of different gender, grade level, refugee status, and baseline literacy and numeracy scores (Brown et al., Reference Brown, Kim, Tubbs Dolan and Aber2020). In Lebanon, we were able to evaluate the impact of SEL-infused remedial programming on a more comprehensive set of academic skills. We found that short-term (16 weeks) access to SEL-infused remedial education programming improved some of the discrete basic literacy and numeracy skills (e.g., letter recognition and number identification), but not higher order literacy (e.g., oral passage reading, reading comprehension) and numeracy (e.g., word problems) skills, or global literacy and numeracy competency (Tubbs Dolan et al., Reference Tubbs Dolan, Kim, Brown, Gjicali and & Aber2020).

As in the DRC, we suspect that implementation and dosage played a role in the consistency and size of the impacts on perceptions of school environment and academic outcomes. Unlike in the DRC study, we have some data that will allow us to unpack these findings. It is illustrative, for example, that the average attendance rate in remedial programming was 50% in Lebanon and 64% in Niger. These low rates of attendance represent the numerous challenges of and competing priorities for engaging in education in crisis-affected settings (Brown et al., Reference Brown, Kim, Annan and Aber2019; Keim & Kim, Reference Keim and Kim2019) and they highlight that context-relevant implementation strategies must be developed to ensure impacts are consistent, sustainable, and meaningful among populations where frequent attendance is not possible.

What doesn't work (yet)?

Overall, we found that moving the needle on children's social and emotional outcomes in crisis contexts remains elusive. In Lebanon, we did not find that any of the three types of SEL programs tested – mindfulness, brain games, and five-component SEL – had conclusive, significant impacts on children's social and emotional skills. These disappointing results may be partially due to the relatively small dosage and treatment contrasts, as these skill-targeted SEL activities and curricula were designed to be short and quick activities implemented within the larger HCB remedial education programs. Combined with the above-described attendance and quality of implementation issues that plague programs in humanitarian settings, these programs – implemented for a relatively short duration – were not sufficient to make meaningful improvements in children's social and emotional skills. We did observe that some of the skill-targeted SEL programs had potentially promising positive impacts on primary social and emotional outcomes, but the impacts were not large enough or consistent enough to provide statistically conclusive evidence. In Niger, there was also evidence that access to additional mindfulness and brain games activities improved students’ grades in their public schools (Brown et al., Reference Brown, Kim, Tubbs Dolan and Aber2020). We will continue to explore potential heterogeneity of impacts to identify how, for whom and under what conditions these skill-targeted programs may improve children's academic and social and emotional outcomes.

Child development in crisis-affected settings

Leveraging the data on the wide range of cognitive, social, emotional, and behavioral processes that are hypothesized to be the primary and secondary outcomes of the 3EA programs, we were also able to examine what child development looks like and what are the predictors of adaptive and maladaptive development in Lebanon and Niger. This gave us an opportunity to explicitly test some of the hypothesized relations between children's developmental processes and outcomes that we hypothesized in the TOCs in the contexts for which the programs were designed.

We have only begun to unravel such complexity and nuances. In the first developmental study to result from the 3EA data (Kim, Brown et al., Reference Kim, Brown, Tubbs Dolan, Sheridan and Aber2020), we found that Syrian refugee children face many pre-, peri-, and post-migration risks, some of which are especially salient among refugee populations (e.g., being assigned to a lower grade than is normative for their age) and some of which are nearly universal risk factors (e.g., poor health). These risks are associated with decrements in children's ability to regulate themselves cognitively and behaviorally, and to achieve literacy and numeracy skills (Kim, Brown et al., Reference Kim, Brown, Tubbs Dolan, Sheridan and Aber2020). We also found that children's regulatory skills, including cognitive (executive function) skills and behavioral regulation skills, are key predictors of children's academic learning. These findings replicate research from WEIRD contexts that emphasize the importance of SEL skills for children's ability to learn. These regulatory skills were also primary targets of some of the 3EA skill-targeted SEL programs and, as such, these findings provide support for the proposed TOCs of these programs. Further analysis on how these developmental processes and risks dynamically interact over time and across different cultural and sociopolitical contexts will help us to better understand the experiences of children growing up in crisis-affected settings and to identify more effective ways to support children's positive development.

Partnerships

Our experiences with the 3EA initiative reinforced our finding that research–practice partnerships can be an effective and ethical strategy for generating and using evidence in crisis contexts. Our experiences with the IRC in 3EA further highlighted the power of deep respect in partnerships to motivate the mutual capacity building required for turning science into action. Colleagues at the IRC now regularly ask questions about effect sizes, treatment contrasts, and heterogeneity of impact, while we at NYU-TIES obsess over what the findings mean for practice and how to communicate findings to practitioners and policymakers clearly and intuitively. These are critical skills for researchers and practitioners to learn if we truly want to move towards evidence-informed programming for children in crisis contexts.

However, building these skills – as individuals and as organizations – has a price. Academic institutions tend to place a higher premium on publications in peer-reviewed journals than on communication to non-technical audiences or the soft skills required for sustained partnership. Many humanitarian organizations believe that funding and time is better spent on programming than on research. Enabling the individual and organizational pursuit of such partnerships thus requires seismic shifts in sectoral norms and imperatives. Even if pursued, maintaining healthy partnerships requires consistency of staff and resources to ensure coordination and sustained capacity. Fortunately, we were able to form strong relationships with key donors that provided the multi-year funding required for such sustained staffing and capacity. By moving from traditional output/accountability models to joint learning/continuous improvement models, donors can encourage a culture of sharing what works and what hasn't worked yet (as reflected in this paper). This information is critical for making decisions about how to invest scarce program and research resources in crisis contexts.

We also recognize that we must do more to ensure that the voices and engagement of local communities, researchers, NGOs, and governments are front and center in research and in partnerships. We were able to begin to build such relationships through our 3EA measurement research (Tubbs Dolan & Caires, Reference Tubbs Dolan and Caires2020). However, as we reflect in the wake of the killing of George Floyd and the protests for racial and social justice, we recognize that we have not done enough to ensure that our interactions, assumptions, methods, and partnerships actively contribute to anti-racism and decolonization in the contexts in which we work. We commit to doing better.

Conceptual frameworks

In the evaluation in the DRC, we lived with the limitations of a reduced-form TOC. In 3EA, we have the opposite issue: we are living with both the benefits and challenges of multiple (at least four) comprehensive TOCs. We say “benefits” because they allowed us to specify and measure key constructs necessary for answering questions within the perspectives of child development and social policy and developmental psychopathology in context. These include measures of the implementation of the skill-targeted SEL activities and curricula, developmental processes, ecological risk factors, and holistic learning outcomes. We say “challenges” because it is difficult to take stock of our findings across all of the constructs in all of the different TOCs. While we are perhaps better poised to answer why we have null results, we are limited in our ability to determine what to do about it if we look contrast by contrast, construct by construct.

We now need to move towards aligning our findings – and those being generated by other stakeholders in both the education and child protection sectors – within a conceptual framework. Conceptual frameworks do not focus on the specific content of strategies and approaches; rather, they specify the expected relationships among broad components of education systems and programs and children's developmental skills and outcomes. In doing so, they provide a heuristic against which diverse stakeholders can map and organize specific findings, comprehensively examine and process the implications of those findings, and communicate findings with others (Blyth, Jones, & Borowski, Reference Blyth, Jones and Borowski2018). Such organization and communication are necessary for ensuring that the limited resources available for research in crisis contexts result in a coherent evidence base from which it is possible to draw recommendations for program and policy decision making (Tubbs Dolan, Reference Tubbs Dolan2018).

Methods and measures

While we made significant improvements over the methods and measures used in the evaluation in the DRC, we learned in 3EA that we still had a long way to go.

First, while we worked with the IRC to create or modify existing administrative data systems to track children over time, the process for doing so was error prone and intensive in terms of time and resources for the scale of data collection to which we aspired: it took thousands of hours to clean and organize 1 year of data from one of the 3EA countries so that we could confidently use it for analyses. Some of the issues we encountered can be solved before and during collection itself: for example, through better enumerator training and monitoring of enumerator quality (Brown & Ngoga, Reference Brown and Ngoga2019). However, more broadly, what is required is investments in the systems and capacity for data cleaning, managing, and curating (Anker & Falek, Reference Anker and Falek2020). In addition, even once cleaned, methodological advances are needed to improve missing data imputation and causal inference in contexts with complex migration patterns and selection processes (Montes de Oca Salinas, Reference Montes de Oca Salinas2020).

Second, we made progress in measuring CHILD outcomes and program implementation, including through performance-based (Ford, Kim, Brown, Aber, & Sheridan, Reference Ford, Kim, Brown, Aber and Sheridan2019), scenario-based (Kim & Tubbs Dolan, Reference Kim and Tubbs Dolan2019), teacher-report (Kim, Gjicali, Wu, & Tobbs Dolan, Reference Kim, Gjicali, Wu and Tobbs Dolan2020), and assessor-report measures. Measures were either adapted or assembled from WEIRD settings (when necessary due to time constraints) or specifically developed with the population and setting in mind (when additional resources were available). All measures, however, required extensive analytical testing that resulted in either (a) adjustments to meet the psychometric standards necessary for impact evaluation or (b), in some rare instances, significant adaptation or being deemed unreliable for the study. These experiences reinforced to us that strong measures are not instantly created: they are developed and refined over time as evidence accumulates with new trials, in different contexts, or for distinct purposes. Identifying stronger measures is like trying to identify what programs work best for children. It is hard to draw broad conclusions from one or two evaluations of programs in different contexts – dozens of trials are needed to have confidence that the program is really working and achieving what is intended. The same is true for measures. Having a ready supply of measures that can be used to generate evidence in crisis contexts requires a community of users who are empowered and committed to using the measures over time and reporting back on psychometric properties (Tubbs Dolan & Caires, Reference Tubbs Dolan and Caires2020).

Joint reflection

Our initial goal was to rapidly prototype effective SEL programs. We aimed to do so by consecutively testing three different types of SEL programs over a 2-year period, making decisions about the next year's programs based on usable and reliable evidence from the first year's evaluation study. However, this approach, combined with large-scale data collection and implementation, turned out to be challenging and stressful for both organizations. While we were able to produce preliminary evidence to inform some of the programmatic and research decisions in this time frame, some of the preliminary findings we have shared for decision making turned out to be inconsistent with the final results, and often insufficient to be used as bases for program decisions. We found that if the practice partner needs rapid feedback, a large-scale randomized controlled trial is not the right answer. If we could do it again, we would conduct different types of studies for different purposes over a longer time frame. First, we would conduct a series of targeted, rapid prototyping, and feasibility studies for program development and measures validation during the inception period. Second, we would be sure to build in time to inform revision of program design. Third, once we had built confidence in a program through short-term, small-scale, and iterative feasibility studies, adequately powered randomized controlled trials would be needed to generate evidence of effectiveness of the program at scale. For this type of iterative research–practice feedback loop to work, a longer term commitment of and partnerships with funders would also be critical. We thus continue.