Executive Summary

Chris Minch October 2016

Introduction

This report focuses on Aidlink’s Immersion Programme (IP), which forms a key part of the organisation’s Development Education (DE) activities. The IP involves Aidlink facilitating groups of secondary school students to undertake educational visits to African countries such as Ghana, Kenya and Uganda. A combination of a sector-wide increase in focus on monitoring and evaluation, increased interest from schools in Ireland, and the maturity of Aidlink’s IP, have meant that the time is right to begin formally evaluating and capturing the learning from more than a decade of IP visits. This report is an output of this increased focus on evaluation, and has several aims:

1. Undertake a review of DE and immersion literature.
2. Compile a detailed overview of Aidlink’s IP.
3. Formalise the aims and desired outcomes of the IP.
4. Analyse quantitative and qualitative data from the 2016 IPs.
5. Review the use of the WorldWise Global Schools Self-Assessment Tool in Aidlink’s 2016 evaluation pilot, and provide recommendations for tools that can be used in future evaluations.

Development Education

DE involves the provision of programmes and information designed to increase people’s understanding of the causes and complexities of global issues such as poverty and underdevelopment. In line with this, Irish Aid has defined four key components of DE:

• Knowledge, Ideas & Understanding
• Attitudes & Values
• Skills & Capabilities
• Behaviour, Experiences & Actions

As such, DE aims to increase specific competencies in communications, intellect, social skills and taking action, hopefully beginning a transformative process for individuals, leading to innovations that bring about positive social and educative change at home and abroad.

Unfortunately, there is a current dearth in research into the impact of DE. Two main reasons have been put forward for this. The first is the disparate nature and lack of centralised governance of the DE sector in Ireland, resulting in evaluations that are often narrow in focus and non-generalisable. The second is methodological concerns regarding the tools and indicators for evaluating the impact of DE. Addressing these concerns will be crucial to establishing the true impact DE has at an individual and societal level.

Immersions

IPs are a form of DE, whereby participants undertake short trips to a community in another country, in order to experience first hand how people there live and work. They aim to highlight the complexity of inequality and underdevelopment in an arena where the power dynamic is more equal than the standard rich-poor or donor-recipient relationship. By doing this, it is hoped that immersion participants will gain increased levels of knowledge, altered attitudes towards development issues, a keener sense of global citizenship, and more desire to participate in movements towards increased social justice.

In Ireland, IPs most commonly involve schools making links with other schools in Lower Income Countries (LICs), often through shared religious orders. Cohorts of students then travel to these LICs to gain experience of, and insight into, life there over a 1-3 week visit.

Again, however, research into the impact of IPs is extremely limited, and findings mostly appear to be either based on anecdotal evidence, or suggest underwhelming medium- and long-term outcomes. As a result, there are concerns that IPs are cost-intensive, non-scalable and not subject to any meaningful evaluation, constituting a risk to continued funding.

However, IPs tend to have a significant short-term emotional impact on participants, creating positive attitudes toward development causes and DE. Therefore, to maximise the potential impact of IPs and turn these short-term gains into longer-lasting changes, it is important to design them according to principles of best practice. For example, IPs must work hard to push participants out of any self-centric visions of development during the pre-departure phase. IPs should also have clear objectives, shared expectations, and mechanisms for feedback and communication. Other important concerns include health and safety, having a well-known and trusted facilitator, and involving host communities as much as possible. Finally, critical reflection is a key part of the IP process, and, ideally, as much emphasis should be placed on reflection as on the trip itself.

Aidlink IP

Aidlink’s first immersion trip occurred in 2004, with ten students and two teachers travelling to Ghana. Since then, a total of 425 students and 79 teachers have been involved in 15 separate immersion trips. An upsurge in interest in 2016 has resulted in four trips and more than 120 participants travelling to three different African countries – Ghana, Kenya and Uganda. Since 2004, there have also been 3 reciprocal trips for Ghanaians travelling to Ireland, allowing for mutual learning and development.

Over these 15 trips, Aidlink have developed a standardised IP model, underpinned by theories of Experiential and Hands-On Learning, to fulfil many of the aims and desired outcomes of the various stakeholders. Initially, before embarking on the trip, Aidlink conducts a number of pre-departure classroom-based workshops with participants on relevant development issues. There are also practical IP preparation sessions aimed at informing the participants and managing their expectations.

The trip itself involves three main elements: one week in a local secondary school; several days viewing the work of local NGOs and partner organisations; and some time to visit local cultural attractions and to reflect on the experiences of the trip.

Finally, upon returning from the host country, participants take part in a formal reflection workshop, where they discuss various aspects of the trip and what they have learnt from the experience.

Evaluation of the IP

In 2015/2016, a number of reasons saw Aidlink attempt to evaluate the impact of the IP in a more formal and rigorous manner. First, the IP is seen as a key part of the organisation’s engagement with Africa and there is a need to show that the IP is adding value for all stakeholders. Second, there is a growing emphasis within the DE community on transparency, analysis, monitoring and evaluation, and sharing learning. Third, the increased diversity of the sample of participants has created the potential for a more robust evaluation. Finally, growing interest in Aidlink’s IP creates a need to capture the learning from previous trips, use it to attract resources and apportion these resources in an appropriate, evidence-informed manner.

Accordingly, Aidlink have sought to improve their capacity to formally evaluate the IP. For example, they have produced a short policy document outlining their approach to IPs. Most pertinently, however, Aidlink have also piloted the WorldWise Global Schools (WWGS) Self-Assessment Tool (SAT) for all four immersion trips in 2016. In conjunction with a qualitative survey, the WWGS SAT is the first attempt to systematically evaluate the impact of Aidlink’s IP on participants. Thus far, data has been collected for two of the four trips in 2016.

Quantitative Results

The WWGS SAT requires participants to self-rate, on a scale of 0-5, where they are ‘now’ in relation to four different areas of development: Knowledge; Skills; Values & Attitudes; and Actions. Some questions were adapted slightly by Aidlink to reflect the nature of the IP. The tool was administered to each participant twice – once prior to the trip in order to form a baseline measure and once post-return. Using the scores provided, average scores and changes were calculated for each area of development. Encouragingly, the table below shows that the IP groups recorded growth in all four areas of personal development.

Impressively, the area of knowledge grew 33% to a final rating of more than 3.9 out of 5. This may have been down to the knowledge accrued from the IP being extremely salient to participants as they gained first-hand experience of development in action.

Skills and Values & Attitudes both started from a much higher baseline of 3.8 and 4 respectively, making their growth of 9% and 17% notable. Impressively, the average score on Values and Attitudes finished close to the highest possible score of 5.

The 14% growth in the area of Action was slightly more disappointing, resulting in a failure to converge on other areas from a lower baseline of 2.6 out of 5. This may have been down to the fact that data was collected just 2-3 weeks after participants had returned, rendering questions about organising and participating in social justice events somewhat obsolete.

Overall, there were limited differences when disaggregating the data by gender or by school, possibly due to small sample sizes. It will be easier to make conclusions about between-groups differences as more data is collected year on year.

Qualitative Results

Supplementary to the WWGS SAT, all participants and stakeholders were asked to complete a qualitative survey after the IP. These surveys asked stakeholders to outline their positive experiences and challenges/concerns in relation to the IP; whether they would participate again; and if they had any suggestions for what should be done differently. This was intended as a reflective and individual way for people involved in the IP to demonstrate their learning and make suggestions for improving the programme.

In terms of positive experiences, there was significant overlap between the two different IPs, indicating the replicability of the Aidlink IP model across different contexts. Of note was the number of times activities that occurred on participants’ first full day in the host country were mentioned. This gives an indication of the importance of initial experiences on the IP and Aidlink should give specific consideration to maximising the potential benefit of these in future.

There was less overlap in the challenges and concerns highlighted from the two IP trips. One group of participants focused mainly on issues such as a lack of resources, poverty and inequality, and associated feelings of helplessness or guilt. The other group focused more on practical challenges such as sickness, travel times, etc. This emphasized the individual nature of the challenges/concerns mentioned.

An interesting topic that frequently appeared in the qualitative feedback was the Irish participants’ desire to have a more tangible impact on local community issues they encountered during the IP. They reported struggling to see any impact from them being there, and many expressed a desire to get more involved in an action project, for example building a water jar. While this desire to help is admirable, undertaking such a project may not be entirely appropriate or desirable, particularly if participants lack specific skills required, e.g. for building work. Aidlink’s IP model focuses on learning rather than providing services, and ways should be sought to manage participants’ expectations in this regard.

Feedback obtained from African partners, teachers and students was largely very positive, highlighting the mutual learning accruing from real engagement among peers. It was encouraging to note that partners suggested they would like to take on more logistical and organisational responsibility for the IP, as they felt it would be beneficial for them in terms of learning and capacity development.

All stakeholders suggested changes and improvements that could be made to the IP, indicating that the programme is not perfect in its current state. However, they all also suggested that they would like to participate in the IP again. Many also provided resounding endorsements of the programme’s personal impact. For example one participant stated:

Review of the Pilot Evaluation Procedure

Participants on the IP in 2016 were found to be able to follow instructions on how to complete the WWGS SAT and familiarised themselves with how the tool works relatively quickly. However, four main areas of concern regarding the delivery of the tool were identified:

1. Time – Concern was expressed that 2-3 weeks is insufficient time for the benefits of the IP to become ingrained in participants’ conscious, and for them to be able to reflect on and express these benefits in any meaningful way.
2. Design – The design/layout of the WWGS SAT was thought to be confusing and distracting for participants, particularly having the wheel prominently displayed in the middle of the page.
3. Consistency of delivery – Some contextual differences in delivery of the tool were inevitable due to the varying school environments. However, fidelity was further impacted due to varying degrees of preparation and focus in different schools.
4. Content of the tool – Some of the concepts measured by the tool were thought to be abstract and inaccessible for some teenagers.

As it is important for Aidlink to seek evaluation methods that measure the outcomes of the IP as accurately as possible and make sure that the IP process is implemented in the best possible way, this report sought to put forward several options for improving the evaluation procedure as follows.

Evaluation Options for Aidlink’s IP

Firstly, two potential options were ruled out due to the high levels of difficulty and resource requirements involved:

• Developing and testing a custom tool would require a huge amount of dedicated resources and time, due to multiple stages of design, piloting, feedback and re-design.
• A longitudinal study to track the development of former participants would require a significant amount of resources over an extended period of time.

 For both of these options, the amount of resources required are not currently available to Aidlink, making them unrealistic prospects at this particular moment in time.

Option 1 – Dynamic adaptation of the WWGS SAT

One option addressing concerns with the WWGS SAT is to further adapt it to so that it better suits the needs of all stakeholders. While adapting evaluation tools is traditionally frowned upon, Aidlink could contact the appropriate representatives in WWGS and work with them to implement a suitable blend of adaptation and fidelity.

Option 2 – Adopting a different quantitative evaluation tool

If Aidlink decides to scrap the use of the WWGS SAT tool, an alternative option would be to find a more effective and insightful tool. This option is difficult but could increase the objectivity and accuracy of evaluating the IP, particularly if tools were found that did not rely on self-report data.

Option 3 – The ‘Most Significant Change’ (MSC) method

The MSC is a form of participatory monitoring and evaluation, involving stakeholders at each level of an organisation. Its flexibility, ability to identify unexpected changes, and easy adaptation are advantageous. However, it may be more useful in conjunction with, or as a short-term replacement for, a quantitative tool.

Option 4 – Developing a custom suite of existing tools

The final option presented here is for Aidlink to adopt a suite of quantitative tools to evaluate the IP. This could help to measure the four DE components in a separate but comprehensive manner. However, selecting and vetting the best tools and surveys would require detailed research and the establishment of a clear set of criteria for inclusion, a task that is beyond the scope of this report.

Recommendations 

1. Increased focus on reflection – While some dedicated reflection sessions are incorporated into the IP, increasing the focus on reflection will help to capitalise on short-term emotional responses and gains in knowledge, attitudes and skills.

2. Promoting continued engagement – Keeping IP participants engaged with development issues over the medium- and long-term is critical to the success of the programme. Aidlink should strive to facilitate former IP participants in this. 

3. Continued focus on best practice – While Aidlink successfully adheres to much of the best practice advice provided in the literature, increased or continued focus should be placed on managing participants’ expectations, minimising disruption to everyday life in the host communities, and minimising in-country travel. 

4. Building the capacity of partner and host organisations – Aidlink should build on the enthusiasm of partner organisations looking to be more involved in the logistics and planning of the programme by allocating them more responsibility and helping them to build their capacity over time. 

5. Addressing flaws in the evaluation process – It will be important for Aidlink to choose a suitable evaluation process based on available resources and future plans for the IP. Once the decision has been made, it will also be important to establish a clear and suitable procedure and timeframe for collecting and analysing evaluation data.

For the Full Report click here