Monitoring and Evaluation ECOUNESCOs experience Elaine Nevin National

Monitoring and Evaluation ECO-UNESCO’s experience Elaine Nevin National Director ECO-UNESCO NYCI Conference 2011

Who we are? v. A National Environmental Education and Youth Organisation. Est. in 1986 environmental education and environmental work with young people. Affiliated to the World Federation of UNESCO Clubs, Centres and Associations (WFUCA)

Our Mission To channel young peoples passion into environmental protection and conservation and to promote the personal development of young people.

Our Aims… • Promote environmental awareness, knowledge and understanding in young people • Promote environmental protection & conservation • Promote the personal development of young people • Promote UNESCO ideals

What we do… Environmental Youth programmes ECO-UNESCO Clubs Young Environmentalist Awards Programme Environmental Workshops Youth Forums, events and activities ECO Choices programme Youth for Sustainable Development programme Education and Training programmes Short Trainings – Teachers, Youth Leaders & Community Worker FETAC accredited centre Sustainable Development L 5 ECO-Community Development L 3 Educational Publications Various interactive resources (Print/Media) Representation & Advocacy

What we do… ECOUNESCO Clubs Youth for Sustainable Development PEP Young Environment alist Awards ECOChoices programm e Childrens Progs Mission Teacher and Leader Training Accredite d trainings Educatio n resources General training

The Scale of our Work • Over 9500 people annually • National Basis / YEA all-Ireland programme • A team of 10 staff members • 2 government work placements at any one time • support 6 graduate interns • over 100 volunteers

What we want to know…M&E p How many people/organisations/ schools are accessing our programmes and services at any one time and over a period? p What are the demographics of these people - age, gender, geographical spread etc? p What is the retention rate of our programmes (for longer term direct contact programmes) – do we have a high fall off rate?

What we want to know… p p p What have people gained from participating in an ECO-UNESCO workshop, programme or training? - knowledge, skills developed, attitudes, behaviour change, how much have they enjoyed it etc. What is the impact of our programme on the participant? (impact from a range of perspectives – due to the nature of our work)

What we want to know… p Are we delivering programmes that address a need? p As an organisation are we achieving our goals, targets and expected outcomes? p Are we delivering quality programmes?

Why do we monitor & evaluate? p Out of our own desire to develop quality programmes; review, learn, modify & improve p Out of our own need to ensure our delivery to greater numbers of young people p To ensure we are doing the job the way we should be doing it, with the people we should we doing it with, achieving the results addressing the needs of young people with an impact on our participants p Are we responding to needs of young people? p Is our M & E funder driven?

Some of our tools of Monitoring & Evaluation p p Performance Management System – Planning, developing, monitoring and appraisal Before and after questionnaires on knowledge, attitude and behavior (baseline/endline)- for programmes p Analysis of programme participant feedback on evaluation forms p Specific programme evaluations e. g. YEA evaluations p Other tools – n Youth worker self-evaluation forms post youth programme/training /workshops n people counter to count numbers of people visiting the Greenhouse etc.

Samples of some of our tools of Monitoring & Evaluation measure p PLANNING (Qualitative/Quantitative) n Annual Implementation Plan p MONITORING (Quantitative/Quantitative) n Direct Participant Analysis Information n Direct Participant Feedback on Learning & on Satisfaction n Evaluation forms as part of programmes e. g. YEA Through YEA/Clubs/YSD, analyse what, how many and the impact of environmental action projects; p EVALUATION n Reviews carried out n Impact Assessment

SAMPLE 1: PLANNING - Annual Implementation Plan – Organisational Plan for a year Programme - Strategic Aims Objective Actions Projected Output KPIs (Key Performance Indicator) Review on a quarterly basis Outcome Target Group Time Frame/ Deadline Result / (Progress)

SAMPLE 2: MONITORING -Participant Analysis – Demographics etc. Age range, gender breakdown & geographical spread of participants <10 PROG CODE DATE Mth 15 -17 18 -21 22 -24 >25 NAME OF EVENT NEW PARTICIPANTS CONTACT HOURS TOTAL PARTICIPANTS M Day 10 -14 F M F M F M F Year Monthly/annual programme -MONITORING

SAMPLE 3: MONITORING - Participant Feedback on Satisfaction & Learning ECO-UNESCO measures the following: • Content relevance • Process (quality of facilitation, presentation and group work/activities) • Resource materials • Learning We analyse reactions and comments to enhance quality of future work 46% - Excellent 44% - Very Good 8% - Good 2% - Fair 0% - Poor 3

SAMPLE 3: MONITORING - Participant Feedback on learning Youth Leader Training Course Participants carry out Self Assessment of Learning in terms of knowledge, skills & attitudes – before & after each activity/ programme Self-assessment of Your understanding, knowledge and Skills Related to: Before Course 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 Q 1. What Environmental Education (Education for Sustainable Development) is Q 2. How to integrate interactive Environmental Education activities into your daily work Q 3. How to encourage critical thinking in young people through Environmental Education Q 4. How to foster Environmental Awareness in Young People through Environmental Education Q 5. How to carry out an Environmental Action Project (Energy Audit) with Young People After Course 1 2 3 4 5 1 2 3 4 5 1 = No understanding or skills 3 = Some understanding or skills 5 = A lot of understanding or skills

SAMPLE 4: Impact Assessment - PILOT INITIATIVE 2010 -2011 p In 2010, we wanted to look at how we could measure the impact of our programmes; reviewed and decided to implement an Impact Assessment mechanism p Impact Assessment Workshops (3 to 6 months post delivery) Workshop used to measure two of our programmes: n Youth for Sustainable Development Peer Education programme n FETAC Level 5 Sustainable Development programme The method/workshop measures in two ways: n In relation to what use and in what ways the learning has been applied in practice by the participant/learners themselves after completing the ECO-UNESCO programme n In a broader context how much the ECO-UNESCO programme has impacted on them and is of use to society in general and on their communities? v v ECO-UNESCO took the role of facilitator Former participants become the impact assessors

Sample Impact Assessment Grid

Sample Impact Assessment Stories Story 1 Name: Frida Besong “Thanks to my participation in ECO-UNESCO’s FETAC Level 5 Sustainable Development course, my interest was aroused on Sustainable Development issues and I finally wrote my M. A thesis on International cooperation regarding the management of water in Lake Chad and other natural resources in the Lake Chad Basin. At this stage I have applied for a P. H. D In Education for Sustainable Development and I am waiting for the outcome” Story 2 Name: Julia Lane Barnes “My update since I finished ECO-UNESCO’s FETAC Level 5 Sustainable Development course is that I am involved with working with a local interfaith youth group with an emphasis on the common theme of the inner spiritual garden. We use lots of the methodology that I learned doing ECO-UNESCO’s FETAC Level 5 Sustainable Development course. Our long term aim is to make a link between the restoration of a local walled garden and going to Mount Sinai with the Makhad Trust to help mend the walled gardens of the Bedouim. ” Story 3 Name: Helen “I helped organize the Greenhouse gig fashion show (recycled materials), which was about organization – there was loads of organization in it. And it was about making people aware about climate change & development, so that was Peer Education. ” (about 50 to 60 Peers)

Challenges p Time - Small organisation with limited capacity – want to do the job and make sure we are doing the job well – but need to balance this with getting the job done – can be a challenge. p Buy in from all involved - depending on people to collect data and information and to input it into a system (Salesforce CRM) p Making sure what you are monitoring and evaluating is what you need to be monitoring and evaluating – using the information you get M & E needs to focus on Quality as much as Quantity. p p p We gradually implemented the M & E system – but it is under constant review Specialist organisation with diverse funding streams with diverse requirements in terms of outcomes and outputs

Future plans… 2011 onwards Review current M & E system – ensure it is useful and informing our work in the way it should be Ensure it is in line with and collecting necessary information for the implementation of National Quality Standards Framework - use this as a tool.

For further information please contact ECO-UNESCO 'the greenhouse' 17 St. , Andrew St. , Dublin 2 T: 00 353 (0) 1 6625491 F: 00 353 (0) 1 6625493 E: info@ecounesco. ie W: www. ecounesco. ie ECO-UNESCO – Conserving the Environment; Empowering Young People ECO-UNESCO The Greenhouse, 17 Saint Andrew Street, Dublin 2, tel +353 1 6625491 www. ecounesco. ie email: info@ecounesco. ie