PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Monitoring and evaluation PSYCHOSOCIAL

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Monitoring and evaluation

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Focus of this workshop session Psychosocial interventions Indicators Monitoring Evaluation What they are? Why they are important? Who is involved? Planning Methods

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION What is monitoring? Write a word or phrase that you associate with the term ’monitoring’ on the cards you have received. When you have finished, display your cards on the board/wall space. If someone has written the same word as you then put your card on top of that card.

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION What is monitoring? Monitoring is the regular and continuous process of collecting and analyzing data to assess progress and development.

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Two main types of monitoring in RCRC Process oriented Progress and development of response Activities implemented as planned? Use of resources Problems? How should they be dealt with? New opportunities for improving response?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Two main types of monitoring in RCRC Results oriented Direct results of the interventions Relevance of present and planned activities Are objectives still realistic/relevant? Changes in targeted population / environment (impact) Is more information needed?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION What is monitoring? Is the response still relevant to the needs of the population? Keeps track of inputs, outputs and outcomes using indicators

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION What are inputs, outputs and outcomes? Input Any resources that is ‘put in’ to the program E. g. funds, materials, personnel, time Outputs: Measurable achievements that have been ‘put out’ or produced as a result of the intervention E. g. numbers of people trained, numbers of meetings held, etc. Outcomes: Changes that have come about as a result of the program E. g. improvement of psychosocial wellbeing, skills and knowledge on PSS improved,

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION INDICATORS Brainstorm: What are indicators?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION What are indicators? • Criteria used to measure the data collected • Measure changes related to implementation Examples of (outcome) indicators in PS responses: • Decrease in stress related symptoms of population A • Increase knowledge and skills in providing PFA • Children regain desire to play • Beneficiaries regain capacity to relate to others

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Examples of Input, output and outcome indicators Input Indicators: AMOUNT $ spent; NUMBERS of personnel working; NUMBERS of hours Output indicators: NUMBERS of people trained; NUMBERS of meetings held; NUMBERS of PS kits distributed Outcome indicators: NUMBER of children playing (quantitative); selfreported INCREASE in desire for social interaction (qualitative); LEVEL of skill improvement in PSS

Keeping track of inputs, outputs and outcomes for activities - An example Logical framework approach Indicators Overall Goal To reduce suffering and risk for development of severe trauma of Population A Adults and children show healthy signs of coping with impact of crisis event Activity 1 Training in PFA Input Resources to enable training Amount of money; personnel; training manuals; materials Output Training has taken place Number of people trained Outcome Volunteers can provide PFA Skill level in PFA has increased Activity 2: Workshops with children Input Resources for workshops Amount of money; personnel; training manuals; materials Output Workshops are held Number of workshops held; number of children attended; number of trainers Outcome Children are coping better Increase in children’s playfulness, self-confidence, trusting of others

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Why is monitoring important? 1. Relevance 2. Responsibility and communication 3. Accountability Simply put: Are we on track? Are we on the RIGHT track?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Who is involved in monitoring? Responsi bility Including monitoring and evaluation Other partners Data collection

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION What are evaluations? In buzz groups of 2 or 3, discuss amongst yourselves what evaluations are and how they differ from monitoring? Try to write a simple and clear definition of what an evaluation does in a psychosocial response.

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION What are evaluations? “Evaluations explore whether the interventions succeeded in achieving the overall goals and aims of the psychosocial response” Discussion points: 1. Evaluations are meant to be OBJECTIVE – what does it mean and why is it important?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION What are evaluations? Evaluations = Monitoring + more Final evaluation Mid term evaluation M 1 M 2 M 3 M 4 M 5 M 6 M 7 M 8

Evaluations focus on assessing OVERALL AIM/GOAL Logical framework approach Indicators Overall Goal To reduce suffering and risk for development of severe trauma of Population A Adults and children show healthy signs of coping with impact of crisis event Activity 1 Training in PFA Input Resources to enable training Amount of money; personnel; training manuals; materials Output Training has taken place Number of people trained Outcome Volunteers can provide PFA Skill level in PFA has increased Activity 2: Workshops with children Input Resources for workshops Amount of money; personnel; training manuals; materials Output Workshops are held Number of workshops held; number of children attended; number of trainers Outcome Children are coping better Increase in children’s playfulness, self-confidence, trusting of others

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Types of evaluations Real time evaluations • Early in response • Internal staff • Focus: process / operational issues Mid-term evaluations • Assess impact • Accountability (budgets/admin) • Guide needed adaptations Final evaluations • Impact and process • key successes & challenges • Invaluable info for future PS interventions

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Evaluating impact • How do you measure the impact of an intervention? • What do you need? Data SAME INDICATORS Data

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION How to define Psychosocial indicators • Assess - How is the population affected? • What kind of change is desired? (Goals/objectives/activities of response) • What is the local definition of psychosocial wellbeing?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Indicators of psychosocial wellbeing

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Psychosocial indicators 1. Write three things about yourself or your life that you think show to others that you are feeling good and doing well. 2. Now think and write about how others could measure whether you felt better from one day to the next. 3. In plenary share some of your indicators of wellbeing. We aim to come up with a good variety of indicators, and also some indicators that may be the same for the whole group.

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Psychosocial indicators People have different things that indicate their psychosocial wellbeing, but if we spent some time we would probably be able to find some common indicators that everyone agree would be both general, and specific, enough to determine the groups’ psychosocial wellbeing. Indicators of psychosocial wellbeing focus on: - How the individuals are doing - How the group is doing. The nature of psychosocial is a focus on the individual’s psyche and on the social interaction and network.

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Psychosocial indicators to d e ne the eel e lp f W he en. r r… d il tte h c be But how will we KNOW if they feel better? I KNOW. . We will ask them how THEY know when they feel better and then we will find ways to measure that…

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Why is evaluation important? 1. Relevance (response relevant to needs of pop. ) 2. Efficiency (time-frame, cost) 3. Impact (succeeded or not) 4. Effectiveness (objective of program met) 5. Sustainability (benefits continue beyond the program)

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Who is involved in evaluations apart from staff, volunteers, partners? Responsi bility Including monitoring and evaluation External consultants Other partners Data collection

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Sampling Broad and varied Targeted population Broad & varied

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION When to monitor and evaluate

PSYCHOSOCIAL INTERVENTIONS ASSESSMENT How to conduct assessments? Two main categories of data collection methods Quantitative Qualitative Surveys, Key informant interviews Questionnaires Focus Group Discussions Psychometric tools Observations (measure reactions, behaviour, feelings, using a scaled measure) 1, 2, 3 Word descriptions =, %, a: b

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Are data needed for these indicators quantitative or qualitative? Logical framework approach Indicators Overall Goal To reduce suffering and risk for development of severe trauma of Population A Adults and children show healthy signs of coping with impact of crisis event Activity 1 Training in PFA Input Resources to enable training Amount of money; personnel; training manuals; materials Output Training has taken place Number of people trained Outcome Volunteers can provide PFA Skill level in PFA has increased Activity 2: Workshops with children Input Resources for workshops Amount of money; personnel; training manuals; materials Output Workshops are held Number of workshops held; number of children attended; number of trainers Outcome Children are coping better Increase in children’s playfulness, self-confidence, trusting of others

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Collecting the data Ethical principals to data collection 1. Well planned and justifiable 2. Coordination 3. Clarifying aims and procedures 4. Participatory and collaborative

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Collecting the data Ethical principals to data collection 5. Comparison groups 6. Conduct and consent 7. Privacy and confidentially 8. Anticipate adverse consequences

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Planning for monitoring and evaluation Group work: What needs to be planned to ensure good monitoring and evaluation?

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Planning for monitoring and evaluation 1. Timing a. Response length (short vs. long term) b. Appropriate time for affected pop. (ex. Do not plan monitoring activities with in-school children during exam time etc) a. Time to plan, implement, analyze, follow up (time and resources should be allocated) 2. Training a. Management on PSS b. Data collectors c. Program team – analysis and reporting

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Planning for monitoring and evaluation 3. Resources a. Staff and volunteers b. Financial c. Time d. Logistics 4. Information sharing a. Internal standards and formats b. External – different audiences (beneficiaries/donors/public)

PSYCHOSOCIAL INTERVENTIONSMONITORING AND EVALUATION Role and importance of M&E in PSS • Helps us understand local – and relevant – meaning of psychosocial wellbeing • Keeps us on the right track • Helps us do our best, now and in the future