Program Evaluation Quantitative Vs Qualitative Approaches India CLEN

Program Evaluation : Quantitative Vs. Qualitative Approaches India. CLEN Program Evaluation Network Narendra K. Arora Department of Pediatrics All India Institute of Medical Sciences New Delhi narendrakumararora@hotmail. com Program Evaluation- Chandigarh 29 th February 2004

Research is not being translated into policy Evidence “ The dominant finding of our review is that there are large gaps between the care people should receive and the care they do receive. This is true for all types of care - preventive, acute, and chronic [. . . ] for different types of health insurance […] for all age groups, from children to the elderly […] ” Schuster et al, Milbank Quarterly, 1998

Research vs. Policy & Practice Research FOR policy Priority setting for health research Program design Program monitoring and evaluation Practice guidelines; Rational use of drugs Needs assessment: introduction of new technology & programs Research ON policy Privatization of health services Clinical Audit

Research and Policy & Practice Policy for Research Promotion of Research Practice (based) on Research Evidence based practice

Research and Policy & Practice Research Publications Natural Passive Diffusion of Information Changes in Policy & Practice

Research and Policy & Practice Information alone is insufficient for changes in policy and practice Researchers are not systematically involved in implementation of their own findings Lack of policy relevant evidence Scientific evidence vs. need of policy makers Guidelines Not commensurate with results Not understandable by Policy makers

Conventional Research Policy Driven Research Implement Policy & Practice Priority Issues Implement Policy & Practice

Components of a Program Policy: Needs & Rationale Policy: Aims & Goal Program: Strategy Planning: Implementation

Program Evaluation • A systematic effort to describe the status of a program • Extent to which program objectives achieved

Consumers of Evaluation Research Government Program Managers Policy Makers Funding Agencies Health Care Research Institutions Clinicians Community Identifying the stakeholders

Use of Program Evaluation Data Policy Makers / Program managers – Needs Assessment – Redefining aims / objectives – Modifying or fine tuning strategies (process) – Sustainability (including fatigue factor) – Judge the worth (impact) – Expense / cost – Interaction with other activities / health systems Needs of Policy Makers

Use of Program Evaluation Data Academia - Wider application of program strategies / Cross country / culture – Unique features (success/failure) – Determinants of provider and client behavior

Types of Evaluation Needs Assessment To identify • Goals • Products • Problems • Conditions

Types of Evaluation Contd… Formative (Process) Evaluation To improve developing or ongoing program Role as helper/advisor/planner Progress in achievements Potential problems/needs for improvements in • Program Management • Inter-sectoral coordination • Social mobilization Implementation Outcomes

Types of Evaluation Contd… Summative (Coverage) Evaluation (To help decide ultimate fate) Summary statement about Program’s achievements Unanticipated Out comes Comparison with other programs

Qualitative Vs Quantitative Research - the dichotomy Qualitative Quantitative Social theory Methods Action Observation, Interview Structure Experiment, Survey Question What is X ? (classification) Inductive How many Xs? (enumeration) Deductive Reasoning Sampling method Theoretical Statistical Strength Reliability Validity

Qualitative Research: Key Strengths • Qualitative methods aim to make sense of, or interpret, phenomena in terms of the meanings people bring to them • Qualitative research may define preliminary questions which can then be addressed in quantitative studies • A good qualitative study will address a clinical problem through a clearly formulated question & using more than one research method (triangulation) • Analysis of qualitative data can & should be done using explicit, systematic, & reproducible methods

Potential limitation of Qualitative Methods • • Accuracy of the information representative nature Cultural inappropriateness Subjectivity of the Investigator

Qualitative Research Methods Findings - Presented alone / in combination with quantitative data - Validity and reliability depends on methodological skills, sensitivity, integrity of the researchers - Skillful interviewing - more than just asking questions - Content analysis - more than just reading to see what’s there - Generate useful and credible findings through observation, interviewing and content analysis How? - Discipline, knowledge, training, practice, creativity,

Multidisciplinary Central Coordinating Team Program Evaluation Expert Health Social Scientist Epidemiologist Clinicians Program Evaluation Economist Biostatistician Anthropologist

Health Program Evaluation - Quantitative Research Methods Approach - Measures the reaction of a great many people to a limited set of questions - Comparison and statistical aggregation of the data - Broad, generalizable set of findings presented succinctly and parsimoniously.

Multi-Centric Evaluation Studies • Cohesive network of partners • • Common understanding of the program • Common understanding of aims & objectives of evaluation exercise • Standardization of research instruments • Standardization of protocol implementation at various sites • Regional variation in program implementation • Multiple layers of quality assurance measures Generalizable and hence more confidence

Challenges in Delivery of ‘Routine Public Health Programs’ in India: Vitamin-A and Iron folic acid Supplementation Program(s) - a case study Supported by Ministry of Health & Family Welfare Government of India, The Micronutrient Initiative, Canada, INCLEN, USAID & AIIMS

Characteristics of A Routine Program?

“We are not bothered about those we could not reach because they are not deficient in Vitamin-A. Otherwise they would have come with night blindness in OPD. ” District level provider (8001): Kolkata “Those who come to us we give them. Clients not coming must be healthy. ” Government health worker (1006): Ballabgarh “We are already over burdened. What can we do there? ” Government health worker (11013): Nagpur

Passiveness in Implementation

Resistance to Polio Drops among Clients

Who are Resistant/Reluctant Clients Rumours • • • Rationale and safety of repeated dosing not clear Rumors regarding vaccine safety (sterility/HIV-AIDS) AFP cases occurring in children who had received OPV drops in the past "We heard that the only purpose (of giving these drops to our children) is that they cannot produce children in future. " Client (Non-utilizer Men) "Cold (OPV) drops will make them (children) cold (no sexual arousal). People believe in rumors that children will become impotent. " Client (Non-utilizer Women)

Resistant/Reluctant Clients Social Mobilization Strategy • Use of force – counter productive/‘fatal’ “Police came and they forcibly gave us the drops. ” Client (Non-utilizer Men) “I have seen it myself that force was not used on Hindus but when they (health workers) approached Muslim houses, they used force to give drops to their children. ” Client (Non-utilizer Men)

“There is no need to use force (for good things like polio drops). If you try to explain by compassion, people will understand (accept it). Suppose you give me something to eat and I refuse to oblige, if you pick up a stick and force (threaten) me to do it, it invites suspicion that something is fishy about it and that is why I am being forced to eat. If you offer with love, I will take it. This problem exists everywhere. ” Client (Male Non-utilizer) "If a Mohammedan doctor or a priest (Maulvi) comes and explains, people will understand give the drops. Otherwise people will think that government is asking us to drink poison (drink. . . . ). " Client (Non-utilizer Men)

“This is an unprecedented event where all people irrespective of caste, creed and religion take part in PPI program on the same day (NID) throughout the country” • Health worker (150): Burdwan “He (my husband) told me that everybody is going for polio drops. Then why should we be left out ? After all, everybody is not a fool” • Utilizer (1422): Delhi

Quotable Quotes • Give a vivid, meaningful flavor which is far more convincing than pages of summarized numbers - These should not be distracters Should not take the reader away from the real issues in hand

Reasons for Non-utilization of FHAC Program Services: Process Evaluation Provider perspective NGO/ leader Perspective Client perspective Lack of awareness about FHAC 1+ 1+ 4+ Low motivation 2+ 2+ 1+ 2+ to 3+ 2+ 1+ Adversities (facilitatory factors) 1+ <1+ Client Conveniences * Season / month * Distance * Timing of camp <1+ 1+ 2+ to 3+ 1+ to 2+ 1+ 3+ Camp facilities 1+ 1+ 1+ Low credibility 1+ <1+ Reasons Shyness / Embarrassment Semi Quantitative Qualifiers: 1+ Some/a few, 2+ About half, 3+ A majority, 4+ Most, 5+ Almost all India. CLEN FHAC Program, 2000 : Process Evaluation

Reasons for Non-participation in FHAC Camps by the Clients (RURAL) Reasons for Nonparticipation Men % (95% CI) Women % (95% CI) Total 82. 3 (79. 9 -84. 8) 79. 7 (76. 9 -82. 5) 81. 0 (78. 7 -83. 4) Indifferent 2. 4 (1. 6 -3. 4) 2. 6 (1. 9 -3. 4) 2. 5 (1. 9 -3. 1) Embarrassed 0. 3 (0. 0 -0. 6) 1. 3 (0. 8 -1. 8) 0. 8 (0. 5 -1. 2) Not aware about camp 0. 8 (0. 4 -1. 3) 0. 7 (0. 3 -1. 0) 0. 7 (0. 4 -1. 0) Need not felt 2. 3 (1. 6 -3. 0) 5. 6 (4. 4 -6. 8) 3. 9 (3. 2 -4. 7) Inconvenient timings 2. 0 (1. 1 -3. 0) 2. 4 (1. 2 -3. 5) 2. 2 (0. 5 -2. 9) Loss of wages (affordability) 8. 0 (6. 5 -9. 4) 5. 8 (4. 5 -7. 3) 7. 0 (5. 8 -8. 1) Not in village (availability) 1. 7 (1. 0 -2. 3) 1. 6 (1. 0 -2. 1) 1. 6 (1. 2 -2. 1) Not aware about FHAC program India. CLEN FHAC Program, 2000 : Coverage Evaluation

What is a Safe Injection? • All injections are safe • Right drug/ good medicine/ expiry of drug has been checked/ right dose/specific drugs are safe/skin sensitivity test should be done before giving injection/ drug dissolve in muscles/drug should be tested/injection should be given with glucose/ single dose vial to be used • Use new plastic syringes/ syringe & needle from a sealed pack / expensive syringe/needle/boiled syringe and needle/sterilized glass syringes & needle/one needle and syringe for one patient/ use of disposable syringes / needles / ISI mark/avoid reuse chemical stylizations of syringes & needle/ new needle from the sealed pack • Needle and syringe – clean / proper/ flushing of syringes and needle using spirit / water/steel needles. • Technique of giving injection: gives slowly/properly/painless injection/proper site of giving injection/giving intramuscularly is safe

Conti …. What is a Safe Injection? • Before injection ask patient to roll up shirt/ clean the injection site with spirit swab • Injection giver wash hands/ use gloves/observe sterile precaution • Injection taken from recognized hospital • Injection given when necessary/ rational use • Disease should get completely cured/ maximum curative effect • Prevention of disease by immunization • Injection that prevent blood borne diseases • Does not cause reaction • Needle and syringe flushed & cleaned before disposal/ syringe thrown away • Doctors know about safe injection • Safe to reuse syringes/ needles to the same person • No injection is safe • Clients should be made aware about the safe injection

When do you (Prescriber) give an injection? • Initial stages of diseases, for quick relief • Patient not improving with oral drug, For serious diseases • Side effects of oral drugs • Doctor decides, Prescriber prefer injection • Doctor wants to earn money • Type of disease • In case of emergency • Better compliance to injection • To restore energy • Patient demand Injection • Certain categories of patients (poor, uneducated, rural patient) • Those who can afford injection • Vaccines and immunization • No preference- both oral & injections are equally effective • Supply is more/ date of expiry is near

Results Reach of Family Health Awareness Campaign Program Rural Areas 73% Not aware about FHAC 7% 2% 7% 11% Aware but not attended camp Contacted at home alone Urban Slums 82% Attended camp only Contacted at home & attended camp 3% 1% 9% 5% India. CLEN FHAC Program, 2000 : Coverage Evaluation

Qualitative Vs. Quantitative • It is not qualitative vs. quantitative but qualitative and quantitative • Mechanism and dynamics of events captured by qualitative methods • Range of possible answers expanded through qualitative methods and then relative significance determined by quantitative methods • Combination of methods make the results as well as recommendations more meaningful and operational

Concluding Remarks Evaluators express their opinions explicitly - Based on evidence gathered - Consistent quality assurance measures - Limitations of methods accepted up front - Politics of evaluation - Remains a scientific endeavor - Efforts to overcome barriers (keep in mind the perspectives & perceptions of the stakeholders)

An India. CLEN Program Evaluation Network 2003 -04 • Srinagar Amritsar Kangra • • Chandigarh¨ Dehradun • • Ghaziabad • • • ¨ • • • ¨ • • • ¨ • • • ¨ • • • • • Network Partners(n==84) • ¨ • • • • • ¨ ¨ Rohtak Aligarh Jodhpur Gangtok Jaipur Agra Lucknow Muzaffarpur Guwahati Dibrugarh(2) Barmer Ajmer Darbhanga Chittorgarh Shillong Kohima Udaipur Gwalior Varanasi Gaya Patna Imphal Bhagalpur Rajkot Agartala Bhuj Ranchi Burdwan Aizwal Bhopal Jabalpur Jamshedpur Jamnagar Kolkata(4) Baroda Raipur Bilaspur Cuttack Nagpur Bhavanagar Sambhalpur Bhubaneshwar Surat Berhampur Mumbai (2) Pune Hyderabad(2) Visakhapatnam Kakinada Gulbarga. Vijayawada Panaji Bijapur Belgaum Kurnool Manipal Bangalore Chennai Vellore Medical Colleges 69 Calicut Coimbatore Kannur Madurai Thrissur NGOs 9 Thirunelveli Kottayam Thiruvananthapuram Pub Health Inst 6

India. CLEN Program Evaluation Network [IPEN] VISION Facilitate development and implementation of people friendly, effective Public Health Programs that are in harmony with their socio-cultural beliefs, thus creating a milieu where service providers get motivated and communities consider Public Health Services beneficial and willingly participate