Quality Research Standards for Qualitative Inquiry Trustworthiness insert

Quality Research Standards for Qualitative Inquiry: Trustworthiness [insert date] Qualitative Methods in Evaluation of Public Health Programs Session 8

The evaluation process

Learning objectives By the end of the session, participants will be able to: § Discuss the relevance of trustworthiness in qualitative evaluations § Justify the choice of qualitative approach to be applied to a particular evaluation § Develop a plan for establishing trustworthiness in a qualitative component of an evaluation

Differing schools of thought How to assess quality § Rigour and quality of qualitative data should be measured according to the same standards as quantitative data (Morse, et al. ) § Qualitative data should adhere to set standards specifically designed for qualitative research (Kvale, Lincoln, & Guba) § This should be assessed for each individual study

Establishing quality Shaking up the establishment Kvale (1989) questioned the appropriateness of the “holy trinity of science” used in quantitative evaluation when applied to qualitative evaluation. § Reliability § Validity § Generalizability The paradigm behind qualitative evaluation differs from quantitative evaluation. In qualitative evaluation, we acknowledge multiple constructed realities.

Trustworthiness is born Lincoln, Y. , & Guba, E. G. (1985). Naturalistic Inquiry. Newbury Park, CA: Sage. § Credibility § Internal validity § Transferability § Generalizability § Dependability § Reliability § Confirmability § Objectivity

Trustworthiness How we have confidence in our findings § To what extent can we place confidence in the outcomes of the study? § Do we believe what the evaluator has reported?

Trustworthiness Menu of options Credibility 1. Prolonged engagement 2. Referential adequacy 3. Triangulation 4. Peer debriefing 5. Negative case analysis 6. Member checks

Trustworthiness Menu of options Credibility § Prolonged engagement: Multiple points of contact with your participants § Referential adequacy: Method used to store raw data in records to examine later and compare to other future studies to show the credibility of data § Triangulation: Using multiple data sources and methods in an evaluation to ensure the best possible understanding

Trustworthiness Menu of options Credibility § Peer debriefing: Having colleagues review and question your findings and emergent hypotheses to see if they seem reasonable and plausible § Negative case analysis: Data that may indicate that there are different patterns for different groups of participants, or that something is different in a certain case § Member checks: Going back to the participants and asking if they agree with the findings or if there are other explanations for the findings

Trustworthiness Menu of options Transferability Thick/rich descriptions § Participants’ voices § Context § Methodology

Trustworthiness Menu of options Dependability and confirmability § Audit trail o Raw data (notes, transcripts, photos, etc. ) o Data reconstruction and synthesis products (codebook) o Process notes (memos and notes) o Material relating to intentions and dispositions (evaluation diaries and reflexivity)

Criteria for assessing Trustworthiness Question asked Issue Quantitative criteria Qualitative criteria Have we measured what we set Truth value out to measure? Internal validity Credibility How applicable are our results to other subjects and other contexts? External validity Generalizability Transferability Would our findings be repeated Consistency if our evaluation were replicated in the same context with the same subjects? Reliability Dependability To what extent are our findings affected by personal interests and biases? Objectivity Confirmability Applicability Neutrality

A note of caution… You may come across other evaluators who adhere to a different paradigm § Critical of trustworthiness for shifting the responsibility from the evaluators to the reader to decide if the findings are worthy of trust § Most strategies to ensure credibility, transferability, dependability, and confirmability happen in the analytic stage, instead of during the evaluation design or data collection stages (Morse, et al. , 2002)

More strategies Verification strategies for establishing reliability and validity in qualitative research (Morse, et al. , 2002): § Investigator responsiveness § Methodological coherence § Theoretical sampling and sampling adequacy § An active analytic stance § Saturation

Putting quality first Evaluation in the real world Group activity

Putting quality first Evaluation topic Your team is conducting a qualitative evaluation of a violence against women community intervention in Tanzania.

Putting quality first Setting the scene Take it to Tanzania

Putting quality first: Activity E 1 valuation in the real world § Split into three groups § Use the template provided and indicate (30 mins. ): o Your theoretical planning, aspects of trustworthiness o Practical implementation • E. g. , how would you conduct member checks? § Present your plan to the class o 10 mins. each

Class discussion

Summary Rigorous qualitative research adheres to standards of: § Credibility—have we measured what we set out to measure? § Transferability—how applicable are our results to other subjects and other contexts? § Dependability—would our findings be repeated if our evaluation were replicated in the same context with the same subjects? § Confirmability—to what extent are our findings affected by personal interests and biases?

References Lincoln, Y. S. , & Guba, E. G. (1985). Naturalistic inquiry. Thousand Oaks, CA, USA: Sage Publications, Ltd. Mays, N. , & Pope, C. (2000). Qualitative research in health care: Assessing quality in qualitative research, BMJ; (320): 50 – 52. Morse, J. M. , Barrett, M. , Mayan, M. , Olson, K. , & Spiers, J. (2002). Verification strategies for establishing reliability and validity in qualitative research. International Journal of Qualitative Methods; 1(2): 13– 22. Rolfe, G. (2006). Validity, trustworthiness, and rigor: Quality and the idea of qualitative research. Journal of Advanced Nursing; 53(3): 304– 10.

References (continued) Sandelowski, M. (1986). The problem of rigor in qualitative research. Advances in Nursing Science; 8(3): 27– 37. Tong, A. , Sainsbury, P. , & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32 item checklist for interviews and focus groups. International Journal for Quality in Health Care; 19(6): 349– 57. Yin, R. K. (2013). Case study research: Design and methods. Thousand Oaks, CA, USA: Sage Publications, Ltd.

This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L-14 -00004. MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc. ; Management Sciences for Health; Palladium; and Tulane University. Views expressed are not necessarily those of USAID or the United States government. www. measureevaluation. org