VALIDITY AND RELIABILITY LOUIS COHEN LAWRENCE MANION AND
VALIDITY AND RELIABILITY © LOUIS COHEN, LAWRENCE MANION AND KEITH MORRISON © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
STRUCTURE OF THE CHAPTER • Defining validity • Validity in quantitative, qualitative and mixed methods research • Types of validity • Triangulation • Ensuring validity • Reliability in quantitative and qualitative research • Validity and reliability in interviews, experiments, questionnaires, observations, tests, life histories and case studies © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
WHAT IS VALIDITY? A demonstration that a particular instrument measures what it intends, purports or claims to measure. A demonstration that an account accurately represents those features that it is intended to describe, represent, explain or theorise. The extent to which interpretations of data are warranted by theories and evidence used. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
BASES OF VALIDITY IN QUANTITATIVE RESEARCH Controllability BASES OF VALIDITY IN QUALITATIVE RESEARCH Natural Isolation, control, manipulation of variables Replicability Predictability Thick description Generalizability Context freedom Fragmentation and atomization Uniqueness Context boundedness Holism Randomization of samples Purposive sample/no sampling Neutrality Value ladenness of observations Objectivity Observability Uniqueness Emergence, unpredictability Confirmability Observable and non observable meanings/ intentions Inference Description, inference, explanation ‘Etic’ research ‘Emic’ research Observations© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; Meanings individual chapters, the contributors
BASES OF RELIABILITY IN QUANTITATIVE RESEARCH BASES OF RELIABILITY IN QUALITATIVE RESEARCH Reliability Dependability Demonstrability Stability and replicability Parallel forms Context freedom Objectivity Coverage of domain Verification of data and analysis Answering research questions Meaningfulness to the research Parsimony Trustworthiness Stability and replicability Parallel forms Context specificity Authenticity Comprehensiveness of situation Honesty and candour Depth of response Meaningfulness to respondents Richness © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
BASES OF RELIABILITY IN QUANTITATIVE RESEARCH Objectivity Internal consistency Generalizability Inter rater reliability BASES OF RELIABILITY IN QUALITATIVE RESEARCH Confirmability Credibility Transferability Inter rater reliability Triangulation Accuracy and precision Neutrality Consistency Alternative forms (equivalence) Split half Triangulation Accuracy and comprehensiveness Multiple interests represented Consistency Inter item correlation © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY IN QUANTITATIVE AND QUALITATIVE RESEARCH Validity in quantitative research often concerns: objectivity, generalizability, replicability, predictability, controllability, nomothetic statements. Validity in qualitative research often concerns: honesty, richness, authenticity, depth, scope, subjectivity, strength of feeling, catching uniqueness, idiographic statements. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
TYPES OF VALIDITY • • • Catalytic • Ecological Concurrent • Evaluative Consequential • External Construct • Face Content • Internal Criterion related • Interpretive Convergent and discriminant • Jury Cross cultural • Predictive Cultural validity • Systemic Descriptive • Theoretical © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY IN QUANTITATIVE RESEARCH • • • Concurrent Construct Content Criterion related Convergent and discriminant Cross cultural • • Evaluative External Face Internal Jury Predictive Theoretical © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY IN QUALITATIVE RESEARCH Credibility • The truth value (replacing the quantitative concepts of internal validity) Transferability • Generalizability (replacing the quantitative concept of external validity) Dependability • Consistency (replacing the quantitative concept of reliability) Confirmability • Neutrality (replacing the quantitative concept of objectivity) © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
MAXWELL’S VALIDITY IN QUALITATIVE RESEARCH (MAXWELL) Descriptive validity • the factual accuracy of the account, that it is not made up, selective, or distorted; objectively factual) and credible Interpretive validity • the ability of the research to catch the meaning, terms, interpretations, intentions that situations and events, i. e. data, have for the participants/ subjects themselves, in their terms Theoretical validity • theoretical constructions that the researcher brings to the research theoretical validity is the extent to which the research © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors explains phenomena; construct validity
MAXWELL’S VALIDITY IN QUALITATIVE RESEARCH (MAXWELL) Generalizability • the view that theory generated may be useful in understanding other similar situations; generalizing within specific groups or communities, situations or circumstances and, beyond, to outsider communities, situations or circumstance Evaluative validity • the application of an evaluative, judgmental stance towards that which is being researched, rather than a descriptive, explanatory or interpretive framework. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY IN MIXED METHODS RESEARCH • • • • Representation Legitimation Sample integration Inside–outside Weakness minimization Sequential Conversion Paradigmatic mixing Commensurability Multiple validities Political Integration (of methods) © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors Integration
INTERNAL VALIDITY Demonstration that the explanation of a particular event, issue or set of data which a piece of research provides can actually be sustained by the data and the research. The findings must describe accurately the phenomena being researched. Truth value and credibility of interpretations and conclusions. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
HISTORY MATURATION TESTING DIRECTION OF CAUSALITY TYPE I AND TYPE II ERRORS INSTRUMENTATION THREATS TO VALIDITY AND RELIABILITY EXPERIMENTAL MORTALITY OPERATIONALIZATION CONTAMINATION REACTIVITY © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
INTERNAL VALIDITY IN QUALITATIVE RESEARCH • Confidence in the data • Authenticity of the data (the ability of the research to report a situation through the eyes of the participants) • Cogency of the data • Soundness of the research design • Plausibility of the data and interpretation • Credibility of the data • Auditability of the data • Dependability of the data • Confirmability of the data • Clarity on the kinds of claim made from the research (e. g. definitional, descriptive, explanatory, theory generative) © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
EXTERNAL VALIDITY The degree to which the results can be general ized to the wider population, cases, settings, times or situations, i. e. the transferability of the findings. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
ESTABLISHING VALIDITY IN QUALITATIVE RESEARCH • • • Prolonged engagement in the field Persistent observation Triangulation Leaving an audit trail Respondent validation Weighting the evidence (giving priority) Checking for representativeness Checking for researcher effects Making contrast/comparisons Theoretical sampling Checking the meaning of outliers Using©extreme cases 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
ESTABLISHING VALIDITY IN QUALITATIVE RESEARCH • • • Ruling out spurious relations Replicating a finding Referential adequacy Following up surprises Structural relationships Peer debriefing Rich and thick description Looking for possible sources of invalidity Assessing rival explanations Negative case analysis Confirmatory data analysis Effect©sizes 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
THREATS TO VALIDITY IN QUANTITATIVE RESEARCH • • • History Maturation Statistical regression Testing Instrumentation Selection bias Experimental mortality Instrument reactivity Selection–maturation interaction Type I and Type II errors © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY PROBLEMS IN CROSS CULTURAL RESEARCH • Failure to operationalize elements of cultures. • Whose construction of ‘culture’ to adopt: ‘emic’/‘etic’. • False attribution of causality to cultural factors rather than non cultural factors. • Directions of causality. • Ecological fallacy. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
VALIDITY PROBLEMS IN CROSS CULTURAL RESEARCH • • • Sampling and instrumentation. Convergent and discriminant validity. Response bias and preparation of participants. Language problems. Problems of equivalence (conceptual, psychological, meaning, instrument, understanding, significance, relevance, measurement, linguistic). © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
THREATS TO EXTERNAL VALIDITY IN QUANTITATIVE RESEARCH Failure to describe independent variables explicitly. Lack of representativeness of available and target populations. Hawthorne effect. Inadequate operationalizing of dependent variables. Sensitization/reactivity to experimental/research conditions. Interaction effects of extraneous factors and experimental/ research treatments. • Invalidity or unreliability of instruments. • Ecological validity. • Multiple treatment validity. • • • © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
THE HAWTHORNE EFFECT • Date – Between 1927 and 1932 researchers carried out experiments at the Western Electric Company’s Hawthorne plant. • Purposes – To examine the effects of changes to working conditions on output of workers • Sample – Six women, chosen as average workers • Method – Women worked in a test room. Output measured under different conditions (e. g. no change → change to method of payment → introduce two rest periods → introduce six rest periods → changes in lighting conditions, early clocking off, five day working week → return to initial conditions) • Duration © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors – 15 weeks
THE HAWTHORNE EFFECT Results: • Output rose steadily during test period and after the test period. Conclusion: • Output did not seem to depend on test conditions. Increased output seemed to be due to the fact that the people had been involved in the experiment itself, i. e. the act of research had affected the results. The results were a research of the research itself. Implications: • The act of being involved in research itself affects the results. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
THREATS TO EXTERNAL VALIDITY IN QUALITATIVE RESEARCH • • Selection effects Setting effects History effects Construct effects © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
ENSURING VALIDITY AT THE DESIGN STAGE • • Choose an appropriate timescale Ensure adequate resources for the research Select appropriate methodology Select appropriate instruments Use an appropriate sample Ensure reliability Select appropriate foci Avoid having biased researcher(s) © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
• • • ENSURING VALIDITY AT THE DATA COLLECTION STAGE Reduce the Hawthorne effect Minimize reactivity Avoid dropout rates amongst respondents Take steps to avoid non return of questionnaires Avoid too long or too short an interval between pre tests and post tests Ensure inter rater reliability Match control and experimental groups Ensure standardized procedures for gathering data Build on the motivations of respondents Tailor instruments to situational factors Address© 2018 researcher characteristics Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
ENSURING VALIDITY AT THE DATA ANALYSIS STAGE • • • Use respondent validation Avoid subjective interpretation of data Reduce the halo effect Use appropriate statistical treatments Recognize extraneous factors which may affect data Avoid poor coding of qualitative data Avoid making inferences/generalizations beyond the data Avoid equating correlations and causes Avoid selective use of data Avoid unfair aggregation of data Avoid degrading the data Avoid Type I and/or Type II errors © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
ENSURING VALIDITY AT THE REPORTING STAGE • • Avoid using data selectively and unrepresentatively Indicate the context and parameters of the research Present the data without misrepresenting the message Make claims which are sustainable by the data Avoid inaccurate or wrong reporting of data Ensure that the research questions are answered Release research results neither too soon nor too late © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RELIABILITY • Reliability is an umbrella term for dependability, consistency and replicability over time, over instruments and over groups of respondents. • Can we believe the results? • Can we trust the results? • Reliability is concerned with precision and accuracy. • Reliability is concerned with consistency. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RELIABILITY IN QUANTITATIVE AND QUALITATIVE RESEARCH Reliability in quantitative research • consistency (stability), accuracy, predictability, equivalence, replicability, concurrence, descriptive and causal potential. Reliability in qualitative research • accuracy, fairness, dependability, comprehensiveness, respondent validation, ‘checkability’, empathy, uniqueness, explanatory and© 2018 descriptive confirmability. Louis Cohen, Lawrencepotential, Manion and Keith Morrison; individual chapters, the contributors
RELIABILITY IN QUANTITATIVE RESEARCH Reliability as stability • Consistency over time and samples Reliability as equivalence • Equivalent forms of same instrument; • Inter rater reliability; Reliability as internal consistency • Split half reliability (e. g. for test items) © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
TRIANGULATION • • • Paradigms Methodologies Instruments Researchers Time Location Theories Samples Participants Data © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
SPLIT HALF RELIABILITY (Spearman Brown) Reliability = r = the actual correlation between the two halves of the instrument; Reliability = = = 0. 919 © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RELIABILITY IN QUALITATIVE RESEARCH • • Credibility Neutrality Confirmability Dependability Consistency Applicability Trustworthiness Transferability © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RELIABILITY AND REPLICATION IN QUALITATIVE RESEARCH Repeat: • The status position of the researcher • The choice of informants/respondents • The social situations and conditions • The analytic constructs used • The methods of data collection and analysis Address: • Stability of observations • Parallel forms • Inter rater reliability © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors • Respondent validation
IMPROVING RELIABILITY Minimise external sources of variation Standardise conditions under which measurement occurs Improve researcher consistency Broaden the sample of measurement questions by: • adding similar questions to the instrument; • increasing the number of researchers (triangulation); • increasing the number of occasions in an observational study. Exclude extreme responses (outliers) © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
RELIABILITY AND VALIDITY AT ALL STAGES • • Design and methodology Sampling Instrumentation Timing Data collection Data analysis Data interpretation Reporting © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
- Slides: 39