8 Validity EDN 6015 F Cheryl HodgkinsonWilliams 3

8. Validity EDN 6015 F Cheryl Hodgkinson-Williams 3 -8 February 2014

1. Trigger problem 2. Context & Rationale 3. Concepts & literature 5. Research Questions 8. Validity 4. Conceptual frameworks & theories 6. Data collection & data analysis 7. Ethics

Once upon a time … • Other people will trust my research findings because I have. . .

Why will people believe your findings? What counts as worthwhile research?

Do we believe this? Why? • “People who weigh themselves daily as part of a diet-and-fitness plan are 82% less likely to regain than those who don’t”. (Readers’ Digest March 2007: 13) • What EVIDENCE do they present to substantiate this claim? What type or quality of evidence would be acceptable?

Quality in research • Key question: – Why should we trust your research? • Key challenges: – Which AUDIENCE are you trying to convince? – What CRITERIA will you select to “ensure” quality? – How might your findings and/or conclusions be incorrect? – What are possible alternative explanations or interpretations? – What strategies will you use to “ensure” quality?

Is this a cake or a biscuit? On what basis did you decide?

Who are you trying to convince? Cake or biscuit? In the UK, value added tax is payable on chocolate covered biscuits, but not on chocolate covered cakes. Mc. Vities defended its classification of Jaffa Cakes as cakes in court, producing a 12" (30 cm) Jaffa Cake to illustrate that its Jaffa Cakes were simply miniature cakes. Mc. Vities argued that a distinction between cakes and biscuits is, among other things, that biscuits would normally be expected to go soft when stale, whereas cakes would normally be expected to go hard. It was demonstrated to the Tribunal that Jaffa Cakes become hard when stale. Other factors taken into account by the Chairman, Potter QC, included the name, ingredients, texture, size, packaging, marketing, presentation, appeal to children, and manufacturing process. Potter ruled that the Jaffa Cake is a cake. Mc. Vities therefore won the case and VAT is not paid on Jaffa Cakes in the UK. [9] However, in 1999, the Sun newspaper published a poll which suggested that the majority of its readers thought of the Jaffa Cake as a biscuit. [ http: //en. wikipedia. org/wiki/Jaffa_Cakes

What does your research intend to do? • MEASURE something? • UNDERSTAND or describe something? • CHANGE the way people act or perceive something? • EXPLAIN something? • This will influence how validity is understood

Purpose • Goal of study Validity • Truth value • Applicability • Consistency • Neutrality MEASURE UNDERSTAND CHANGE EXPLAIN

Purpose • Goal of study MEASURE Validity • Truth value • Applicability • Consistency • Neutrality Internal validity External validity Reliability Objectivity Value-free Methodology • Type of instrument • Observation, questionnaires • Methodologic al approach • Experiments, quasiexperiments, surveys • Type of enquiry • QUANTITATIV E

Purpose • Goal of study MEASURE Validity • Truth value • Applicability • Consistency • Neutrality Internal validity External validity Reliability Objectivity Value-free Methodology • Type of instrument • Observation, questionnaires • Methodologic al approach • Experiments, quasiexperiments, surveys • Type of enquiry • QUANTITATIV E Worldview • Epistemology (knowledge) • Ontology (reality) Positivist • Objectivist – findings are true • World is real & apprehensible

Purpose • Goal of study MEASURE UNDERSTAND Validity • Truth value • Applicability • Consistency • Neutrality Internal validity External validity Reliability Objectivity Value-free Trustworthiness Credibility Transferability Dependability Confirmability Methodology • Type of instrument • Observation, questionnaires • • • Methodologic al approach • Experiments, quasiexperiments, surveys • Type of enquiry • QUANTITATIV E Worldview • Epistemology (knowledge) • Ontology (reality) Positivist • Objectivist – findings are true • World is real & apprehensible • Interview schedules, participative observation schedule, open-ended questionnaires Interviews, participant observation, action research, case studies, ethnography QUALITATIVE Interpretivist / Hermeneutic • Subjectivist created findings • Multiple local

Purpose • Goal of study MEASURE UNDERSTAND CHANGE Validity • Truth value • Applicability • Consistency • Neutrality Internal validity External validity Reliability Objectivity Value-free Trustworthiness Credibility Transferability Dependability Confirmability Self-reflexivity Catalytic validity Democratic participatory validity Methodology • Type of instrument • Observation, questionnaires • • Experiments, quasiexperiments, surveys • • • Methodologic al approach Type of enquiry Worldview • Epistemology (knowledge) • Ontology (reality) • QUANTITATIV E Positivist • Objectivist – findings are true • World is real & apprehensible • Interview schedules, participative observation schedule, open-ended questionnaires Interviews, participant observation, action research, case studies, ethnography QUALITATIVE Interpretivist / Hermeneutic • Subjectivist created findings • Multiple local • • • Interview schedules, participative observation schedule, open-ended questionnaires Interviews, participant observation, discourse analysis QUALITATIVE & quantitative Critical theory • Subjectivist – valuemediated findings • Reality is

Purpose • Goal of study MEASURE UNDERSTAND CHANGE EXPLAIN Validity • Truth value • Applicability • Consistency • Neutrality Internal validity External validity Reliability Objectivity Value-free Trustworthiness Credibility Transferability Dependability Confirmability Self-reflexivity Catalytic validity Democratic participatory validity Descriptive validity Interpretive validity Theoretical validity Generalizability Evaluative validity Methodology • Type of instrument • Observation, questionnaires • • Experiments, quasiexperiments, surveys • • • Methodologic al approach Type of enquiry Worldview • Epistemology (knowledge) • QUANTITATIV E Positivist • Objectivist – findings are • Interview schedules, participative observation schedule, open-ended questionnaires Interviews, participant observation, action research, case studies, ethnography QUALITATIVE Interpretivist / Hermeneutic • Subjectivist - • • • Interview schedules, participative observation schedule, open-ended questionnaires Interviews, participant observation, discourse analysis QUALITATIVE & quantitative Critical theory • Subjectivist – value- • • • Interview schedules, participative observation schedule, open-ended questionnaires Interviews, participant observation, discourse analysis Qualitative & quantitative Critical realist • Objectivist – findings are

Purpose • Goal of study MEASURE UNDERSTAND CHANGE EXPLAIN Validity • Truth value • Applicability • Consistency • Neutrality Internal validity External validity Reliability Objectivity Value-free Trustworthiness Credibility Transferability Dependability Confirmability Self-reflexivity Catalytic validity Democratic participatory validity Descriptive validity Interpretive validity Theoretical validity Generalizability Evaluative validity Methodology • Type of instrument • Observation, questionnaires • • Experiments, quasiexperiments, surveys • • • Methodologic al approach Type of enquiry Worldview • Epistemology (knowledge) • QUANTITATIV E Positivist • Objectivist – findings are • Interview schedules, participative observation schedule, open-ended questionnaires Interviews, participant observation, action research, case studies, ethnography QUALITATIVE Interpretivist / Hermeneutic • Subjectivist - • • • Interview schedules, participative observation schedule, open-ended questionnaires Interviews, participant observation, discourse analysis QUALITATIVE & quantitative Critical theory • Subjectivist – value- • • • Interview schedules, participative observation schedule, open-ended questionnaires Interviews, participant observation, discourse analysis Qualitative & quantitative Critical realist • Objectivist – findings are

Validity differently understood in various traditions • “… theories and intellectual traditions you are drawing on in your research will have implications for what validity threats you see as most important and vice versa” (Maxwell 2005: 4)

Research orientations, traditions or paradigms • “Scientific research paradigms are overall conceptual frameworks within which some researchers work, that is, a paradigm is a world-view or ‘a set of linked assumptions about the world which is shared by a community of scientists investigating the world’” (Deshpande, 1983 cited in Healy & Perry 2000: 118 ).

Research orientations / traditions • There is no general consensus as to the exact categorisation of research orientations, traditions or paradigms, but 4 broad categories are usually mentioned: – Positivism – Interpretivism or Hermeneutics or Constructivism – Critical theory – Critical realism

Revisiting some Positivist assumptions Researchers who maintain that a real world exists and there is a correct way of describing it are also likely to believe that worthwhile research can demonstrate: • • Internal validity (Congruency with reality) External validity (Generalisability) Reliability (Repeatability or replicability) Objectivity

Value-neutral? • The attempt to produce value-neutral social science is increasingly being abandoned as at best unrealizable, and at worst self-deceptive, and is being replace by social sciences based on explicit ideologies (Hesse 1980 cited in Lather 1986: 63).

Alternative to validity Validity Reliability (Guba & Lincoln 1994) Trustworthiness

Alternative criteria • Credibility Trustworthiness • Transferability • Dependability • Confirmability

Positivist vs Interpretivist (adapted from Janse van Rensburg (n. d. : 9) Criterion Positivist Interpretivist Truth value Internal validity Credibility Applicability External validity Transferability (Generalisability) Consistency Reliability Dependability Neutrality Objectivity Confirmability

Some strategies for establishing validity (adapted from Janse van Rensburg (n. d. 9) Criterion Strategies Credibility Prolonged & varied field experience Time sampling Triangulation Member checking Transferability Nominated sampling Time sampling Dense description or “thick description” Dependability Audit trail Stepwise replication Peer examination Confirmability Audit trail Reflexivity

Validity criteria in critical theory /postmodern research traditions • Self reflexivity – making one’s standpoint explicit (see Lather 1986) • Catalytic validity - establishing to what extent one’s research has brought about change (see Lather 1986) • Democratic participatory validity – what are the extent of the opportunities created for democratic forms of participation in the research process, the extent to which participants actually come to own and shape the different parts of the research process in its entirety (Vithal n. d. 3)

Validity in critical realist tradition • • • Descriptive validity Interpretive validity Theoretical validity Generalizability Evaluative validity

Descriptive validity • This concerns that factual accuracy of [the] account – Primary descriptive validity • … the descriptive validity of what the researcher reports having seen or heard (or touched, smelled, and so on) (Maxwell 1992: 286) – Secondary descriptive validity • … the validity of accounts of things that could in principle be observed, but that were inferred from other data

Interpretive validity • Interpretive validity is inherently a matter of inference from the words and actions of participants in the situations studied (Maxwell 1992: 290). • Interpretive validity does not apply only to the conscious concepts of participants; it can also pertain to the unconscious intentions, beliefs, concepts, and values of these participants, and to what Argyris and Schoen (1974) call “theory-in use, ” as opposed to “espoused theory” (Maxwell 1992: 290).

Theoretical validity • Theoretical validity thus refers to an account’s validity as a theory of some phenomenon (Maxwell 1992: 291). • This theory can refer to either physical events or mental constructions. It can also incorporate participants’ concepts and theories, but its purpose goes beyond simply describing these participants’ perspectives … [The] theoretical understanding refers to an accounts’ function as an explanation (Maxwell 1992: 291).

Generalizability • Generalizability refers to the extent to which one can extend the account of a particular situation or population to other persons, times, or settings than those directly studied • Internal generalizability – generalizing which the community, group, or institution studied to persons, events, and settings that were not directly studied • External generalizability – generalizing to other communities, groups or institutions (Maxwell 1992: 293)

Evaluative validity • … it involves the application of an evaluative framework to the objects of study (Maxwell 1992) • It involves some kind of judgment about what is legitimate or justified.

Coherency across design • “Design in qualitative research is an ongoing process that involves “tacking” back and forth between the different components of the design, assessing the implications of goals, theories, research questions, methods and validity threats for one another” (Maxwell 2005: 3)

Validity • Validity is not a commodity that can be purchased with techniques. … Rather, validity is like integrity, character and quality, to be assessed relative to purposes and circumstances (Brinberg and Mc. Grath 1985 cited in Maxwell 1992: 280 -281) .

Written by Cheryl Hodgkinson-Williams cheryl. hodgkinson-williams@uct. ac. za in 2013 and adapted in 2014 This work is licensed under the Creative Commons Attribution-Share Alike 2. 5 South Africa License. To view a copy of this license, visit http: //creativecommons. org/licenses/bysa/2. 5/za/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.

References • Healy, M. & Perry, C. (2000). Comprehensive criteria to judge validity and reliability of qualitative research within the realism paradigm. Qualitative Market Research: An International Journal, 3(3), 118 -126. • Janse van Rensburg (n. d. ). They say size doesn’t matter … Criteria for judging the validity of knowledge claims in research. Rhodes Environmental Education Unit Research Methods Course. • Lather, P. (1986). Issues of validity in openly ideological research: Between a rock and a soft place. Interchange, 17(4), 63 -84. • Maxwell, JA (2008) Designing a qualitative study. In Bickman, L & Rog, D. J (2008) The Sage handbook of applied social research methods, London: Sage. Available online: http: //www. corwin. com/upm-data/23772_Ch 7. pdf (Last accessed 24 January 2009).

• Maxwell, J. A. (2005). Qualitative research design: An interactive approach (2 nd ed. ). Thousand Oaks: Sage Publications. • Maxwell, J. A. (1992). Understanding and validity in qualitative research. Harvard Educational Review 62(3), 279 -300. • Seale, C. (2004). Quality in qualitative research. In C. Seale, G. Gobo, J. F. Gubrium & D. Silverman Qualitative Research Practice. London: Sage. • Vithal, R. (n. d). In search for criteria of quality and relevance for mathematics education research: The case of validity. In S. Maholmaholo (Ed. ). Proceedings of the 8 th Annual Conference of the Southern African Association for Research in Mathematics and Science Education (SAARMSTE). University of Port Elizabeth, January 19 -22, 567 -573.

Purpose • Goal of study Validity • Truth value • Applicability • Consistency • Neutrality Methodology • Type of instrument • Methodological approach • Type of enquiry MEASURE UNDERSTAND CHANGE EXPLAIN Internal validity External validity Reliability Objectivity Value-free Trustworthiness Credibility Transferability Dependability Confirmability Self-reflexivity Catalytic validity Democratic participatory validity Descriptive validity Interpretive validity Theoretical validity Generalizability Evaluative validity • • • Observation, questionnaires Experiments, quasiexperiments, surveys QUANTITATIVE • • • Worldview • Epistemological assumptions (knowledge) • Ontological assumptions (reality) Positivist • Objectivist – findings are true • World is real & apprehensible Interview schedules, participative observation schedule, open -ended questionnaires Interviews, participant observation, action research, case studies, ethnography QUALITATIVE Interpretivist / Hermenutic • Subjectivist - created findings • Multiple local constructed reality • • • Interview schedules, participative observation schedule, open -ended questionnaires Interviews, participant observation, discourse analysis QUALITATIVE & quantitative Critical theory • Subjectivist – valuemediated findings • Reality is shaped by values • • • Interview schedules, participative observation schedule, open -ended questionnaires Interviews, participant observation, discourse analysis Qualitative & quantitative Critical realist • Objectivist – findings are probably true • World is real & imperfectly apprehensible