Research for Social Workers Elspeth Slayter Ph D

Research for Social Workers Elspeth Slayter, Ph. D. Assistant Professor, Salem State University Lecture notes on threats to validity, threats to trustworthiness and the optimization of rigor in social work research

Today’s class • • Administrative matters/Check-in Tools for critical thinking in “research-y language” • Qualitative work: • Threats to trustworthiness; Optimization of rigor • Quantitative work: • Threats to validity

Some generalizations about differences between the methods QUALITATIVE • Concerned with how people think and feel about the topics of concern to the research • Gather broader, more in-depth information from fewer respondents (micro-analysis) • Open questions for greater depth and personal detail QUANTITATIVE • • • Use a structured survey instrument that asks all respondents the same questions in the same order to allow for statistical analysis Gather a narrow amount of information from a large number of respondents (macro-analysis Closed questions for quantification, can be coded and processed quickly

QL a different approach, not the easy way out • • • Not “opposite” of QN – a different way to answer different questions Different underlying assumptions about how individual/group behavior is best studied Reflects a systematic approach to the conduct of research

Optimization of rigor/validity in qualitative research QN: Threats to validity QL: Optimization of rigor/trustworthiness

Optimization of rigor/validity in qualitative research • • Rejection of threats to validity vs. evaluative standards, strategies for rigor Flexibility of design - need for transparency • As subjectivity is embraced – need to document • Careful reporting on implementation of approaches • Note-taking, auditing, memo-ing • REFLEXIVITY – self-awareness in process

Optimization of rigor/validity in qualitative research • Threats to trustworthiness • Reactivity: Distorting effects of researcher’s presence on respondent beliefs/behaviors • Researcher biases: Observations/Interpretations clouded by researcher’s worldview • Respondent biases: Social desirability response to withholding information

SIX STRATEGIES FOR ENHANCING RIGOR 1. 2. Member Checking: Empowerment of respondents, keeps you honest Negative Case Analysis: Testing and challenging interpretations to date 1. Fear management paper example 2. Searching for disconfirming evidence 3. Audit Trail: Replicability, accountable for interpretive decisions

SIX STRATEGIES FOR ENHANCING RIGOR 4. 5. 6. Triangulation: Theory, method, observer, data Prolonged Engagement: Goal is saturation Peer Debriefing and Support (PDS) 1. Keeping us honest in interpretation 2. Emperor’s new clothes check

Example: Fear management study • • Interviews followed by memos Individual • Listening to tapes • Reading of transcripts • Coding of memos, transcripts for themes • Team • Comparison of each transcript • Write up of themes identified • Member checking • Review of transcripts, write-up

Review: Addressing skeptical reviewers 1. 2. 3. 4. 5. 6. How can this study be generalizable with low n (sample size)? Where are your hypotheses? Why isn’t this study more objective? Will attrition be a problem with so few in the study? Why can’t you be more forthcoming about what your findings will look like in advance? How is this different from journalism?

Thinking about threats to validity in quantitative research Tools for critical thinking, names

Validity in the quantitative world �Must specify type �Refers to the accuracy and trustworthiness of: • instruments (surveys, variables) • data (and how it was gathered) • findings

Validity depends on 2 things: 1) The utility of the device or method that measures the issue at hand 2) The collective judgment of the research community that a concept, measure and method are valid

External validity: Generalizability �Are results of the study applicable to settings other than the one in which the study was conducted? �Can be addressed by using: • Large population samples • Many control or comparison groups • Replicating the study in a different setting or with a different population

Limited generalizability is not always a bad thing Why?

Thinking about your projects: �What are the challenges to external validity in your project? �Who is your study generalizable to?

Face validity �Looking at how a concept was operationalized and deciding whether or not “on the face of it” the measurement makes sense �Generally based upon consensus by researchers • Example: Measuring prevalence of child abuse in families through parent interviews

Content validity � Achieved when a measurement has the appropriate content for “getting at” all the issues present in a complex construct � Think “inner mechanics” � Examples: • Socioeconomic status • Quality of life • Client satisfaction

Construct validity �Close fit between the construct being measured and the actual observations made �D �o your questions (and therefore variables) adequately “get at” the �construct your are measuring? �Examples: • Does the IQ test measure intelligence? • Does income alone measure SES?

Content validity vs. construct validity Content � Technical presence of all theory-pieces of the measure in question Construct � Overall ‘vibe’ about whether the measure really gets at what it intends to � Does � Do it have all of the parts? � Presence of the parts make sense as a whole? � Quality of the parts � Representativeness

Internal validity �Mostly an issue in longitudinal research �Concerned with reducing or maximizing errors in research design in several ways �Must say which type of threat to internal validity

Internal validity � History (something happens to effect study’s process) � Maturation the way) (something naturally happens along � Testing effect (people get used to the testing, questions) � Measurement issues/instrumentation error (questions, inter-rater reliability) � Regression to the mean (over time, people will score close to the average)

Internal validity � Selection bias/differential selection of participants (can happen even with randomization) � Attrition/mortality another) (subjects leave the study one way or � Reactive effects of participants (behavioral response to being in treatment/control group) “Hawthorne effect” � Diffusion of treatment (control or comparison groups end up experiencing some of the treatment effect…) � Interaction effect (of any of the above)

How we want you to use today’s material �Critical consumer of research • What the heck the terms mean! • What the authors don’t say �Comment on potential threats in end-ofsemester proposal (limitations) • Quantiative – threats to validity • Qualitative – threats to trustworthiness, optimization of rigor