Reliability and Validity Quality of Data 1 Are

Reliability and Validity Quality of Data 1

Are we testing what we think we’re testing? 2

3

Quantitative Data n n Reliability Validity ¡ ¡ ¡ Face External Internal 4

Reliability n n Implies that the same data would have been collected each time over repeated tests/ observations. Would a particular technique (or survey question) yield the same result each time? ¡ n “Did you go to church last week? ” vs. “How many times have you been to church in your life? ” Reliability does not ensure accuracy. ¡ Taken from Babbie, E. 5

Reliability n n Problem if interpret questions differently Poorly worded questions Inconsistent coding: coding errors as with open-ended questions Lack of definition of key terms 6

Reliability n n Poorly worded: Does the library have adequate facilities and equipment for physically disabled students Better: Can patrons in wheelchairs retrieve books from the browsing collection? 7

Reliability (indicators) n n n Pretest Repeat question(s) Test/retest Split half and Parallel Interscore or scorer 8

Validity n n n A term to describe a measure that accurately reflects the concept it is intended to measure. Which is a more “valid” indicator of intelligence- an IQ score, or number of hours spent studying? Ultimate validity cannot be proven, but can be supported by face, internal, and external measures. ¡ Babbie, E. 9

10

Types of Validity n Face validity: The quality of an indicator/ question/ test that makes it a reasonable measure of a variable. ¡ ¡ Church attendance is an indication of religiosity. Number of grievances filed is an indicator of worker morale 11

Internal Validity n n n Approximate truth about inferences regarding causal relationships Typically applied to studies using inferential statistics (i. e. quantitative measures) than descriptive or observation studies. Especially useful for studies assessing affects of programs Only applicable to the study in question- not generalizable. Why not? Key question: Whether observed changes can be attributed to your program (the cause) and NOT other possible impacts/ causes. 12

Internal Validity Trochim, W. 13

Internal Validity n n History or specific events: events raises the issue that some variable other than the independent variable accounted for the change in the dependent variable. E. G. : the length of time between conducting the pretest and posttest may have a detrimental effect. Maturation: Maturation the change results from biological or psychological processes, which occurred over time, and not from the treatment itself. Maturation becomes more a concern the longer the period between the pretest and posttest Pretesting: Pretesting may affect the dependent variable. Pretesting may alert participants or educate them about the topic under investigation. Therefore if subjects are administered a posttest, their performance may reflect a marked improvement Measuring instruments or observational techniques: techniques These—not the treatment—may account for the change in the dependent variable. Further, the validity of study findings may have been influenced by the fact that the evaluators as observers, raters, graders, interviewers, and coders gained experience, became tired, obtained a more complete understanding of the project, or eased their expectations of test subjects 14

Internal Validity (continued) n n A nonrandom assignment of subjects to groups may signify that the groups were dissimilar from the beginning. Therefore any change might be attributed to the differential selection of subjects, rather than the actual treatment. Statistical regression refers to the tendency for extreme scores to regress or move toward the common mean of subsequent measures. The assignment of subjects to a particular test group on the basis of extreme views may affect study findings. 15

Internal Validity (continued) n n Mortality refers to the possibility that some subjects may have dropped out of the study after completion of the pretest but before the administration of the posttest. In such instances, every effort should be made to identify any common patterns or characteristics to ensure that any difference between a group’s pretest and posttest scores cannot be attributed to the loss of subjects. Interaction refers to the fact that more than one of the previous threats might be in play. This is especially likely in those cases where subjects were not randomly assigned to groups and the evaluation was based on existing, intact groups. 16

External Validity n n The approximate truth of generalizations drawn from a study. The degree to which conclusions drawn from your study sample would hold true to other persons in other places at other times ¡ Trochim, W. 17

External Validity Trochim, W. 18

External Validity n n n Example: institutions of higher education in Massachusetts: control, highest degree offered, and some characteristics of library (staff number, budget, and volume number) Return rate? Do respondents differ from nonrespondents as a group? 19

Validity n Content validity (for achievement test): How well does the test sample what the students learned? How well does a standardized test cover what was taught in the information literacy program? 20

Validity (continued) n n Criterion-related (predictive) (attitude test to predict performance in a library skills program): Who well does the test predict achievement for college freshmen? Criterion-related (diagnostic): How well does the test diagnose current problems with library use? 21

Validity (continued) n Construct validity: How well does the test measure comprehension of library use? Does a test on the use of an OPAC really measure effective and efficient use rather than one’s ability to read test items? 22

Qualitative Study Equivalent n n Credibility Dependability Confirmability Transferrability 23

Qualitative Reliability n Researcher is the “instrument”- how to test for reliability? ¡ ¡ ¡ ¡ Provide details of method, and abundance of evidence Provide evidence of qualifications as observer Make assumptions (and possible biases) clear State research questions clearly Use early stages of study to generate focus Observe for an adequate period of time, across a full range of activities Collect data from multiple sources Save data for reanalysis 24

Qualitative Validity n Depends upon reliablity. Like reliability, asserted by documenting steps ¡ ¡ ¡ ¡ Triangulation- data from different sources/ methods Full documentation of data- “chain of evidence” Logical connections between data and conclusions Conscious and deliberate inclusion of data that might not support thesis Preparedness to entertain alternatives Self-reflection, acknowledgement of own biases Review of preliminary reports by objective observers Awareness of limitations ¡ Gorman and Clayton 25

Qualitative Study: Increasing Reliability and Validty Inquiry affected by Results in Account for by: During After To lead to For findings that are: Factor patternings Noninterpretability Prolonged engagement Persistent observation Peer Debriefing Triangulation Member checks Establish structural corroboration (coherence) Credibility Plausible Situational Uniqueness Noncomparability Collect thick descriptive data Do theoretical/ positive sampling Develop thick description Transferability Context relevant -Gorman and Clayton 26

Qualitative Study: Increasing Reliability and Validity cont’d Inquiry affected by Results in Account for by: During After To lead to For findings that are: Instrumental changes Instability Use overlap methods Use stepwise replication Leave audit trail Do dependability audit (process) Dependability Stable Investigator Predilections Bias Do triangulation Practice reflexivity (audit trail) Do confirmability audit (product) Confirmability Investigator-free -Gorman and Clayton 27

Example For a sweeping study When conduct it? For how long? How deal with reliability and validity? Course evaluation 28

References n n Babbie, E. (2005). The basics of social research. Belmont, CA: Wadsworth Publishing. Gorman, G. E. & Clayton, P. (2005). Qualitative research for the information professional: A practical handbook. London: Facet Publishing. Trochim, W. M. K. (2006). Research methods knowledge base. Retrieved July 8, 2008 from http: //www. socialresearchmethods. net/kb/intval. php http: //www. socialresearchmethods. net/kb/external. p hp 29

Data Collection Intro to Methods 30

Methodology n n Quantitative, Qualitative, Mixed, and Triangulation How to select an appropriate method? ¡ ¡ ¡ What does quantitative data yield? What does qualitative data yield? What is the purpose of our study? 31

Data Collection n Right kind of data (available sources) Enough data (sample size) Goes back to problem statement 32

Data Collection n Primary data collection ¡ ¡ ¡ n Information you gather directly Reduce bias and increase accuracy Measure, don’t ask Secondary data collection ¡ ¡ ¡ Using data collected by someone else for a different purpose Report information or, Conduct secondary analysis 33

Primary Data n Quantitative or Qualitative ¡ ¡ ¡ ¡ Surveys/ Questionnaires Interviews Observations Case Studies Diaries/ Journals Tests Experiments 34

Secondary Data n Can be useful for ¡ ¡ ¡ n Supplementing your own research Offering background Providing larger scope/ scale Must consider ¡ ¡ Appropriateness Where it has come from Currency If you are going to compare it to other data- are they alike (same units etc. ) 35

Secondary Data n Main sources ¡ ¡ ¡ Governments General business resources International/ national organizations 36