1 Reliability and Validity 2 Not Reliable Not

1 Reliability and Validity

2 Not Reliable, Not Valid

3 Reliable, Not Valid

4 Reliable and Valid

5 Relationship of Reliability and Validity n A measure can be reliable without being valid. n A measure must be reliable to be valid.

6 • Validity • Types of Validity • • • Content Construct Criterion-Related

7 • Validity • Content Validity • To what extent do the items on the test adequately sample from the intended universe of content? • Judgmental Evidence • Literature Review • Expert Review

8 • Validity • Construct Validity • To what extent do certain explanatory concepts explain: • • covariation in the responses to the items, or relationships with other know indicators? • Empirical evidence • • • Correlations Factor analysis Structural equation modeling Item response theory Known groups

9 • Validity • Criterion-Related Validity • What is the relationship between scores on the instrument and some external criterion that provides a more direct measure of the targeted characteristic? • Types • Concurrent • Predictive

10 • Reliability • “Does the instrument provide us with accurate assessments? ”

11 • Reliability • Theory • r 12 True = Reliability • “Proportion of variance in scores that can be considered ‘true’ variance”

12 • Reliability • Sources of Error • Inadequate sampling of items • Cronbach’s alpha internal consistency reliability • Different occasions • Stability reliability

13 • Reliability • Types of Reliability • Internal consistency • Stability

14 • Reliability • Generalizability Theory • “Simultaneously examines homogeneity of the items and stability of responses over time”

15 • Qualitative Research • Procedures which produce descriptive data -- peoples’ written or spoken word, or observations of behavior

16 • Qualitative Research • • Ethnography - Naturalistic Inquiry Study behavior in a natural setting Involvement with subjects See the world through their eyes

17 • Research Methods • Selecting a Research Strategy • Qualitative • Quantitative

18 • Research Methods Quantitative • Philosophical Base Logical Positivism • Purpose Phenomenology Verification Discovery • Framework/Design Preordinate/Fixed • Conditions Controlled • Treatment Qualitative Emergent/Flexible Invited Interference Stable Variable

19 • Research Target • Process Product • Implementation Outcome • “Implementation as an independent variable”

20 • Qualitative Research Access and Entry • Front Stage • Packaged Reality • Back Stage • Reality • Gatekeepers “Schools are not researchers’ sandbox”

21 • Qualitative Research Data Collection • Interview • What is said • Document Analysis • What is written • Participant Observation • What is done

22 • Qualitative Research Triangulation • Combination of Methodologies • Multiple Reference Points • Increase Accuracy • Increase Confidence • Convergence of Evidence • Among Methods • Within Methods

23 • Qualitative Research Data Analysis Qualitative • • • Ongoing Inductive Quantitative At conclusion Deductive Generating Theory Testing Theory

24 • Qualitative Research • Reporting Results • • Natural History Thematic Analysis Social Networks Life History Allocation and Distribution of Resources Organizational Analysis Critical Incidence

25 • Research Design • Development of Education Research Methods • Genetics • Correlation (Galton) • Agriculture • Controlled experiments (Fisher - ANOVA)

26 • Research Design • Complexity of Educational Processes • Experimental research: failure of the promise • Trend toward qualitative methods

27 • Research Design • Experimental and Quasi-Experimental Designs for Research (1963) • Donald Campbell • Julian Stanley • Shift focus from statistics to problems of acquiring data in education • Develop benchmarks for research designs • Compare research designs with respect to benchmarks

28 • Research Design Focus • 1960’s - 1970’s: Experimental • The External Validity of Experiments Bracht & Glass, 1968 • 1980 +: Quasi-Experimental • Quasi-Experimentation: Design and Analysis Issues for Field Settings Cook & Campbell, 1979

29 • Research Design • Internal Validity • To what extent have extraneous variables been controlled so that the observed effect (dependent) can be attributed to the treatment variable (independent)? • Is it reasonably plausible that the outcome (dependent) is created by the treatment (independent)?

30 • Research Design • Threats to External Validity • Results generalizable to other: • • People? Settings? Independent variables? Dependent variables?

31 • Research Design • Internal Validity Threats • Testing • Instrumentation • Statistical regression • Differential selection • Experimental treatment diffusion

32 • Research Design • Internal Validity Threats • Compensatory rivalry by the control group • Compensatory equalization of treatments • Resentful demoralization of the control group

33 • Research Design One Group Pre-Post X

34 • Research Design Pre-Post Comparison Group X

Research Design 35 Pre-Post Comparison Group with Random Assignment R R X

Research Design 36 Post-Only Comparison Group with Random Assignment R R X

37 • External Validity of Research • “Generalizability” • Population Validity • Generalize from sample to population? • Personalogical variables interact with treatment?

38 • External Validity of Research • “Generalizability” • Ecological Validity • • Explicit description of treatment Multiple treatment interference Hawthorne effect Novelty and disruption effects Experimenter effect Pretest sensitization Measurement of dependent variable

39 • Statistical Analysis Change Scores Experimental Post Pre = Change Xchange Comparison Post Pre = Change Xchange

40 • Statistical Analysis Repeated Measures ANOVA RMANOVA Pre Experimental Comparison Post

41 • Statistical Analysis RMANOVA Sources of Variance Between Group Error (b) Fgroup Within Time x Group Error (w) Ftime x group Total

• Statistical Analysis 42 Analysis of Covariance ANCOVA Research Question Will there be a difference between groups with respect to posttest scores after controlling for initial differences in pretest scores?

43 Exp Post Dependent Control Pre Covariate

44 Exp Post Dependent Control Pre Covariate PÕST = b(PRE) + a

45 45 Marginal Distribution YE Exp Post Dependent Control YC XCont Pre Covariate XExp

46 46 Marginal Distribution Exp YE Post Dependent YC Control XCont X Pre Covariate XExp

47 • Effect Size “The difference between two means expressed in standard deviation units” Mean 1 - Mean 2 ES = Standard Deviation ES. 20. 50. 80 Qualitative Difference Small Medium Large

• Statistical Significance: Sample Size and Effect Size 48 “I have a meaningful say in designing my job/work. ”* Mean Standard Deviation 10, 747 3. 00 1. 20 Females 8, 088 3. 26 1. 19 Males 3. 03 1. 17 Group Males N 111 Sign. Level Effect Size . 000 . 22 . 157 . 21 Females 81 3. 28 1. 22 * Responded to on a 5 -point Likert agreement scale.