1 Research Design in Quantitative Research 2 Research

1 Research Design in Quantitative Research

2 • Research Design Experimental or Quasi-Experimental • “Is it reasonably plausible that the outcome is created by the treatment? ”

3 • Types of Research • • Historical Descriptive • Survey • Qualitative • • Correlational Causal-Comparative Quasi-Experimental

4 • Historical Research • “What Was” • Study the past to better understand predict the future • Examples: • Teacher Termination • PSSA Test • Legislature Support for Education

5 • Descriptive Research • “What Is” • Describe the existing situation • Examples: • Needs Assessment • Special Education Attitudes • Graduate Follow-Up

6 • Survey Research • Things You Must Know • Survey Development • Characteristics of the Sample • Distribution and Follow-Up Procedures • Return Rate Issues/Actions

7 • Survey Research Survey Development - Form • • Demographic Questions Item Content Response Format Written Comments • • • Thematic Analysis Valence (+, - ) Self-Coding • Layout and Printing

8 • Survey Research Survey Development - Considerations • Personal Request Letter • Pre-contact • Authority Figure • Incentives • Length • Time of Year

9 • Survey Research Characteristics of the Sample • Education Level • Group Identification with Issue • Knowledge of Subject • Interest in Subject • Perceived Importance

10 • Survey Research Distribution and Follow-Up • Time of Year • Cost • Materials • Postage • Distribution Method • Return Method • Follow-Up Procedures

11 • Survey Research Return Rate Issues/Actions • Response-Rate Bias • Sample vs. Population Demographics • Leslie • Murphy, Gable, Owen • Analysis of Returns • Trend Analysis • Sample Non-Respondents

12 • Correlational Research “What relationship exists? ” 1. Measure the degree of relationship between two or more variables 2. Examples: 1. 2. 3. 4. Construct Validity (Correlations, Factor Analysis) Economic and Organizational Factors Feedback Seeking and Managerial Effectiveness Criterion - Related Validity (Attitudinal Predictors of Dropping Out of School)

13 • Causal-Comparative (Ex-Post-Facto) “Attempt to deduce how? ” • Independent variable(s) have already occurred • Start with observation of the dependent variable • Study independent variable(s) in retrospect for possible relation to the dependent variable

14 • Causal Comparative “Self-Perception of Culturally Disadvantaged Children” Soares and Soares Adv. School Disadv. F Gender M 4 5 6 7 8 Grade

15 • Quasi-Experiment “Attempt to deduce how and why” • Field Study - Can’t do a true experiment • No random assignment • Can not control all relevant variables • Example • Material Incentives Project

16 • Experiment “Determine how and why” • Examine cause and effect • Random assignment • “Manipulation” of independent variable • Observe effect on dependent variable • Examples • Alternate school project • Project Concern • Police attitudes

17 • Experiment Alternate School Project • Selection • • • Low Attitude (Grade 8) Teacher Recommendations Student/Parent Volunteer High Absent/Low Grades Random Selection • Participants/Non-Participants

18 • Experiment Alternate School Project • Check Random Assignment • • • Absent Suspensions Grades (Math, English) • Program • Academic • Occupational

19 • Experiment Alternate School Project - Dependent Variables • Behavioral • • • Absent Suspensions Drop Out Rate Cognitive • Achievement Affective • Mc. Cook Attitude Scale Occupational • Knowledge • Aspiration

20 • Sampling • Population • The target group of interest • All individuals who possess a certain characteristic • High school teachers • Principals • 4 th grade students • Sample • Any subgroup on which information is obtained

21 • Sampling • Does the sample represent the population? • Target Population • Accessible Population • Sample

22 • Sampling • Why Sample? • Save Time • Save Money • Impossible to work with the population

23 • Sampling • Types of Samples • Random • Simple • Stratified • Cluster • Non-Random • Systematic • Convenience • Purposive

24 • Sampling • Sample Size • Use of tables • Bias • Precision

25 • Validity • Are the inferences made from test scores appropriate, meaningful and useful? • Does the accumulated evidence (judgmental and empirical) support the inferences? • Inferences are validated, not the test

26 • Validity • Types of Validity • • • Content Construct Criterion-Related

27 • Validity • Content Validity • To what extent do the items on the test adequately sample from the intended universe of content? • Judgmental Evidence • Literature Review • Expert Review

28 • Validity • Construct Validity • To what extent do certain explanatory concepts explain: • • covariation in the responses to the items, or relationships with other know indicators? • Empirical evidence • • • Correlations Factor analysis Structural equation modeling Item response theory Known groups

29 • Validity • Criterion-Related Validity • What is the relationship between scores on the instrument and some external criterion that provides a more direct measure of the targeted characteristic? • Types • Concurrent • Predictive

30 • Reliability • “Does the instrument provide us with accurate assessments? ”

31 • Reliability • Theory • XTotal = XTrue + XError • VTotal = VTrue + VError VTrue • Reliability = VTotal

32 • Reliability • Theory • r 12 True = Reliability • “Proportion of variance in scores that can be considered ‘true’ variance”

33 • Reliability • Sources of Error • Inadequate sampling of items • Cronbach’s alpha internal consistency reliability • Different occasions • Stability reliability

34 • Reliability • Types of Reliability • Internal consistency • Stability

35 • Reliability • Generalizability Theory • “Simultaneously examines homogeneity of the items and stability of responses over time”

36 • Qualitative Research • Procedures which produce descriptive data -- peoples’ written or spoken word, or observations of behavior

37 • Qualitative Research • • Ethnography - Naturalistic Inquiry Study behavior in a natural setting Involvement with subjects See the world through their eyes

38 • Research Methods • Selecting a Research Strategy • Qualitative • Quantitative

39 • Research Methods Quantitative • Philosophical Base Logical Positivism • Purpose Phenomenology Verification Discovery • Framework/Design Preordinate/Fixed • Conditions Controlled • Treatment Qualitative Emergent/Flexible Invited Interference Stable Variable

40 • Research Target • Process Product • Implementation Outcome • “Implementation as an independent variable”

41 • Qualitative Research Access and Entry • Front Stage • Packaged Reality • Back Stage • Reality • Gatekeepers “Schools are not researchers’ sandbox”

42 • Qualitative Research Data Collection • Interview • What is said • Document Analysis • What is written • Participant Observation • What is done

43 • Qualitative Research Triangulation • Combination of Methodologies • Multiple Reference Points • Increase Accuracy • Increase Confidence • Convergence of Evidence • Among Methods • Within Methods

44 • Qualitative Research Data Analysis Qualitative • • • Ongoing Inductive Quantitative At conclusion Deductive Generating Theory Testing Theory

45 • Qualitative Research • Reporting Results • • Natural History Thematic Analysis Social Networks Life History Allocation and Distribution of Resources Organizational Analysis Critical Incidence

46 • Research Design • Development of Education Research Methods • Genetics • Correlation (Galton) • Agriculture • Controlled experiments (Fisher - ANOVA)

47 • Research Design • Complexity of Educational Processes • Experimental research: failure of the promise • Trend toward qualitative methods

48 • Research Design • Experimental and Quasi-Experimental Designs for Research (1963) • Donald Campbell • Julian Stanley • Shift focus from statistics to problems of acquiring data in education • Develop benchmarks for research designs • Compare research designs with respect to benchmarks

49 • Research Design Focus • 1960’s - 1970’s: Experimental • The External Validity of Experiments Bracht & Glass, 1968 • 1980 +: Quasi-Experimental • Quasi-Experimentation: Design and Analysis Issues for Field Settings Cook & Campbell, 1979

50 • Research Design • Internal Validity • To what extent have extraneous variables been controlled so that the observed effect (dependent) can be attributed to the treatment variable (independent)? • Is it reasonably plausible that the outcome (dependent) is created by the treatment (independent)?

51 • Research Design • Threats to External Validity • Results generalizable to other: • • People? Settings? Independent variables? Dependent variables?

52 • Research Design • Internal Validity Threats • Testing • Instrumentation • Statistical regression • Differential selection • Experimental treatment diffusion

53 • Research Design • Internal Validity Threats • Compensatory rivalry by the control group • Compensatory equalization of treatments • Resentful demoralization of the control group

54 • Research Design One Group Pre-Post X

55 • Research Design Pre-Post Comparison Group X

Research Design 56 Pre-Post Comparison Group with Random Assignment R R X

Research Design 57 Post-Only Comparison Group with Random Assignment R R X

58 • External Validity of Research • “Generalizability” • Population Validity • Generalize from sample to population? • Personalogical variables interact with treatment?

59 • External Validity of Research • “Generalizability” • Ecological Validity • • Explicit description of treatment Multiple treatment interference Hawthorne effect Novelty and disruption effects Experimenter effect Pretest sensitization Measurement of dependent variable

60 • Statistical Analysis Change Scores Experimental Post Pre = Change Xchange Comparison Post Pre = Change Xchange

61 • Statistical Analysis Repeated Measures ANOVA RMANOVA Pre Experimental Comparison Post

62 • Statistical Analysis RMANOVA Sources of Variance Between Group Error (b) Fgroup Within Time x Group Error (w) Ftime x group Total

• Statistical Analysis 63 Analysis of Covariance ANCOVA Research Question Will there be a difference between groups with respect to posttest scores after controlling for initial differences in pretest scores?

64 Exp Post Dependent Control Pre Covariate

65 Exp Post Dependent Control Pre Covariate PÕST = b(PRE) + a

66 66 Marginal Distribution YE Exp Post Dependent Control YC XCont Pre Covariate XExp

67 67 Marginal Distribution Exp YE Post Dependent YC Control XCont X Pre Covariate XExp

68 • Effect Size “The difference between two means expressed in standard deviation units” Mean 1 - Mean 2 ES = Standard Deviation ES. 20. 50. 80 Qualitative Difference Small Medium Large

• Statistical Significance: Sample Size and Effect Size 69 “I have a meaningful say in designing my job/work. ”* Mean Standard Deviation 10, 747 3. 00 1. 20 Females 8, 088 3. 26 1. 19 Males 3. 03 1. 17 Group Males N 111 Sign. Level Effect Size . 000 . 22 . 157 . 21 Females 81 3. 28 1. 22 * Responded to on a 5 -point Likert agreement scale.