METAANALYSIS Dr Ken Kavale METAANALYSIS Quantitative Research Synthesis

META-ANALYSIS Dr. Ken Kavale

META-ANALYSIS Quantitative Research Synthesis “What the research says” Knowledge Accumulation Usable Knowledge

METHODS OF RESEARCH SYNTHESIS Narrative Review Serial Presentation No Accumulation Findings remain independent and isolated If Accumulated SUBJECTIVE

QUANTITATIVE REVIEWS Voting Method Box-Score Analysis Studies classified by statistical significance or nonsignificance What is statistical significance? Publication Bias (“File Drawer Problem”)

META-ANALYSIS • • • Advantages Uses quantitative (statistical) methods for organizing and extracting information Eliminates study selection bias – no prejudgments about research quality are made Makes use of all information – findings are transformed to commensurate expressions of effect magnitude

META-ANALYSIS (con’t) l l Detects interactions – study characteristics that may mediate findings are defined and measured, and their covariation is investigated Seeks general conclusions – practical simplicity OBJECTIVITY VERIFIABILITY REPRODUCABILITY

METHODS OF META-ANALYSIS Scientific Method Formulating Problems Broadly defined “Is Method x effective? ” Sampling Literature search - Comprehensive - Published and unpublished

METHODS OF META-ANALYSIS (con’t) Classifying and Coding Studies Study features Data Analysis Effect Size – z score (standard deviation units) __ __ ES = XE – XC SDC

METHODS OF META-ANALYSIS (con’t) Effect Size Interpretation • Overlapping distributions and comparable percentiles • Comparative context • Statistical power – small (. 20), medium (. 50), large (. 80>) • Binomial effect size display (BESD) – practical significance • Common language effect size – research significance

SUMMARY OF META-ANALYSIS BY EFFECT SIZE MAGNITUDE Intervention ES Number of Studies 1. 62 24 Reading-comprehension strategies (Talbot, Lloyd, & Tankersley, 1994) (Mastropieri et al. , 1996) (Swanson, 1999) 1. 13 0. 98 0. 72 48 68 58 Behavior modification (Skiba & Casey, 1985) 0. 93 41 Direct instruction (White, 1988) 0. 84 25 Large Mnemonic strategies (Mastropieri & Scruggs, 1989)

SUMMARY OF META-ANALYSIS BY EFFECT SIZE MAGNITUDE (con’t) Intervention ES Number of Studies Cognitive behavior modification (Robinson, Smith, Miller & Brownell, 1999) 0. 74 23 Psychotherapy (Weisz & Weiss, 1993) 0. 71 110 Formative evaluation (Fuchs & Fuchs, 1986) 0. 70 21 Early intervention (Casto & Mastropieri, 1986) 0. 68 74 Stimulant medication (Crenshaw, Kavale, Forness, & Reeve, 1999) (Kavale, 1982) 0. 67 0. 58 115 135 Medium

SUMMARY OF META-ANALYSIS BY EFFECT SIZE MAGNITUDE (con’t) Intervention ES Number of Studies Medium Computer-assisted instruction (Schmidt, Weinstein, Niemie, & Walberg, 1985 -86) 0. 66 18 Peer tutoring (Cook, Scruggs, Mastropieri, & Casto, 0. 58 19 1985 -86) Word recognition strategies (Swanson, 1999) 0. 57 54

SUMMARY OF META-ANALYSIS BY EFFECT SIZE MAGNITUDE (con’t) Intervention ES Number of Studies Psycholinguistic training (Kavale, 1981) 0. 39 34 Reducing class size (Glass & Smith, 1979) 0. 31 77 Psychotropic medication (Kavale & Nye, 1984) 0. 30 70 0. 21 0. 20 53 35 Small Social-skills training (Forness & Kavale, 1996) (Quinn, Kavale, Mathur, Rutherford, & Forness, 1999)

SUMMARY OF META-ANALYSIS BY EFFECT SIZE MAGNITUDE (con’t) Intervention Small Modality instruction (Kavale & Forness, 1987) ES Number of Studies 0. 14 39 Diet restrictions (Kavale & Forness, 1983) 0. 12 23 Perceptual training (Kavale & Mattson, 1983) 0. 08 180 Special class training (Carlberg & Kavale, 1980) -0. 12 50

PERCEPTUAL-MOTOR TRAINING EFFECT SIZE. 08 Table 1 Average ES for Perceptual-Motor Outcome Classes Outcome Class Mean ES Power Analysis Perceptual/Sensory Motor . 166 Small Academic Achievement . 013 Negligible Cognitive . 028 Negligible

PERCEPTUAL-MOTOR TRAINING (con’t) Table 2 Average ES for Perceptual-Motor General Outcome Categories Mean ES Power Analysis Perceptual/Sensory Motor Gross motor Fine motor Visual perception Auditory perception . 214. 178. 149. 122 Small

PERCEPTUAL-MOTOR TRAINING (con’t) Table 2 Average ES for Perceptual-Motor General Outcome Categories Mean ES Power Analysis Academic Achievement Readiness Reading Arithmetic Language Spelling Handwriting . 076 -. 039. 095. 031. 021. 053 Negligible Negative Negligible Cognitive/Aptitude Verbal IQ Performance IQ -. 007. 068 Negative Negligible

PERCEPTUAL-MOTOR TRAINING (con’t) Table 3 Average ES for Perceptual-Motor Specific Outcome Categories General Outcome Categories Mean ES Power Analysis Gross Motor Skills Body awareness/image Balance/posture Locomotor skills . 256. 263 -. 017 Small Negative Visual Perceptual Skills Visual discrimination Figure-ground discrimination Visual-motor ability Visual integration Visual spatial perception Visual memory . 146. 173. 222. 086. 144. 062 Small Negligible

PERCEPTUAL-MOTOR TRAINING (con’t) Table 3 Average ES for Perceptual-Motor Specific Outcome Categories General Outcome Categories Mean ES Power Analysis Reading Achievement Word recognition Comprehension Oral reading Vocabulary Speed/rate -. 016 -. 055 -. 037 -. 012 -. 038 Negative Negative

PERCEPTUAL-MOTOR TRAINING (con’t) Table 4 Average ES for Subject Groups Subject Mean ES Power Analysis Normal Educable Mentally Retarded (IQ = 50 -75) Trainable Mentally Retarded (IQ = 25 -50) Slow Learner (IQ = 75 -90) Culturally Disadvantaged Learning Disabled Reading Disabled Motor Disabled . 054. 132 Negligible Small . 147 Small . 098. 045. 018 -. 007. 121 Negligible Negative Small

PERCEPTUAL-MOTOR TRAINING (con’t) Table 5 Average ES for Grade Level Mean ES Power Analysis Preschool Kindergarten Primary (Grades 1 -3) Middle (Grades 4 -6) Junior high school High school . 053. 099. 079. 066. 085. 088 Negligible Negligible

MODALITY-MATCHED INSTRUCTION (Learning Styles) EFFECT SIZE. 14 Table 1 Effects of Modality Assessments Modality Mean ES % of Subjects (uncorrected) (corrected) Differentiated from Comparison Group Auditory . 925 . 552 71% Visual . 899 . 506 70% Kinesthetic . 970 . 430 67%

MODALITY-MATCHED INSTRUCTION (Learning Styles) (con’t) Table 2 Effects of Modality Matched Instruction Method Number Mean Standard % of Subjects of ES ES Error of Differentiated ES from Comparison Group Auditory 80 . 184 . 028 57 Visual 81 . 086 . 037 54 Kinesthetic 44 . 175 . 045 57

MODALITY-MATCHED INSTRUCTION (Learning Styles) (con’t) Table 2 Effects of Modality Matched Instruction Modality Total ES Auditory % ES % Visual ES Kinesthetic % ES % Word Recognition . 150 56 . 203 58 . 081 53 . 197 58 Comprehension. 046 52 . 062 52 . 034 51 . 041 52 Vocabulary . 174 57 . 194 58 . 141 56 . 185 58 Spelling . 184 57 . 249 60 . 088 54 . 216 59

STIMULANT MEDICATION EFFECT SIZE. 58. 67

STIMULANT MEDICATION (con’t) Table 1 Average Effect Sizes for Stimulant Medication Research Outcome Classes Mean ES Power Analysis Behavioral Global improvement ratings . 886 Large Rating scales and checklist . 837 Large Activity level . 846 Large Attention and concentration . 782 Large Behavior (social and classroom) . 634 Medium-Large Anxiety . 118 Small

STIMULANT MEDICATION (con’t) Table 1 Average Effect Sizes for Stimulant Medication Research Outcome Classes Mean ES Power Analysis Cognitive Intelligence . 391 Medium Achievement . 383 Medium Drawing and copying . 467 Medium Perceptual, memory and motor . 412 Medium Learning characteristics . 367 Small-Medium

FEINGOLD DIET EFFECT SIZE. 12 Table 1 Average Effect Sizes for Feingold Diet Research Outcome Categories Category Mean ES Power Analysis Conners Scale-Parents . 156 Small Conners Scale-Teachers . 268 Small Global improvement . 128 Small Hyperkinesis rating . 293 Small Attention . 015 Negligible Disruptive behavior . 052 Negligible Impulsivity . 153 Small Learning ability -. 055 Negative

INTERVENTIONS FOR ADHD Stimulant Medication Effect Size. 63 Feingold Diet Effect Size. 12 Treatment Duration Stimulant Medication 10 weeks Feingold Diet 39 weeks Comparison Stimulant medication better than 5 X more effective than the Feingold diet in ¼ the time

SPECIAL CLASS PLACEMENT EFFECT SIZE -. 12 Table 1 Average ES by Special Education Classification Diagnosis Mean ES Mental retardation (IQ 50 -75) -. 14 Slow learning (IQ 75 -90) -. 34 Learning disability or Emotional/behavioral disorder . 29

SOCIAL SKILLS TRAINING EFFECT SIZE. 21. 20

SOCIAL SKILLS TRAINING (con’t) Table 1 Average ES for Teacher Assessments of Social Skills Training for Students with Learning Disabilities Training Program Mean ES % Equivalent Adjustment . 29 62 Dependency . 25 60 Conduct disorder . 22 59 Interaction . 11 54 Hyperactivity . 07 53 Academic competence . 05 52 Teacher assessment composite mean . 16 56

SOCIAL SKILLS TRAINING (con’t) Table 2 Average ES for General Education Peer Assessments of Social Skills Training for Students with Learning Disabilities Components Mean ES % Equivalent Communication . 25 60 Acceptance . 23 59 Cooperation . 22 59 Friendship . 22 59 Rejection . 20 58 Interaction . 20 58 Social status . 13 55 Peer assessment composite mean . 21 58

SOCIAL SKILLS TRAINING (con’t) Table 3 Average ES for Self-Assessments of Social Skills Training for Students with Learning Disabilities Components Mean ES % Equivalent Social status . 38 65 Self-concept . 28 61 Social problem solving . 28 61 Social competence . 27 61 Interaction . 19 58 Locus of control . 08 53 Self-assessment composite mean . 24 59

SOCIAL SKILLS TRAINING (con’t) Table 4 Average ES of Social Skills Training by Teacher, Peer, Self, Experimenter, and Parent Assessments of Students with Emotional/Behavioral Disorders Rater Mean ES % Equivalent Teacher . 22 59 Peer . 22 59 Self . 22 59 Experimenter . 19 58 Parent . 15 56 Rater composite mean . 20 58

SOCIAL SKILLS TRAINING (con’t) Table 5 Average ES of Training on Social Skill Dimensions for Students with Emotional/Behavioral Disorders Dimension Mean ES % Equivalent Social behavior . 27 61 Social relations . 27 61 Social problem solving . 26 60 Social competence . 22 59

SOCIAL SKILLS TRAINING (con’t) Table 6 Average ES of Training on Social Skill Areas for Students with Emotional/Behavioral Disorders Area Mean ES % Equivalent Family relations . 20 58 Communication . 18 57 School behavior . 18 57 Conduct disorder . 13 55

SOCIAL SKILLS TRAINING (con’t) Table 7 Average ES of Training on Social Skill Variables for Students with Emotional/Behavioral Disorders Variable Mean ES % Equivalent Anxiety . 42 66 Adjustment . 27 61 Cooperation . 26 60 Interaction . 24 59 Self-concept/esteem . 16 56 Aggression . 13 55