Quality Indicators of Rigor in Quantitative Methods Dr

Quality Indicators of Rigor in Quantitative Methods Dr. Jackie Goodway The Ohio State University

Essential & Desirable Indicators of Research § Determine quality research by identifying essential & desirable indicators. § Gersten, Fuchs, Compton, Coyne, Greenwood, & Innocenti, 2005 § An assessment device designed to determine if an article is scientifically rigorous § Study Design & Implementation Assessment Device (DIAD, Valentine & Cooper, 2003)

Research Questions § Purpose statement is clear and precise outlining the intent of the study and constructs being studied. § Research Questions are precisely stated including variables and outcomes and participants § Directional – specify the perceived outcome of the study - hypotheses § Non-directional – suggest a relationship but don’t specify a direction. 3

Purpose & Research Questions The primary purpose of the study was to determine the effectiveness of an eight-week INDO-SKIP motor skill program on Indonesian preschool children’s Fundamental Motor Skill Competence. o RQ 1: What are the effects of an eight-week INDO-SKIP intervention implemented by trained early childhood teachers on preschool children’s Fundamental Motor Skill Competence? o RQ 2: To what extent do teachers implement INDOSKIP with fidelity and how does that influence the effect of the INDO-SKIP intervention on children’s Fundamental Motor Skill Competence? 4

Methods üContext üParticipants üEthics üDesign üDependent variable/s üInstrumentation 5 üIndependent variable/s (Intervention) üProcedures üData Analysis

Context § Detailed description of where the study took place. Help the reader understand the nature of the environment. § Country, geographic region, urban, suburban, rural § Location – e. g. school, childcare, youth sport team § Other relevant information such as socio-cultural factors e. g. socio-economic status, religion 6

Context: Large Urban Midwestern City in USA • 99% single parent families • Low income & disadvantaged community – Mean combined family income $16, 000 – 99% free & reduced lunch 7 • High rates of crime • 1 park - unsafe with gangs and drug deals in the park, lots of broken glass • All rental accommodation, with no gardens • Sidewalks have glass & debris • No recreation facilities within 3 miles

Methods Participants § Detailed description of participants (age, gender, ethnicity, income level, sports experience, home environment etc) 8 Sampling § Appropriate procedures to ensure participants are comparable across conditions: § Random assignment of individual to condition § Random assignment of group to condition § Matched pairs § Purposeful sampling § Link participant characteristics to literature definitions & prove participants meet them. § Description of characteristics relevant to § Appropriate procedures to intervention ensure interveners are comparable across § Description of teachers conditions. who teach intervention

Ethics with Participants § Informed consent § Review of the study § Adults can consent – over procedures by faculty on 18 years an Ethics Board § Parental permission § You cannot start a § Parents must consent to a child’s participation study until you have an approved protocol § Cannot change your methods without § Child agrees to participate approval in study § USA - Must keep all § Support letter from site consent documents for § Must get approval from the 5 years site you are working in § Child Assent 9

Participants & Setting • A large urban city in West Sumatera, Indonesia. • Muslim population. Minangkabau ethnic group. Participants • Teachers (N=12) from 4 early childhood centers • Purposively assigned into INDO-SKIP group (n=6 teachers) and control group (n=6 teachers). • Preschoolers (N=156) age 4 to 6 years (67 boys) • 4 centers ( 2 centers of INDO-SKIP-6 classrooms, 2 centers of control-6 classrooms) • 15 -20 children in a classroom • INDO-SKIP children (n=85), control (n=71) Ethics – University IRB approval 10 • Informed consent of teachers. Parental permission & child assent. Approval by Child Care centers.

Design of the Study § Specify the design of the study § Descriptive § Correlational designs § Quasi-experimental designs – condition is not randomly assigned § Experimental designs – e. g. randomized controlled trail, group randomized design Dependent Variables (DV) § Clearly operationalize (define) your construct what you mean by your DVs? 11

Dependent Variables (DV) § DV was operationalized § Multiple measures of the clearly e. g. “fundamental DV if possible motor skill competence” § Training of testers § Explain how the § Inter-observer instrument selected agreement (IOA) actually measures the construct/DV in the study § Inter – between 2 or (consider all options of instruments) § Description of instrument 12 more coders § Intra – consistency across time with 1 coder § Blind coders to condition § Reliability & stability of instrument - Internal consistency (Cronbach’s § Multiple time frames of alpha) testing pretest-posttest § Range of scores § Add a retention test § Validity

Independent Variable Intervention § Theoretical origins of § Describe how child intervention moves through an § How theory drives intervention session decisions in designing § Intervention fidelity – intervention did the intervention get § Detailed description of implemented as intervention - replication intended § Content § Document fidelity § DOSE - amount § Detailed description of § Teacher language, comparison conditions behaviour/pedagogy § Student behaviour § Attendance/attrition rates 13

Design & Instruments Research Design § Quasi experimental (no random selection and assignment), pretest-posttest with a control group design. 14 Dependent Variable – Fundamental motor skill competence- child’s ability to demonstrate FMS with critical elements of form.

Test of Gross Motor Development 2 Locomotor Subscale Object Control Subscale § 6 skills: Run, Gallop, Hop, Leap , Jump, Slide § 6 skills: Throw, Bounce, Strike, Catch , Kick, Roll § Raw score - 0 -48 points § Raw score 0 -48 points § Standard score based on age & Percentile rank for age § Standard score & percentile rank based on age & gender • Gross Motor Quotient (overall motor skills) • All skills videotaped & coded from videotape • Coders trained prior to coding & blind to group • Inter-rater reliability - reliability between 2 independent raters – 97% • Intra-rater reliability – reliability across time – 95% across 2 time points 15 Ulrich, 2000

INDO-SKIP Content § Theory - Manipulation of environmental constraints (e. g. equipment) & developmental tasks to align with individual child constraints. § Content - Fundamental motor skills – 6 object control skills § Throw, Catch, Kick, Roll, Strike, Bounce, § Dose: 8 week, 2 sessions a week, 30 minutes per session. Each skill had 80 -110 mins of instruction over intervention. § Direct instructional approach by Early Childhood Teachers. § Typical Lesson § 3 -4 minute vigorous warm up game or music activity § 2 object control skills in a lesson (7 -8 minutes per skill) § Closure – stretching & fun balancing activities § Skills – cycles of skills implemented over the intervention.

Study Procedures § Outline the procedures of the study in detail for replication § In complex studies it is helpful to break the study into phases: § Phase 1 – Teacher Training on intervention § Phase 2 – Pretest >>> Intervention>>Posttest § Phase 3 – Retention test § Have someone who knows nothing about your study read the procedures and see if they can explain your study to you 17

Procedures § Phase 1: § Teacher Initial Training on INDO-SKIP- 9 hours • 3 formative assessments on teachers’ knowledge on MC, teachers’ accuracy in demonstration, and fidelity on teaching mini lesson. § Pretest children on TGMD-2 § Phase 2: § INDO-SKIP Implementation: 16 sessions, 8 weeks. o Fidelity of INDO-SKIP Assessment (FIA) on 6 lessons/teacher o On-going coaching: week 1 to 5. § Control Condition – observation of 4 playground sessions/teacher § Phase 3: § Posttest on children fundamental motor skills § Data analyses

Data Analysis & Results § List RQs - Data analysis ties to research questions in sequential order § Explains sample size and statistical power at the appropriate unit of analysis (individual, class) § Unit of analysis may change with RQ § Underlying assumptions for statistical tests documented e. g. normality, homogeneity of variance, independence 19 § Account for variability in data via sampling or design § In intervention research include effect sizes – e. g. Cohens D § Don’t duplicate data e. g. data in table & figure & text § Results presented in sequential order tied to RQs §

Data Analysis & Results o RQ 1: What are the effects of an eight-week INDO-SKIP intervention implemented by trained early childhood teachers on preschool children’s Fundamental Motor Skill Competence? o Hierarchical Linear Modeling (HLM) with fundamental motor skill competence as the dependent variable. o Level 1 Predictors – Pretest score, Gender o Level 2 - Class o RQ 2: To what extent do teachers implement INDO-SKIP with fidelity and how does that influence the effect of the INDOSKIP intervention on children’s Fundamental Motor Skill Competence? o HLM with fundamental motor skill competence as the dependent variable, INDO-SKIP subgroup. o Level 1 Predictors – Pretest score, Gender o Level 2 – Teacher fidelity 20

RIGOR, ALIGNMENT Detail necessary for replication