Student gains withwithout simulationbased inference centered Beth Chance

Student gains with/without simulation-based inference centered Beth Chance curricula Cal Poly – San Luis Obispo bchance@calpoly. edu JSM – Chicago - 2016 1

Simulation-based inference (SBI) n n Using repeated sampling (e. g. , bootstrapping) and randomization tests as the primary vehicle for learning about statistical inference Introduction to Statistical Investigations (ISI, Tintle, Chance, Cobb, Rossman, Roy, Swanson, Vander. Stoep) q q q Tactile simulations, applets From week 1 in the course Focus on entire statistical investigation process n n q Research question Data collection Analysis Conclusions One proportion One mean Multiple proportions … Regression Comparison to “theory-based methods” JSM – Chicago - 2016 2

Simulation-based inference (SBI) n Claims q q q Big picture of statistical investigation process Appreciation for meaningful research contexts Improved conceptual understanding n n q p-value: could this have happened by chance alone Confidence interval: Interval of plausible values, 2 SD Ability to explore, investigate n e. g. , non-traditional statistics (Median, Mean Abs Diff) JSM – Chicago - 2016 3

Instruments n Survey of Attitudes Toward Statistics (SATS, Schau, 2003) q q q n “Concept Inventory” (~35 questions) q q n 36 questions 6 scales: Affect, Interest, Difficulty, Effort, Value, Cognitive Competence (plus a few more) Demographic data Adapted from CAOS and GOALS (U of Minn) Plus a few more At beginning and end of course JSM – Chicago - 2016 4

Does SBI help? n Earlier work q Within our institutions n n n q q n Pre/post curriculum change (Tintle et al, 2011) Retention data (Tintle et al, 2012) Flipped vs. non-flipped implementations (with N. Pablo) Modifications to curriculum Modifications to concept inventory Current work q q Expand assessment to users and non-users of SBI curricula across institutions as part of NSF phase II grant Include section/instructor/institutional level variables JSM – Chicago - 2016 5

Implementation n Faculty and student recruitment q q q n Small incentive Generally out of class (Survey Monkey) Option to opt out (names to instructor) Instructor survey, e. g. : q q q Years of experience Type of institution, department How administered instrument Amount of class time spent lecturing vs. activities Knowledge of GAISE guidelines JSM – Chicago - 2016 6

Data cleaning process n n n Match pre/post data (duplicate names) Reverse coding of SATS questions Self-reported ACT/SAT data, GPA data, age q q n n “ 5. 8 haha j/k 2. 8” Converted ACT/SAT to z-scores Merging student and instructor data based on section number What about… q q q Time spent on instrument Student response rate Section response rate JSM – Chicago - 2016 7

Year 1 (Fall 13/Spring 14) n Initially 37 instructors, 1877 students q q q n 4 high school, 2 community college 2 “non-users” and 1 “flipper” 10 instructors using the curriculum for the first time Gain vs. “achievable gain” (Haake , 1998) q Pre: 80% Post: 90% 50% achievable gain JSM – Chicago - 2016 8

Year 1 (Fall 13/Spring 14) n n Achievable gain n JSM – Chicago - 2016 24 instructors 36 sections 1116 students Overall average =. 155 ICC 0. 10 9

Preliminary Observations (Chance, Tintle, Wong, to appear JSE) n n n No significant differences between experienced and first year ISI instructions Lower gains in sections with lower pre-attitude Some associations with … q q q n GPA (+) Cognitive competence (+) Difficulty (–) except for more experienced instructors Possible interactions with instructor sex and pretest and student-cluster variable q Gains generally lower with male instructors except for more prepared students JSM – Chicago - 2016 10

Year 2 (2014/2015) n Started with 43 institutions, 155 sections, 76 instructors, 5584 students q q q n n 8 high school, 15 two-year college, 1 Canada Class sizes: 9 to 285 (mean 36. 3, SD 35. 6) Years of experience: 0 to 39 (mean 9. 9, SD 9. 4) After data cleaning/opt out/outlier: 142 sections, 70 instructors, 38 institutions, 3146 students Textbook choice q q ISI: 10 institutions, 40 sections ISI for first time: 7 institutions Other SBI: 7 institutions, 52 sections Not SBI: 21 institutions, 43 sections JSM – Chicago - 2016 11

Year 2 (2014/2015) Overall mean ISI 1 st Other SBI Not SBI 2* . 170 . 216 . 214 . 185 . 124 . 061 JSM – Chicago - 2016 12

Concept subscales by Textbook ISI-1 st Other SBI Not SBI 2 JSM – Chicago - 2016 13

Student attitudes Cronbach’s Alpha Affect Cog Difficulty Effort Interest Value Pre. 82. 84. 71. 78. 88 Pre-attitudes (SD 1 - 7 SA) Post. 86. 87. 75. 71. 91 Affect Cog Difficulty Effort Interest Value JSM – Chicago - 2016 Mean 4. 24 4. 84 3. 68 6. 29 4. 80 5. 10 14

Attitude gains by Textbook JSM – Chicago - 2016 15

Multilevel model n n n lme (nlme) and lmer (lme 4) in R Post concept score as response variable Null model q q Overall intercept: 0. 576 Standard deviations (percentage of variance) n n n Section: Instructor: Institution: Carnegie classification: Residual: . 010 (. 4%). 032 (4%). 069 (20%). 055 (13%). 122 (63%) JSM – Chicago - 2016 16

Year 2 (2014/2015) n By institution JSM – Chicago - 2016 17

Multilevel Model Student variables n n n Pre-concept Pre-attitudes GPA, SAT/ACT z-score Sex Previous stat courses Area of study Section variables n n n Instructor variables n n n Years of experience Graduate student GAISE guidelines Attended ISI workshop Sex Average pre-concept Average prior attitude Average GPA Class size TA Time of day Institution variables n n n Carnegie Classification Math/Stat dept Scheduling system JSM – Chicago - 2016 18

Multilevel Models n n AIC: null = -4105. 3, model = -5101. 9 Student variables q q q n Pre-concepts score (t = 4. 4) SAT/ACT z-score (t = 9. 2, 5. 2); GPA (t = 9. 4) Student sex (t = -2. 9, lower for males) Pre-Attitudes q q Affect (t = -1. 7) Cognitive Competence (t = 4. 1) Effort (t = -3) Value (t = 4. 2) JSM – Chicago - 2016 19

Multilevel Models n Section/Instructor variables q n Section level variables less significant Interactions q q Pre-concept*student sex(male) (t = 3. 4) Instructor(male)*student sex(male) (t = -2. 1) JSM – Chicago - 2016 20

Textbook n Reference group: ISI q q ISI first (t = 1. 0) Other SBI (t =. 4) Not SBI (t = -3. 0) Not SBI 2 (t = -2. 6) JSM – Chicago - 2016 21

Most difficult questions n Two questions showed negative gains q q q n Could a small sample size explain an insignificant result Does an insignificant result provide strong evidence in favor of the null Across the texts, less negative for ISI Lowest post q q Recognize histogram vs. normal shape Confidence interval vs. prediction interval Necessary sample size for 3% margin of error Simulation design JSM – Chicago - 2016 22

Summary n n n No more (or less) gain in attitudes Significantly higher gain on concept inventory with SBI curricula after adjusting for preability, pre-performance, and pre-attitudes SBI curricula need more focus on q q q Proper conclusions with insignificant results Writing “pseudo-code” for simulations Relating study results to every day situations, role of population size JSM – Chicago - 2016 23

Next Steps n More model exploration q q n Year 3 data q q n Section time of day More (AP) high schools Learning progression q q n Section, institution level variables Interaction trees (Levine et al. ) Subgroup of instructors: data from midterm, transfer questions Retention study Data sharing JSM – Chicago - 2016 24

Acknowledgements n n Participating instructors NSF Grant DUE-1323210 Cal Poly undergraduates: Jimmy Wong, Jake Jaffe, Leticia Esperanza, Stephanie Mendoza, Brannden Moss Questions/Suggestions? q bchance@calpoly. edu JSM – Chicago - 2016 25