Using Simulations to Teach Statistical Inference Beth Chance

Using Simulations to Teach Statistical Inference Beth Chance, Allan Rossman (Cal Poly) ICTCM 2011 1

Joint Work with n Soma Roy, Karen Mc. Gaughey (Cal Poly), q n n Alex Herrington (Cal Poly undergrad) John Holcomb (Cleveland State), George Cobb (Mt. Holyoke), Nathan Tintle, Jill Vander. Stoep, Todd Swanson (Hope College) This project has been supported by the National Science Foundation, DUE/CCLI #0633349 ICTCM 2011 2

Outline n n Motivation/Goals Examples q q q n n n Binomial process, randomized experiment- binary, randomized experiment - quantitative response Series of lab assignments Discussion points Student feedback, Evaluation results Design principles & implementation Observations, Open questions ICTCM 2011 3

Motivation n Cobb (2007) – 12 reasons to teach permutation tests… q q q Model is “simple and easily grasped” Matches production process, links data production and inference Role for tactile and computer simulations Easily extendible to other designs (e. g. , blocking) Fisherian logic --”The Introductory Statistics Course: A Ptolemaic Curriculum” (TISE) ICTCM 2011 4

Goals n Develop an introductory curriculum that focuses on randomization-based approach to inference q q n vs. using simulation to teach traditional inference From beginning of course, permeate all topics Improve understanding of inference and statistical process in general q More modern (computer intensive) and flexible approach to inferential analysis ICTCM 2011 5

Brief overview of labs n n Case-study focus Pre-lab q n n 50 -minute (computer) lab period Online instructions q q n n Background, Review questions submitted in advance Directed questions following statistical process Embedded applets or statistical software Application/Extension Lab report with partner ICTCM 2011 6

Example 1: Friend or Foe (Helper/Hinderer) n n n n Videos Research question Pre-lab Descriptive analysis Introduction of null hypothesis, p-value terminology Plausible values Conclusions ICTCM 2011 7

Discussion Points n Can this be done on day one? q Yes if can motivate the simulation n n Loaded dice Before reveal the data? ICTCM 2011 8

<<After tactile simulation>> How many infants would need to choose the helper toy for you to be convinced the choice was not made “at random, ” but they actually prefer the helper toy? n Many students can reason inferentially q q q “If a choice is made at complete random, then having 13 infants would be highly unlikely” “Based on the coin flipping experiment, the results stated that at/over 12 was extremely rare. Therefore, at least 12 infants … “Would be around 12 -16 because it seems highly unlikely that given a 50 -50 option 12 -16 would choose the helper toy” ICTCM 2011 9

<<After tactile simulation>> How many infants would need to choose the helper toy for you to be convinced the choice was not made “at random, ” but they actually prefer the helper toy? n But maybe not as well “distributionally” q Is it unusual? = “barely over half” n n Examine language carefully q q n vs. unusual compared to distribution “Unlikely that choice is random” “Prove” “Simulate”, “Repeated this study” “At random” = 50/50, “model” “Random” = anything is possible ICTCM 2011 10

Discussion Points n Can this be done on day one? q Yes if can motivate the simulation n n q Loaded dice Before reveal the data? Enough understanding of “chance model”? Use of class data instead? (“observed” vs. research study) Yes, if return to and build on the ideas throughout the course n So what comes next? ICTCM 2011 11

Discussion Points n Tactile simulation q n Population vs process q n Defining the parameter 3 Ss: statistic, simulate, strength of evidence q n One coin 16 times vs. 16 coins “could have been” distribution of data “what if the null was true” distribution of statistic Fill in the blank wording Timing of final report q Follow-up in-class discussion ICTCM 2011 12

Example 2: Two Proportions n Is Yawning Contagious? q q q Modelling entire process: data collection, descriptive statistics, inferential analysis, conclusions Parallelisms to first example Could random assignment alone produce a difference in the group proportions at least this extreme? Card shuffling, recreate two-way table Extend to own data ICTCM 2011 13

Lab Instructions ICTCM 2011 14

Exam Questions n n n Horizontal axis Shade p-value Make up a research question ICTCM 2011 15

Discussion Points n n n Starting with a significant result but when ready to discuss insignificant? How critical is authentic data? Choice of statistic (count vs. difference in proportion) Role of traditional symbols and notation? Visualization of bar graphs from trial to trial Implementation of predict and test ICTCM 2011 16

Example 3: Two means n Are there lingering effects to sleep deprivation? q q q Randomized experiment Quantitative data Parallel inferential reasoning process n n Index cards Possible follow-up/extensions: what if -4. 33? , medians, plausible values ICTCM 2011 17

Discussion Points n n Role of tactile simulation Scaffolding of lab report q q n When should “normal-based” methods be introduced q q n Introductory sentences, labeling of graphs Write conclusion to journal Alternative approximation to simulation Position, method for confidence intervals Choice of technology q Advantages/Disadvantages n Applets, Minitab, R, Fathom ICTCM 2011 18

Post-Lab Assessment (Fall 2010) n Following the lab comparing two groups on a quantitative variable (65 responses) q q n n n Discuss the purpose of the simulation process What information does the simulation process reveal to help you answer the research question? Essentially correct: 35. 4% demonstrated understanding of the big picture (looking at repeated shuffles to assess whether the observed results happened by chance) Partially: 38. 5% (one of null or comparison) Incorrect: 26. 1% (“better understand the data”) ICTCM 2011 19

Post-Lab Assessment (Fall 2010) n Did students address the null hypothesis? q n Did students reference the random assignment? q n 36. 9% E/ 36. 9% P/ 26. 2% I Did students focus on comparing the observed result? q n 33. 9% E/ 38. 5% P/ 27. 7% I 64. 6% E/ 13. 8% P/ 21. 5% I Did students explain how they would link the pieces together and draw their conclusion? q 24. 6% E/ 60% P/ 15% I ICTCM 2011 20

Student Surveys ICTCM 2011 21

Student Surveys ICTCM 2011 22

Student Surveys n Example 3 simulation ICTCM 2011 23

Student Surveys ICTCM 2011 24

Student Surveys ICTCM 2011 25

Student Surveys n Helper/Hinderer (Winter 2011) – Did the lab help you understand the overall process of a statistical investigation? ICTCM 2011 26

Student Surveys n Did subsequent labs increase understanding? ICTCM 2011 27

Remainder of labs n n Lab 4: Random babies Lab 5: Reese’s Pieces (demo) q q n Lab 6: Sleepless nights (finite population) q n n n Normal approximation, CLT for binary Transition to formal test of significance (6 steps) t approximation, CLT for quantitative, conf interval Lab 7: Simulation of matched-pairs Lab 8: Simulation of regression sampling Chi-square, ANOVA ICTCM 2011 28

Lab Report ICTCM 2011 29

Student Feedback (Winter 2011) n n Google docs survey during last week of course Two instructors ICTCM 2011 30

Student end-of-course surveys (W 11) ICTCM 2011 31

Student end-of-course surveys ICTCM 2011 32

Student end-of-course surveys ICTCM 2011 33

Student end-of-course surveys ICTCM 2011 34

Student end-of-course surveys ICTCM 2011 35

Student end-of-course surveys ICTCM 2011 36

Student end-of-course surveys ICTCM 2011 37

Student end-of-course surveys ICTCM 2011 38

Student end-of-course surveys ICTCM 2011 39

Student end-of-course surveys ICTCM 2011 40

Student end-of-course surveys ICTCM 2011 41

Student end-of-course surveys ICTCM 2011 42

Student end-of-course surveys ICTCM 2011 43

Student end-of-course surveys ICTCM 2011 44

Student end-of-course surveys ICTCM 2011 45

Top 2 most interesting labs n Instructor A q q n Is Yawning Contagious? Heart Rates (matched pairs) Instructor B q q q Friend or Foe Is Yawning Contagious? Reese’s Pieces ICTCM 2011 46

Top 2 most/least helpful labs n Most helpful: q n Friend or Foe Least Helpful (Instructor B): q q Random babies Melting away (intro two-sample t, paired) ICTCM 2011 47

Exam 1 n n In a recent Gallup survey of 500 randomly selected US adult Republicans, 390 said they believe their congressional representative should vote to repeal the Healthcare Law. Suppose we wish to determine if significantly more than threequarters (75%) of US adult Republicans favor repeal. The coin tossing simulation applet was used to generate the following two dotplots (A) and (B). Which, if either, of the two plots (A) and (B) was created using the correct procedure? Explain how you know. ICTCM 2011 48

Exam 1 n 35% picked B (usually citing null. 75 500) q n n n But some look at shape, or later p-value 29% picked A (observed result) 23% neither (wanted. 5 500 = 250) 13% other responses: 0, . 75, 50, can’t tell, anything possible, label is wrong ICTCM 2011 49

Exam 2 n Heights of females are known to follow a normal distribution with a mean of 64 inches and a standard deviation of 3 inches. Consider the behavior of sample means. Each of the graphs below depicts the behavior of the sample mean heights of females. a. One graph shows the distribution of sample means for many, many samples of size 10. The other graph shows the distribution of sample means for many, many samples of size 50. Which graph goes with which sample size? ICTCM 2011 50

Exam 2 n 85% matched n=10 and n = 50 ICTCM 2011 51

Exam 2 Suppose we wish to test the following hypotheses about the population of Cal Poly undergraduate women: n For which graph (A or B) would you expect the p-value to be smaller? Explain using the appropriate statistical reasoning. n ICTCM 2011 52

Exam 2 n 77% picked B q q Mixture of appealing to smaller SD/outliers, larger sample size means smaller p-value, and thinking in terms of test statistic A few choices not internally consistent ICTCM 2011 53

Student understanding of p-value n CAOS questions (final exam) q Statistically significant results correspond to small p-values n n q Recognize valid p-value interpretation n n q Traditional (National/Hope/CP): 69/86/41% Randomization (Hope/CP): 95%/95% Traditional (National/Hope/CP): 57/41/74% Randomization (Hope/CP): 60/72% p-value as probability of Ho - Invalid n n Traditional (National/Hope/CP): 59/69/68% Randomization (Hope/CP): 80%/89% ICTCM 2011 54

Student understanding of p-value n CAOS questions (final exam) q p-value as probability of Ha – Invalid n n q Traditional (National/Hope/CP): 54/48/72% Randomization (Hope/CP): 45/67% Recognize a simulation approach to evaluate significance (simulate with no preference vs. repeating the experiment) n n Traditional (National/Hope/CP): 20/20/30% Randomization (Hope/CP): 32%/40% ICTCM 2011 55

Student understanding of p-value n p-value interpretation in regression (final exam) ICTCM 2011 56

Student understanding of process n Video game question (Final exam: NCSU, Hope, Cal Poly, UCLA, Rhodes College) q q What is the explanation for the process the student followed? Which of the following was used as a basis for simulating the data 1000 times? What does the histogram tell you about whether $5 incentives are effective in improving performance on the video game? Which of the following could be the approximate pvalue in this situation? ICTCM 2011 57

Student understanding of process n Simulation process q q Fall: over 40% chose “This process allows her to determine how many times she needs to replicate the experiment for valid results. ” About 70% pick “The $5 incentive and verbal encouragement are equally effective at improving performance. ” as underlying assumption Still evidence some look at center at zero or shape as evidence of no treatment effect 1/3 to ½ could estimate p-value from graph ICTCM 2011 58

Example – 2009 AP Statistics Exam n A consumer organization would like a method for measuring the skewness of the data. One possible statistic for measuring skewness is the ratio mean/median…. q q Calculate statistic for sample data… Draw conclusion from simulated data … ICTCM 2011 59

Design Principles n n Tactile simulation Visual, contextual animation of tactile simulation Intermediate animation capability Level of student construction q q n Carefully designed, spiraling activities q q n Ease of changing inputs Connect elements between graphs “Stop!” Thought questions Allow for student exploration ICTCM 2011 60

Implementation n n Early in course Repetition through course, connections Normal approximations Lab assignments q q q q n Focus on entire statistical process Motivating research question Follow-up application Thought questions Screen captures Pre-lab questions Minitab demos (Adobe Captivate) Exam questions ICTCM 2011 61

Observations n n Students quickly get sense of trying to determine whether a result could be “just due to chance” Still struggle with more technical understanding q q n Under the null hypothesis Observed vs. hypothesized value Students may fail to see connections between scenarios ICTCM 2011 62

Suggestions/Open Questions n Begin with class discussion/brain-storming on how to evaluate data before show class results q q n Student data vs. genuine research article q n Loaded dice, biased coin tossing Thought questions “the result” vs. “your result” Choice of first exposure q q Significant? Random sampling or random assignment ICTCM 2011 63

Suggestions/Open Questions n Scaffolding q Observational units, variable n q q q How would you add one more dot to graph? At some point, require students to enter the correct “observed result” (e. g. , Captivate) At some point, ask students to design the simulation? Start with fill in the blank interpretation? ICTCM 2011 64

Suggestions/Open Questions n One crank or more? When connect to normal approximations? n How make sure traditional methods don’t overtake once they are introduced? q How much discuss exact methods? Confidence intervals n q ICTCM 2011 65

Summary n Very promising but also need to be very careful, and need a strong cycle of repetition closely tied to rest of course… ICTCM 2011 66