Training Spatial Skills A Metaanalysis Linda Liu Hand

Training Spatial Skills: A Meta-analysis Linda Liu Hand, David H. Uttal & Loren Marulis Northwestern University Nora S. Newcombe Temple University 1

Importance of training? • Potential to improve skills relevant to STEM (Hedges & Chung, in prep; Shea, Lubinski & Benbow, 2001) – High spatial ability: More likely to have STEM major and STEM job – Can also reduce disparities in STEM achievement • How and how much? • Goal: To aggregate systematically past research on spatial training to determine consensus in literature. 2 http: //www. spatialintelligence. org

Overview v What is training? v How can we compare training effectiveness across studies? v Research questions: 1. How much do (vs. can) spatial skills improve? • Might vary by task – Embedded Figures vs. Water-level Task? 2. What works? • Impact of grouping variables 3. Are training effects durable? 4. Does training generalize (transfer) to untrained tests? 3 http: //www. spatialintelligence. org

Examples: Video games • Effect of playing videogames (Tetris) on mental rotation and Paper Folding Test (Wright, Thompson, Ganis, Newcombe & Kosslyn, 2008). MRT (g = 1. 09) 4 http: //www. spatialintelligence. org PFT (g =. 87)

Examples: Spatial experience • Effects of various life experiences on spatial skills: – Fashion designers: Effect of experience with patternmaking on spatial skills (Workman, Caldwell & Kallal, 1999) Dress pattern making Differential Aptitude Test-Spatial Relations (g =. 32) 5 http: //www. spatialintelligence. org

Examples: Spatial coursework Engineering course using multi-media software and workbook Improved Purdue Spatial Visualization Test performance (Sorby, 2008) • Isometric pictorials from coded plans • Multi-view drawings • Paper folding/2 -D to 3 -D transformations • Object rotations about one axis • Object rotations about two or more axes • Cutting planes and cross sections • Surfaces and solids of revolution • Combining solids g = 2. 02 6 http: //www. spatialintelligence. org

Examples: Repeated practice on different Group Embedded Figures (Chance & Goldstein, 1971; Schaeffer & Thomas, 1999) Pretest GEFT Training Posttest GEFT g = 1. 12 http: //www. spatialintelligence. org

Methods • Searched for both published and unpublished work: – Dissertations, conference posters, technical reports. – Electronic searches, references lists, direct contacts • Coded on several grouping variables, including: – Age, sex, ability level (i. e. , prescreened for low performers? ) – Outcome measure, type of training – Publication status, random assignment, location of study (classroom? ), feedback provided (yes/no), training frequency 8 http: //www. spatialintelligence. org

Effect Sizes • Standard measure of efficacy across studies – Does not depend on individual measurement (raw score) – Expresses mean change, as a result of training or experience, in standard deviation units. • Final “sample” – 101 published (76) and unpublished (25) studies 9 http: //www. spatialintelligence. org

Analysis Plan • How do we make sense of various training methods and dependent variables? • Created 5 conceptual categories of dependent variables and 3 categories of training. • Describe each category then compare size of training effects in each category. 10 http: //www. spatialintelligence. org

• Categories of dependent variables Spatial perception Perceive objects amidst distracting background Perspective taking Visualize a scene from a different location Assembly/ transformation Put together objects into larger config. or transform (3 -D to 2 -D) Mental rotation Rotation of 2 -D or 3 -D pictures or objects Spatial principles Understand abstract principles (e. g. , horizontality) 11 http: //www. spatialintelligence. org Example: Mazes, Embedded Figures Task

• Categories of dependent variables Spatial perception Perceive objects amidst distracting background Perspective taking Visualize a scene from a different location Assembly/ transformation Put together objects into larger config. or transform (3 -D to 2 -D) Mental rotation Rotation of 2 -D or 3 -D pictures or objects Spatial principles Understand abstract principles (e. g. , horizontality) 12 http: //www. spatialintelligence. org Example: Three mountains task

• Categories of dependent variables Spatial perception Perceive objects amidst distracting background Perspective taking Visualize a scene from a different location Assembly/ transformation Put together objects into larger config. or transform (3 -D to 2 -D) Mental rotation Rotation of 2 -D or 3 -D pictures or objects Spatial principles Understand abstract principles (e. g. , horizontality) 13 http: //www. spatialintelligence. org Example: Form Board Test Yes or No?

• Categories of dependent variables Spatial perception Perceive objects amidst distracting background Perspective taking Visualize a scene from a different location Assembly/ transformation Put together objects into larger config. or transform (3 -D to 2 -D) Mental rotation Rotation of 2 -D or 3 -D pictures or objects Spatial principles Understand abstract principles (e. g. , horizontality) 14 http: //www. spatialintelligence. org Example: MRT, Card rotation Same or different?

• Categories of dependent variables Spatial perception Perceive objects amidst distracting background Perspective taking Visualize a scene from a different location Assembly/ transformation Put together objects into larger config. or transform (3 -D to 2 -D) Mental rotation Rotation of 2 -D or 3 -D pictures or objects Spatial principles Understand abstract principles (e. g. , horizontality) 15 http: //www. spatialintelligence. org Example: Water level task A B Draw a line in B for the water.

Types of Training Video games Designed for recreation and entertainment. Courses Full-length or short-term enhanced courses. Spatial task training - Specific Direct rehearsal or practice on outcome measure of interest. Spatial task training- Transfer of training to reference tests. 17 http: //www. spatialintelligence. org Example: Tetris, Zaxxon

Video games Designed for recreation and entertainment. Courses Full-length or shortterm enhancements. Spatial task training - Specific Direct rehearsal or practice on outcome measure of interest. Spatial task training- Transfer of training to reference tests. Example: Dress-making, spatial modules, Drafting (vs. water purification) 18 http: //www. spatialintelligence. org

Video games Designed for recreation and entertainment. Courses Full-length or short-term enhancements. Example: Repeated practice on the GEFT, VMRT, WLT Spatial task Direct rehearsal or training - Specific practice on outcome measure of interest. Spatial task training- Transfer of training to reference tests. Same or different? 19 http: //www. spatialintelligence. org A B

Video games Designed for recreation and entertainment. Courses Full-length or short-term enhancements. Spatial task training - Specific Direct rehearsal or practice on outcome measure of interest. Spatial task Transfer of training to training -Transfer reference tests. 20 http: //www. spatialintelligence. org Example: Regular WLT test on irregular WLT; Tetris test on PFT A B

Results • Overall effectiveness of training • Control group effects • Age and Sex • Are some kinds of training better than others? Are some outcome measures more malleable than others? • Duration • Transfer

Overall Effectiveness • 101 studies – Mean effect size =. 65 (i. e. , 2/3 a SD of improvement) – “Moderate” improvement (Cohen, 1988) For IQ (SD = 15), . 65 SD would be an increase of 9. 75 points.

Control group effects

Experimental groups do significantly exceed control groups • Overall, treatment groups improve significantly more from training than control groups do. Control groups g =. 56 24 Treatment groups g =. 75 g http: //www. spatialintelligence. org

But, …. • Type of control group and improvement in control group really matters for understanding – Overall effectiveness • What works depends on what did or did not happen to the control group – Different effects of different types of training 26 http: //www. spatialintelligence. org

Why control groups matter • Crestor = New anti-cholesterol drug. • Similar drug Vytorin halted = Clinical trial failed to show it was any better than an alreadyavailable medication. • Difference? – Vytorin study: Head-to-head comparison with another drug. – Crestor study based on comparing its effectiveness to placebo = nothing 27 http: //www. spatialintelligence. org

Why control groups matter • Important to separate control and treatment groups – Spatial principles highest Ec effect size, lowest Control group g – Spatial perception lowest Ec effect size, highest g for Control. A B 28 http: //www. spatialintelligence. org

Why control groups improve so much • Classic sources of test-retest effects – Understanding the task (e. g, which key to press when) – Test-specific strategies (e. g. , eliminating foils quickly; keeping fingers on the correct keys; looking for similar problems) • But also, the possibility of more interesting learning from the tests – Fluency in finding correct structures – Better allocation of attention and working memory – Multiple tests provide a form of indirect training • Alignment and comparison (Gentner & Markman, 1994, 1997) 29 http: //www. spatialintelligence. org

Our specific claim • Some of the learning in the control groups is not just of the boring type • Some people learn something from taking the tests • Points to malleability of spatial skill – Even without instruction, just a chance to practice, people improve, often rather dramatically 30 http: //www. spatialintelligence. org

What’s the evidence to support this claim? • Acknowledge: Post-hoc—NEED EXPERIMENTAL RESEARCH • But…. • Magnitude of control group improvement is about twice as great as for other tests • Control groups show transfer! – Hard to explain on basis of “boring” effects alone • Variation helps 31 http: //www. spatialintelligence. org

Test variety is effective training • Test-retest effect: Not just number of repetitions • Number of separate tests given during pretest-posttest: 32 http: //www. spatialintelligence. org

Age effects: It’s all in the control group • Does malleability vary by age? – On average, effect size significantly higher for children than for adults, p <. 05 – Initially, appears that children are more malleable… 33 http: //www. spatialintelligence. org

Age effects: Control and Experimental • When looking at control groups, adult control groups improve significantly more. • Thus, children may appear more malleable because their control groups improve less than adults’ do. Age Children Adults Treatment Group Effect Size Control Group Effect Size . 70 (. 05). 77 (. 03) (ns) . 42 (. 05). 62 (. 03) p <. 01 34 http: //www. spatialintelligence. org

Sex differences • Does malleability vary by sex? No difference in mean effect size, both sexes respond to training (same g) • Overall, results from prior work are most consistent with last scenario. • Male advantage is similar in magnitude at pre and post M F Training 35 http: //www. spatialintelligence. org

Training works – What works? • Focus on treatment effect sizes: Grouping variable Results Outcome measure Outcomes largely similar in malleability: Only significant difference: Spatial perception (. 96) > mental rotation (. 67). Training frequency More frequent training larger g for mental rotation only (. 81 for multi-session vs. . 38 for single). Feedback during training FB led to larger effect sizes for most outcome measures except spatial perception (opposite is true) Random assignment Led to lower effect sizes (more rigorous). Variables involved in Age, Publication status, Location (classroom or not) NO significant effects 36 http: //www. spatialintelligence. org

Training works – What works? • Focus on treatment effect sizes: Type of training Courses Full-length superior to short-term enhancement • Due to training being distributed (over longer period of time, in more sessions) • Even though total number of hours of training did not differ for the two. Video games Results similar for all outcome measures, but video games involving mental rotation produced especially strong training effects. 37 http: //www. spatialintelligence. org

Duration: Training lasts • Majority of studies (85%) tested only immediate impact of training. • Among treatment groups: No significant decline in effect size measured immediately, 2 weeks after, or more than 2 weeks after end of training (which includes up to 3 months later). 38 http: //www. spatialintelligence. org

Training transfers • Why does this matter? – Suggests training is NOT just a practice effect – If spatial training has effects that extend beyond mere practice, training should transfer to untrained tasks. • Near vs. Far transfer: – Near g = 1. 01 – Far g =. 56 A B – But Far is more durable. 39 http: //www. spatialintelligence. org

Training transfers • Studies expecting to obtain far transfer might use training that produces especially durable effects: 40 http: //www. spatialintelligence. org

Real impact of training? Real value of a. 65 SD increase? Marginal improvements on raw scores may lead to important gains in other areas: • Increase of. 65 SD in height = 1. 63 inches in height (among females 18 -24 yrs)? • Increase of. 65 SD on the LSAT = 3 points (Average score is 156/180 among current law school students)? • Value of 1 inch increase above average? $789 per year 1 • Value of 1 point gain on LSAT? $2, 600 in starting salary 2 1 Berkowitz, Ruth. "One Point on the LSAT: How Much Is It Worth? " American Economist 42 (2) 1998. 2 Judge, T. A. , & Cable, D. M. “The Effect of Physical Height on Workplace Success and Income” Journal of Applied Psychology, 89(3) 2004.

M F Training 42 http: //www. spatialintelligence. org

Conclusions • Training leads to improvements in spatial skills that are: – Durable - No significant losses in pretest-posttest improvement, even when retested 3 months later. – Generalizable to other tasks – Training leads to improvements on untrained tasks. 43 http: //www. spatialintelligence. org

Conclusions • How much can spatial skills improve? – Use longer periods of training • 47% of studies performed only one single session of training • 85% conducted only an immediate posttest • When long periods of training are used, durable effects AND far transfer are observed. – Test a larger range of outcome measures • 48% of outcome measures are mental rotation • Vs. 9% perspective taking, 11% spatial principles, etc. – Include a variety of methods of training • Allows for alignment and comparison across problems (Gentner & Markman, 1994, 1997) 44 http: //www. spatialintelligence. org

Future directions • To develop best-practice guidelines for spatial interventions at elementary and high school levels. • Investigate transfer to STEM in more detail. • Understand thresholds for success 45 http: //www. spatialintelligence. org

Acknowledgements • • • Larry Hedges (NU) Spyros Konstantopoulos (NU) David B. Wilson (George Mason University) Chris Warren and Alison Lewis Research assistance: – Kate O’Doherty – Bridget O’Brien – Eleanor Tushman – Maggie Carlin – Laura Mesa, Bonnie Vu, Melissa Sifuentes 46 http: //www. spatialintelligence. org