Thinking and Acting Like a Scientist Investigating the

Thinking and Acting Like a Scientist: Investigating the Outcomes of Introductory Science and Math Courses Kevin Eagan Jessica Sharkness Sylvia Hurtado Higher Education Research Institute, UCLA Association for Institutional Research 49 th Annual Forum May 30 - June 3, 2009 Atlanta, Georgia

Background ¡ Relatively few students earn degrees in natural science or engineering in the U. S. l 15% of U. S. BA degrees are in science/engineering ¡ ¡ Compared to 67% in Singapore, 50% in China, 47% in France, 38% in South Korea U. S. needs more undergraduate science majors to maintain achievement and innovation in science and engineering l Also need to diversify the scientific workforce and increase representation of women and minorities

Background ¡ To graduate more bachelor’s degrees in science, U. S. needs students to choose science majors and to maintain interest in science majors l l ¡ National increases in proportion of freshmen indicating interest in science, technology, engineering and math (STEM) majors However, low proportion of students who intend to major in STEM actually graduate with STEM majors One obstacle to STEM major completion: Introductory “gatekeeper” courses l l Mechanism for sorting students What is rewarded?

Introductory “Gatekeeper” Courses ¡ ¡ First course in a series of courses in which knowledge is cumulative In science – relatively high drop-out and failure rates in gatekeeper courses l l Large lectures Un-engaging Highly competitive Grading on a curve

Classroom Environments & Instructor Pedagogies ¡ Classroom climates have an impact on learning and performance l ¡ Competitive environments have negative impact on learning, performance, retention, self -confidence Collaborative environments that emphasize group work can mitigate negative effects of large lectures and competitive environments l Can also promote critical thinking about scientific concepts and their applications

Supportive Learning Environments and the Skills Needed for Scientific Success ¡ Six necessary conditions for a supportive learning environment: l l ¡ Quality of instruction, Teacher’s interest, Social relatedness, Support of competence, Support of autonomy Engender greater self-motivation, encourages self-directed learning Two primary pedagogical techniques in science l l Domain-specific learning = memorization of facts and causal relationships Domain-general learning = reasoning strategies and critical thinking skills

Additional influences on student success in STEM courses ¡ Experiences external to the classroom environment l l ¡ Participation in research projects Peer Tutoring Prior academic achievement and preparation l Most significant influence on outcome of introductory courses?

Goals of Current Study ¡ ¡ Untangle effect of prior preparation and background factors from performance assessment in introductory STEM courses Identify how students develop in introductory courses the critical thinking dispositions necessary for science careers, and whether these dispositions are reflected in student grades

Conceptual model High School Science Achievement Tutoring & Research Participation Demographic variables Course Grade Amount of student effort expended on course Course Learning Environment & Pedagogy Ability to act and think like a scientist (Pre-Test) Critical thinking dispositions Ability to act and think like a scientist (Post-test)

Data & Sample ¡ ¡ ¡ Data collected via online survey from students in 12 introductory science and math courses at 5 institutions Two surveys – one at beginning of course (pre-survey) and one at end (post-survey); final analytic sample = 255 Final longitudinal sample: l l 70% female 34% White, 43% Asian, 8% Black, 13% Latino 86% majoring in STEM field 64% first-year students, 27% second-year

Thinking and Acting like a Scientist Factor Items* Thinking like a scientist pre-test See connections between different areas of science & math Understand scientific concepts Identify what is known and not known in a problem Ask relevant questions Draw a picture to represent a problem or concept Make predictions based on existing knowledge Come up with solutions and explain them to others Investigate alternative solutions to a problem Understand/translate scientific terminology into non-scientific language Acting like a scientist pre-test Relate scientific concepts to real-world problems Synthesize or comprehend several sources of information Conduct an experiment Look up scientific research articles and resources Memorize large quantities of information Loadings Pre-test Post-test 0. 78 0. 77 0. 72 0. 66 0. 75 0. 71 0. 74 0. 73 0. 80 0. 76 0. 72 0. 58 0. 82 0. 81 0. 73 0. 72 0. 77 0. 85 0. 64 0. 60 0. 61 0. 80 0. 78 0. 71 0. 62 0. 53 Measurement model fit statistics: χ2=300. 69 (305, N=255), NNFI = 0. 98, CFI = 0. 98, RMSEA = 0. 03, reliability = 0. 82. *All items were asked as part of a questions stem that read, “Rate your ability in the following areas as it pertains to your academic learning in the sciences. ” Response options were Major Strength (5), Above Average (4), Average (3), Below Average (2), Major Weakness (1)

Variables ¡ Independent Variables: l l l ¡ Demographic characteristics Prior preparation Course pedagogy Classroom environment College experiences Critical thinking dispositions (CCTDI subscales) Dependent Variables l l Final course grade Dispositions toward science

Analysis Plan ¡ Identification of latent constructs (factors), representing acting like a scientist and thinking like a scientist l ¡ Exploratory Factory Analysis Confirmatory Factor Analysis (measurement model in SEM) Structural Equation Modeling (SEM) to model how student experiences in introductory courses affect three outcomes

Structural Modeling Procedure ¡ ¡ Added hypothesized predictors and paths to structural equation model Used prior research, theory and Wald and La. Grange multiplier tests to add and remove paths that did not contribute to model fit l When all paths were deleted from a variable, variable was removed from the model

Results (non-significant paths not shown) External Support/ Science Experiences Course Learning Environment Student Effort Consistently got support needed HPW Engage in lab activities Felt competition in course HPW prof’s research project (college) Crammed for exams Felt Overwhelmed Course employed group activities Sought tutoring on campus Demographics Participated in HS research program Income Pretests URM Student Avg HS GPA in STEM courses Female AP Chemistry Score Tutored Student in HS Grade in intro course Format primarily lecture Think like a scientist Post-test Act like a scientist Post-test BBS Major Act like a scientist Pre-test Think like a scientist Pre-test Course Pedagogy Open. Critical Thinking Mindedness Confidence Critical thinking dispositions Analyticity

Discussion of findings: Thinking and Acting Like a Scientist ¡ CCTDI – Openmindedness negatively predicted both dispositions l ¡ CCTDI – Critical thinking self-confidence positively predicted both dispositions l ¡ Indicative of students’ development of domain-general science skills CCTDI – Analyticity positively predicted thinking like a scientist l ¡ Relativism in conflict with objective, empirical perspective of science Indicative of students’ ability to think more carefully and critically during problem-solving activities Students who felt overwhelmed scored lower on both dispositions l May be indicative of naïveté about what is required of science majors

Discussion of findings: Final Course Grade ¡ Thinking and acting like a scientist and CCTDI subscales unrelated to final grade l ¡ Suggests that introductory courses focus too much on the acquisition of knowledge rather than development of higher-order thinking skills Final grade in large part predicted by prior preparation (high school grades, research) l Suggests that failure to earn top grades may merely lack the preparation necessary for success in these courses rather than sciencerelated skills

Discussion of findings: Final Course Grade ¡ Cramming for exams positively predicted course grades l ¡ Group work positively predicted final course grade l ¡ Underscores that grades reward rote memorization rather than development of higher-order thinking skills Supports prior research that concluded that peer learning provides learning reinforcement, which may have longer-term benefits Tutoring (receiving and providing) positively predicted final course grade l Reinforces the benefits of peer-to-peer teaching and learning

Implications and Conclusions ¡ ¡ ¡ Can we afford to cram content into courses at the expense of development of scientific skills and thinking? Adjust grading practices so that they more accurately reflect learning rather than prior preparation Further examination of pedagogical practices and interventions in large, lecture -based gatekeeper courses is needed

Contact Information Faculty and Co-PIs: Sylvia Hurtado Mitchell Chang Administrative Staff: Aaron Pearl Graduate Research Assistants: Kevin Eagan Jessica Sharkness Lorelle Espinsoa Minh Tran Christopher Newman Paolo Velasco Papers and reports are available for download from project website: http: //heri. ucla. edu/nih Project e-mail: herinih@ucla. edu Acknowledgments: This study was made possible by the support of the National Institute of General Medical Sciences, NIH Grant Numbers 1 R 01 GMO 71968 -01 and R 01 GMO 71968 -05 as well as the National Science Foundation, NSF Grant Number 0757076. This independent research and the views expressed here do not indicate endorsement by the sponsors.