THE ORIGINS BUILDAMEL A SCAFFOLD TO EXPLORE THE

THE ORIGINS BUILD-A-MEL: A SCAFFOLD TO EXPLORE THE UNIVERSE’S ORIGINS Timothy G. Klavon Janelle M. Bailey Archana S. Dobaria Doug Lombardi

2 THE ORIGINS BUILD-A-MEL: A SCAFFOLD TO EXPLORE THE UNIVERSE’S ORIGINS Our aims and supporting theory Plausibility Judgements in Conceptual Change Our instrument Model-Evidence Link Diagram (MEL) Our research Aligning student discourse with written explanations The next step in MEL diagram The Origins of the Universe Build-a-MEL activity

3 Evaluative reasoning is foundational to many—if not all —scientific practices (Ford, 2015) The three “spheres” of activity for scientists (NRC, 2012, p. 45)

4 Plausibility: A judgment about the “truthfulness” of explanations (Lombardi et al. , 2016) Epistemic judgement Often unconscious Tentative Open for re-appraisal Other types of judgments concern evidence, such as credibility and reliability

5 Explicit & purposeful evaluation about scientific explanations may facilitate shifts in plausibility & deeper understanding But… how do we achieve this? The Plausibility Judgements in Conceptual Change (PJCC) model (Lombardi et al. , 2016)

6 Model-Evidence Link (MEL) Diagrams (Chinn & Buckland, 2012; Lombardi, et al. , 2016, 2018)

7 Model-Evidence Link Diagrams (Chinn & Buckland, 2012; Lombardi, et al. , 2016, 2018) l o ca i m on r t s a g n gi 020) d Ju l. , 2 : g ta n i rn ey e a le ail t en 3; B d stu F-0 t or PAR p up ion s to ess s ld 20 S o f af SM c S PT e se (AA , re ns o m atio r Fo plan ex

8 Provides a social context for scientific discourse…

9 … but do we get them to write about what they talk about?

10 Our current research… Looks at the alignment of MEL group discourse and written explanations around evaluation Subset of MEL research participants with transcripts Quantitatively. Impact of student discourse quality on predictive structural equation modeling Qualitatively. Text comparison analysis of transcripts and explanation tasks

11 Methodological Framework Discourse Evaluation Sociocultural Discourse Analysis Lombardi, Brandt et al. (2016) Mercer (2004) Identifies 4 levels of evaluation Language as a tool of culture Erroneous Archetypes of Talk Descriptive Cumulative Relational Exploratory Critical Disputational

12 Materials and Procedure Materials Previously scored explanation task from MEL 1 Project Transcripts of student work group conversations SDA rubric Evaluation rubric Text analysis rubric Procedure Organization of data Identification of episodes Transcript coding Calibration Reliability Comparative text analysis

13 Discourse Analysis Rubrics

14 SDA Codes Preliminary Coding Scheme for Sociocultural Discourse Analysis (Mercer, 2004, 2010) Archetype 1. Non-productive talk Description Participant is off-task, erroneous, or disruptive to group dynamic 2. Cumulative talk 3. Exploratory talk 4. Disputational talk Off-task behavior Erroneous statement about the task Erroneous statement about the science content Code D 1 o D 1 et D 1 ec Participant builds upon group discourse without being critical Participant repeats statements of others Participant confirms statements of others Participant elaborates on statements of others D 2 r D 2 c D 2 e Participants critically engage with each other's talk in a constructive manner Participant offers a statement for consideration Participation challenges or counterchallenges with justification Participant offers an alternative hypothesis Participants seek and consider opinions prior to decision-making Participant self-questions or expresses doubt about their own thinking D 3 c D 3 j Participant exhibits individualized decision-making Participant critically disagrees with another participant Participant critically disagrees with an evidence Participant critically disagrees with a model Participants partake in short exchange of assertions and challenges (Yes, it is! No, it’s not!) D 4 i D 4 do Participants engage in critical discourse through individual decision making and disagreement. Disagreements are constructive and civil in nature. Examples D 3 ah D 3 o D 3 q D 4 de D 4 dm D 4 se

Evaluation Codes 15 Preliminary Coding Scheme for Sociocultural Discourse Analysis (Lombardi, et al. , 2016) Archetype 1. Erroneous evaluation Description Erroneous—Explanation contains incorrect relationships between evidence and model. The explanation may also be mostly inconsistent with scientific understanding and/or include nonsensical statements. (see Appendix B) Examples A statement is inconsistent with scientific understanding A statement is nonsensical A statement inaccurately portrays the relationship between an evidence and a model Code E 1 i 2. Descriptive evaluation Descriptive—Explanation contains a correct relationship without elaboration, or correctly interprets evidence without stating a relationship. (see Appendix B) A correct relationship without elaborating E 2 e A correct interpretation of an evidence without stating a relationship E 2 wr The use of elimination-based logic when evaluating a “has nothing to do with the model” relationship. E 2 ebl 3. Relational evaluation Relational—The explanation addresses text similarities and includes both specific evidence and an associated model or reference to a model. (see Appendix B) The evaluation distinguishes between evidences and models in an associative and/or a correlational manner The evaluation is possibly based on text similarities. 4. Critical evaluation Critical—Explanation describes a causal relationship and/or meaning of a specific relationship between evidence and model. (see Appendix B) The evaluation includes a cause and effect relationship The evaluation is correct (not including has nothing to do with) and shows evidence of deeper thinking The evaluation distinguishes between evidence and explanatory models The evaluation allows for sophisticated connections between evidences and models The evaluation concurrently examines alternative models E 1 n E 1 r E 3 ac E 3 ts E 4 ce E 4 dt E 4 em E 4 sc E 4 eam

16 Analysis: Descriptive Statistics for Participation Rate, Average Discourse Score, Average Evaluation Score, and Total Discourse Score (N = 60) Mean S. D. . 11. 41 Skewness Statistic Std. Error. 195. 309. 155. 309 Kurtosis Statistic Std. Error -. 517. 608 -. 762. 608 Participation Rate Average Discourse Type Score . 29 2. 74 Average Discourse Evaluation Score 2. 06 . 45 . 275 . 309 . 578 . 608 Average Total Discourse Score 4. 77 . 67 . 430 . 309 -. 341 . 608

17 Quantitative Analysis Test SEM (t. SEM)

18 Quantitative Analysis Test SEM (t. SEM) Structural Equation Model Path Coefficients for the Test Model (N = 60) Preplausibility Postplausibility Evaluation Pre-plausibility Postplausibility Evaluation --- 0. 255* 0. 162 --- 0. 345*** Preknowledge Postknowledge Qo. C Notes: * p <. 05, ** p <. 01, *** p <. 001 --- Pre-knowledge Post-knowledge Qo. C (-0. 122) 0. 318** --- 0. 11 0. 05 (-0. 233) * 0. 357** 0. 213* 0. 07 -----

19 Quantitative Analysis Experimental SEM (e. SEM)

20 Quantitative Analysis Experimental SEM (e. SEM) Structural Equation Model Path Coefficients for The Test Model (N = 60) Preplausibility Pre-plausibility --- Post-plausibility Evaluation 0. 259* 0. 187* Post-plausibility --Evaluation Pre-knowledge Postknowledge 0. 357*** . 312 0. 110 --- 0. 312 (-0. 233) 0. 357 Pre-knowledge --Post-knowledge --Notes: * p <. 05, *** p <. 001

21 Qualitative Analysis Rubric Analytical Frameworks Used for Analyzing the Explanation Tasks and Alignment with Episode Discourse Evaluation framework Alignment scores Component Criteria Score Component Criteria Evaluation of link Does the student link match the one spoken during the activity? 0 -1 Level of alignment Does the student explanation align with 0 -3 the explanation developed by the 0 - not at all group? 3 - matches word for word Level of agreement Does the student’s explanation reflect their original thinking over the group’s developed one? Level of evaluation Did the student change during the discourse? 0 -1 How well does the level of evaluation of the discourse match the written evaluation? (-1)-1 (-1)- written < discourse 0 - written = discourse 1 - written > discourse Score 0 -1 0 - no 1 - yes

22 Qualitative Analysis Selection Selected Students for Alignment Analysis High DES Low DES High Early Evaluation High-High (HH) Student 4206 High-Low (HL) Student 4101 Low Early Evaluation Low-High (LH) Student 3408 Low-Low (LL) Student 2625

23 Qualitative Analysis Student 4206 (HH) Time Codes 03: 48. 3 04: 02. 6 Speaker Student 4206 04: 02. 6 04: 04. 4 04: 05. 6 Student 4210 Student 4205 Student 4222 04: 04. 4 04: 04. 5 04: 07. 4 Transcription … I had that it… uh… I had it supports. I got contradicts. I said nothing to do with. I said supports. First Discourse Student made a correct link, but only described that link. Scored a 2, the lowest score a correct answer can earn The Explanation Task (ET) was written the same way Time Codes Speaker Transcription 00: 24. 5 00: 29. 9 Student 4210 00: 28. 9 00: 29. 9 Student 4206 Evidence one to model A. I have supports. That's what I got, what did you get? Supports. 00: 29. 9 00: 32. 9 Student 4210 00: 32. 9 00: 36. 2 Student 4206 You got supports? So, that looks like that was the easy one. That goes to supports! Evidence one model B? Second Discourse Also descriptive, though the ET was more relational and scored a 3.

24 Qualitative Analysis Student 4101 (HL) Spoke in only one episode Indications of descriptive language Use of past tense in transcript Use of “talks about” in ET

25 Qualitative Analysis Student 3408 (LH) Use of past tense Indicating that links were formed individually, prior to discourse Evaluation scores rise from 1(erroneous) to 2 (descriptive) Time Codes 00: 20. 8 00: 21. 4 Speaker Student 3405 Transcription So— 00: 21. 4 00: 31. 9 Student 3415 00: 31. 9 00: 33. 9 Student 3405 For the first piece of evidence, um, I— I said it would—it supported model A. Um — [Whispers] (Student name), shut up! 00: 33. 9 00: 36. 9 Student 3413 I said it strongly supported. 00: 36. 9 00: 37. 8 Student 3405 Yeah, I said— 00: 37. 8 00: 38. 7 Student 3408 Strongly supported. 00: 38. 7 00: 39. 8 Student 3405 I just said supports. 00: 39. 8 00: 42. 7 Student 3408 Okay. Be that way. 00: 42. 0 00: 44. 8 Student 3415 And I said it contradicted model B. 00: 44. 8 00: 44. 9 Student 3413 Yeah. 00: 45. 1 00: 46. 8 Student 3405 Yeah, I said—I also said that. 00: 46. 8 00: 50. 1 Student 3408 00: 50. 1 00: 50. 6 Student 3415 Yeah, I said it contradicted model B as well. Okay.

26 Qualitative Analysis Student 2625 (LL) Descriptive language No attempt to expressing the thinking behind this statement Transcript and ETs use this level of evaluation “I feel as if evidence one has nothing to do with model A because model A discusses about the Moon was once an object, and that Evidence 1 talks about the density. ”

27 Qualitative Analysis Reflects findings of the MEL Project and the quantitative analysis Mean evaluation scores of just above 2 Qo. C aligns with evaluation in e. SEM

28 Analysis: Summary Quality of Conversation construct has a significant pathway to post-plausibility through evaluation Text analysis of both transcripts and ETs show high levels of descriptive evaluation

29 Where do we go from here? Student agency facilitates learning (Roth, 2008) The Build-A-MEL (ba. MEL) Evolution of the preconstructed MEL (pc. MEL) Students select MEL components Two of three models Four of eight lines of evidence

30 Where do we go from here? Student agency facilitates learning (Roth, 2008) The Build-A-MEL (ba. MEL) Evolution of the preconstructed MEL (pc. MEL) Students select MEL components Two of three models Four of eight lines of evidence

31 Where do we go from here? Student agency facilitates learning (Roth, 2008) The Build-A-MEL (ba. MEL) Evolution of the preconstructed MEL (pc. MEL) Students select MEL components Two of three models Four of eight lines of evidence

32 Where do we go from here? Student agency facilitates learning (Roth, 2008) The Build-A-MEL (ba. MEL) Evolution of the preconstructed MEL (pc. MEL) Students select MEL components Two of three models Four of eight lines of evidence Complete as the pc. MEL

33 Thank you! Dr. Timothy G Klavon Tim. Klavon@temple. edu www. sciencelearning. net https: //serc. carleton. edu/mel/index. html