Designing and Fostering Knowledge Building Using Embedded and

Designing and Fostering Knowledge Building Using Embedded and Transformative Assessment Carol K. K. Chan & Eddy Y. C. Lee The University of Hong Kong

Problem and Goals How to assess and examine individual and collective nature of learning in CSCL? n What characterizes knowledge building as “Idea Improvement” (Bereiter, 2002; Scardamalia & Bereiter, 2006)? n How can we design to foster its growth? And would it work for students with diverse backgrounds? To design a knowledgebuilding environment augmented with Assessment principles to characterize and to scaffold collaborative knowledge building 2

Knowledge Building as Collective Cognitive Responsibility (Bereiter, 2002; Scardamalia & Bereiter, 2006) n n n Production & continual improvement of ideas of value to a community Knowledge as conceptual artifacts that can be examined, inquired for improvement of community knowledge Progressive discourse in knowledge-building communities supported by Knowledge Forum. TM 3

Barriers and Challenges for Knowledge Building Assessment of Knowledge Building as Idea Improvement. How to track both individual and collective advances? Knowledge building is only attainable by high achievers. . Could it work for mediocre and lowachieving learners? n n Higher Order thinking for low achievers (Zohar & Dori, 2005) Students of different streams could engage in knowledge-building discourse (Niu & van Aalst, in press) Embedded & Transformative Assessment (Scardamalia, 2002) is part of effort to advance community growth; it is embedded in kb practice 4

Problems & Challenges for CSCL Assessment n Problems & Challenges n n n Online discussion is often fragmented (Hewitt, 2005) Misalignment of learning, collaboration & assessment (Reeve, 200) Changing models of learning need to be aligned with changing practice of assessment n Two metaphors of learning (Sfard, 1998); Knowledge building, a third metaphor? CSCL Assessment: Alignment of learning, collaboration and assessment Assessment of • Learning and Assessment for • Learning Assessment of individual and. collective learning Assessment of knowledge process and products 5

–CSCL Research focuses on quantitative & qualitative analyses in examining collaboration –Alignment of learning, collaboration and assessment CSCL analyses and assessments are often researcherbased tools; few designed to foster student agency How to design assessments that tap into theoretical nature of collaboration while providing pedagogical support in scaffolding student collaboration and understanding? 6

Focus of Study: Assessment as a Design Principle To examine the roles of an embedded and transformative assessment approach to characterize and scaffold collaborative knowledge building for students of different achievement levels • • • Cognitive Apprenticeship & Reflection (Collins, 1989; 2006) Formative Assessment in Project. Based Design (Barron et al. , 2003) Reflective Assessment (Suthers, 2001; White & Fredericksen, 1998); Progress Portfolio From Individual to Collaborative Assessment 7

Design of Embedded and Transformative Assessment 8

Background: Design of Embedded Assessment Using Knowledge-Building Portfolio n Knowledge-building portfolio (Lee, Chan, & van Aalst, 2006; van Aalst & Chan, 2007) - Students’ records and reflections on community’s knowledge advances supported with evidence from the discourse Pedagogical Knowledge building principles n Electronic Portfolios Students submit a portfolio of the best notes in the community that exemplify knowledge building guided by a set of kb principles and write an explanatory statement Portfolios were posted at Knowledge Forum using direct links to original notes n 9

Pedagogical Knowledge-Building principles (van aalst & Chan, 2007) Related to Scardamalia’s kb principles (2002) Working at Cutting Edge n Work on productive problems advancing community knowledge; grapple with different models as they chart their course of advances Progressive Problem Solving n Reinvest efforts in problem solving; ideas are continuously improved and refined Collaborative Efforts n Produce ideas of value to others; contributions to shared goals & values and collective knowledge advances Monitoring Knowledge Advances n Move to meta (higher)level of understanding; negotiate fit between own and other models Constructive Uses of Resources n Knowledge advances require the use of present state and growing edge of knowledge in the field 10

Portfolio note Referencing others’ KF notes 11

Evolution of Design on Assessment Study One: Designing knowledge building portfolios in a joint graduate course (van Aalst & Chan 2001) n Initial Exploration – task design Study Two: Assessing knowledge building using portfolios among Grade 12 students n From individual to Community portfolios (van Aalst & Chan, 2007) Study Three: Knowledge building for Grade 10 students (Lee, Chan & van Aalst, 2006) n Portfolios & Principles scaffolded knowledge building Would Knowledge-Building portfolios and assessment work for students of different achievement and tracking growth ? 12

Portfolio note Referencing others’ KF notes Portfolio Scaffolds 13

Classroom Study: Designing a knowledge-building environment using assessment for student diversity 14

Research Questions 1) What were the roles of knowledge building principles and assessments on learners of different achievement? 2) How did the knowledge building principles and portfolio characterize and scaffold collective knowledge advances? 3) Did students make individual and collective growth and how were they manifested? 15

Design of Knowledge-Building Environment Augmented with Assessment Participants • 141 Secondary Grade 9 students in four different classes in a regular high school in Hong Kong 1) Develop collaborative culture 2) Develop open inquiry aligned with curriculum (posing questions & comments; conjecture; explanation etc. ) 3) Emerging ideas; rise above; and deepening the discourse 4) Embedded and Transformative Assessment 16

Design of Embedded & Transformative Assessment 1) Analytic Toolkit - Students were provided information on database usage formative assessment quantitative indices of participation 2) Rubrics for examining level of questions and explanations – Individual knowledge advances 3) Knowledge building principles and portfolios in note writing and note selection to capture the best knowledge building incidents – Collective Knowledge Advances 17

Research Design – (Knowledge-Building principles x achievement Level) Class KB Principles Ability Portfolio & Rubrics 1 X Low √ 2 X High √ 3 √ Low √ 4 √ High √ 18

Welcome View World Problems: Jigsaw Learning Design Assessment 19

Assessment View ATK Measures 20

21

KB Principles for Writing Good Notes & Portfolio Assessment 22

Data Sources n n n Analytic Toolkit & Database Participation (Burtis, 1998) n Students’ database usage on Knowledge Forum (write, read, links, scaffold, keywords, word, build-on tree) Depth of Inquiry and Depth of Explanation (Chan, Burtis & Bereiter, 1997; Lee et al. , 2006) (individual growth) n Students’ questions and comments rated for inquiry and explanations tracking individual advances Knowledge-Building and Group Review Portfolio (Lee et al. , 2006; van Aalst & Chan, 2007) (individual + collective growth) n n A 6 -point scale to examine portfolio tracking individual and group advances Conceptual Understanding n Students wrote an essay assessing their domain understanding 23

Results I Roles of knowledge building principles & assessment on collaboration and conceptual understanding for students of different achievement 24

Overview of Forum Participation Class Total No. of computer notes Notes per students Low-achievers No kb principles 239 6. 73 Low-Achievers Kb principles High-achievers No kb principles High Achievers Kb principles 282 7. 78 505 12. 9 552 13. 18 25

Differences on Forum Participation (ATK indices) No-KBP Percentage KBP Note link Note read Keywords 26

Differences in Change in Forum Participation (Gain Scores) Over time No-KBP Percentage of Keywords Percentage of note link Scaffolds Views worked Words per note 27

Without KBP & Low Achievers Without KBP & High Achievers With KBP & Low Achievers Percentage Interaction Effects on forum participation for KB Principles and Achievement Levels Linked Notes With KBP & High Achievers Period 1 Views worked in Percentage Keywords Period 2 Period 1 Period 2 28

Interaction Effects on domain understanding for principles and achievement levels Roles of KB Principles on conceptual understanding Interaction effects indicated that the effects of principles were present only in the class for low achievers Differences for low-achievers using kb principles 29

Multiple regression of academic achievement, participation and portfolios on conceptual understanding R R 2 Change Prior Achievement (Gr 9) . 35 . 12*** Forum Participation . 36 . 13 . 009 Portfolio Scores . 41 . 17 . 04* 30

Results 2 n How did portfolios and principles scaffold individual and collective understanding? 31

A portfolio note illustrating assessment using criteria Evaluation of Depth of Inquiry We have chosen this cluster, because their notes have the advantages of: 1. High level of explanation Depth of Inquiry & Explanation 2. High level of questions 3. High level of discussion Schemes Rating This cluster is very good at elaborate people’s ideas and explain their views (will be discussed below): This cluster is very good at using questions to support or generate discussion. Reasons For examples 7 Question Eric Y. First, he stated the problem and asked for further reasons and solution (level 3 -4). In 10 I don’t understand, Eric. Y stated the fact and asked for reasons (level 3 -4). In 11 I am gonna to tell you… he was asking some relevant questions in order to investigate why Less Developed Countries (LDCs) economic situation kept falling even with money borrowed from the MDCs. Level of Inquiry (Question) level 3 -4 32

A portfolio note illustrating a knowledge building principle, and individual-collective knowledge advances • Collective advances • Individual advances 33

Results 3 - Tracking individual and collective advances Did students make individual and collective knowledge building advances and how were they manifested? 34

Depth of Question Level Categorization 1 Questions on definitions, examples or clarification. 2 Factual, topical, general questions 3 Questions with conjectures 4 Explanation-based questions 35

Depth of Explanation Level 1 Categorization n n 2 Expression of agreement or disagreement without evidence support or elaboration Restating an argument Indicating other sources of information or providing examples without discussion Clarification Assemblage of half-truths, invalid argument or contradictions n Factual Information Descriptive or linear treatment of information 3 n Responses supported with relevant information from various sources 4 n Making assertion with logical or relevant examples 5 n n n Refocusing discussion or highlighting key issues for further inquiry Identifying Misconceptions or Invalid Response Identifying Relationship 6 n Synthesizing different ideas into one’s theory 7 n Personal Reflection n 36

Changing Depth of Inquiry and Explanation over time for an inquiry thread (104 notes) Depth of Inquiry Depth of explanation 37

PROBLEM: Overfishing IDEA Fish protection by marine park establishment IDEA Use of DNA Technology Safety/Technical Issue: Ecological Impacts Diverse Idea Economic Issue: Diverse Idea Ethical Issue: Violation of Law of Nature? Rise Above Will increasing fish stocks benefit our economy? Fish Quantity Vs Quality Rise Above Alternate Idea Debate Alternate Effective Solution: Legislation or Education? NEW PROBLEM Who should be responsible? GOD? MDCs? LDCs? Human Beings? Rise Above 38

Implications (1) Characterization and Fostering Knowledge Building n Characterizing n n n Development n n Knowledge building portfolios are collective work and artefacts that illuminate collective growth and ‘improvement of ideas’…. Assessment and growth of both individual and collective knowledge building and their intertwined relations Recognition of knowledge building may be necessary to help students engage in knowledge building Methodology n CSCL often focuses on group cognition in synchronous manner and knowledge-building assessment provides another approach to understand track sustained growth over time…. 39

Implications (2) Knowledge Building for Student Diversity n n n Knowledge building principles can be comprehended by low-achieving students… principles as scaffolds Knowledge building not exclusive to high-achieving students…. may be an approach to address student diversity… community model of diverse expertise… Democratizing knowledge building -- how students of various background understand principles and scaffolds… why and how? 40

Implications (3) Assessment as a Design Principle: From individual to community-based assessment Emerging principles n n n Reflective -- Examine one’s own process and comparing different models & community Transformative -- Assessment of learning and Assessment for learning Agency -- Maximize personal and collective agency…. Student as assessors Epistemic – Thinking & Collaboration Standards – Criteria for Assessment Pervasive – Community engages in assessment to chart progress… Contributing to Others’ Understanding is valued Alignment of model of collaboration with model of Assessment to explain and to design for knowledge advances … 41

Alignment of learning, collaboration & assessment Assessment for learning AND of learning n Examine as well as scaffold collaborative inquiry Assessment of individual AND collective Learning n Examine how idea grow collectively in community Assessment of both knowledge AND process n n n Examine inquiry and subject-matter knowledge Student discussion - Quality, participation, and fragmented online discussion Teacher can have overviews of community progress from portfolios 42

Emerging design principles n n Concurrent and embedded - Align learning goals with assessment Maximize personal and collective agency - Student as assessors Transformative - Provide criteria for assessment Pervasive - Develop a collaborative classroom culture 43

44

45