Using PLA to Liberate Learning PLA participatory learning
Using PLA to Liberate Learning (PLA: participatory learning approach) Michael Bieber, Jia Shen, Dezhi Wu, Vikas Achhpiliya Information Systems Department College of Computing Sciences New Jersey Institute of Technology http: //web. njit. edu/~bieber November 2003 Bieber et al. , NJIT © 2003 1
Outline • Motivation • PLA: Participatory Learning Approach • A bit of theory • Experimental results • Interesting issues Bieber et al. , NJIT © 2003 2
Motivation • To increase learning of course content • Learning through active engagement – involve students as active participants – with the full problem life-cycle – through peer evaluation • Minimize overhead for instructors Bieber et al. , NJIT © 2003 3
Outline • Motivation • PLA: Participatory Learning Approach • A bit of theory • Experimental results • Interesting issues Bieber et al. , NJIT © 2003 4
PLA Process • • All entries posted on-line Each student creates 2 exam problems Instructor edits the problems if necessary Each student solves 2 problems Students evaluate (grade) the solutions to the problems they authored, writing detailed justifications Ph. D. students evaluate each problem a second time Instructor gives a final grade optional: Students can dispute their solution’s grade, by evaluating it themselves and writing detailed justifications Instructor resolves the dispute Bieber et al. , NJIT © 2003 5
Bieber et al. , NJIT © 2003 6
Instructor Control Process Course Process. Design Flow: Learning from doing Set up on-line the PLAenvironment activities from Examadditional Processlearning Control reading Assigneverything ID peers write Edit questions Assign who answers questions Assign level-2 graders Determine Final Grades Resolve Disputes Bieber et al. , NJIT © 2003 Student Learning Process Make up problems Read Solve problems - other solutions - grade justifications - disputes Level-1 and Level-2 graders grade solutions Dispute final grade 7
Instructor Control Process Student Learning Process Confirmation Course Design Set up on-line environment ID, understand process Make up problems Read Exam Process Control Assign ID Edit problems Assign who solves problems Assign level-2 graders Determine Final Grades Resolve Disputes Bieber et al. , NJIT © 2003 Solve problems - other solutions - grade justifications - disputes Level-1 and Level-2 graders grade solutions Dispute final grade 8
Evaluation (grading) • Evaluation includes: – Written critique or “justification” (positive or negative) – Optional: separate sub-criteria to critique • Solution result is correct and complete (40%) • Solution was well explained (30%) • Solution demonstrated class materials well (10%) • Solution cited appropriate references (20%) – Grade (optional; recommended to save instructor time) • Evaluation/grade may be disputed (optional) – Student must re-evaluate own solution when disputing Bieber et al. , NJIT © 2003 9
Instructor should provide… • Detailed instructions and timetable • Solution: what is expected • Critiquing and grading guidelines Bieber et al. , NJIT © 2003 10
Outline • Motivation • PLA: Participatory Learning Approach • A bit of theory • Experimental results • Interesting issues Bieber et al. , NJIT © 2003 11
Constructivism (Learning Theory) • The central idea is that human learning is constructed, that learners build new knowledge upon the foundation of previous learning {learning throughout the exam process} • Two classic categorizations – Cognitive Constructivism (Piaget’s theory) – Social Constructivism (Vygotsky’s theory) Bieber et al. , NJIT © 2003 12
Cognitive Constructivism (Piaget 1924) • Knowledge is constructed and made meaningful through individual’s interactions and analyses of the environment. --> knowledge is constructed in the mind of individual • Knowledge construction is totally studentcentered. Bieber et al. , NJIT © 2003 13
Learning • Learning is a constructivist, often social activity occurring through knowledge building (Vygotsky, 1978) • Knowledge building activities include contributing to, authoring within, discussing, sharing, exploring, deploying a collective knowledge base (O’Neill & Gomez 1994; Perkins 1993). Bieber et al. , NJIT © 2003 14
Learning • People learn as they navigate to solve problems (Koschmann et al, 1996) and design representations of their understanding (Suthers 1999) • Learning requires cognitive flexibility (Spiro et al. 1991), and results from interaction with people having different experiences and perspectives (Goldman-Segall et al. 1998) Bieber et al. , NJIT © 2003 15
Expert-like Deep Learning • Categorizing knowledge and constructing relationships between concepts are likely to promote expert-like thinking about a domain (Bransford 2000). • To design appropriate problems for their peers, students must organize and synthesize their ideas and learn to recognize the important concepts in the domain. • This results in deep learning (Entwistle 2000): – seeing relationships and patterns among pieces of information, – recognizing the logic behind the organization of material – achieving a sense of understanding Bieber et al. , NJIT © 2003 16
Where is Knowledge Constructed in PLA? • In all PLA stages: constructing problems, solutions, grade justifications, dispute justifications • When reading everything their peers write – Students also are motivated to learn more when peers will read their work (Mc. Connell, 1999). Bieber et al. , NJIT © 2003 17
Assessment & Learning • Main goals of tests: – To measure student achievement – To motivate and direct student learning • The process of taking a test and discussing its grading should be a richly rewarding learning experience (Ebel and Frisbie 1986) • Assessment should be a fundamental part of the learning process (Shepard 2000) Bieber et al. , NJIT © 2003 18
Outline • Motivation • PLA: Participatory Learning Approach • A bit of theory • Experimental results • Interesting issues Bieber et al. , NJIT © 2003 19
Course Information NJIT CIS 677: Information System Principles • • • Graduate level core course (Masters/Ph. D. ) Aim: study how IS/IT can be used effectively Both on-campus and distance-learning sections software: Virtual Classroom/Web. Board Traditional Exam: – Three-hour, in class, 3 -4 essay questions, 6 pages of notes • Used PLA 5 times between Fall 1999 and Summer 2002 • We compared control groups without PLA and treatment groups with PLA • Also, we used with shorter essay questions in CIS 365, undergraduate course on file structures in Fall 2002, with similar survey results. Bieber et al. , NJIT © 2003 20
Enjoyability Cronbach’s Alpha=0. 68 Questions I enjoyed the flexibility in organizing my resources I was motivated to do my best work I enjoyed the examination process SA A N D SD Mean S. D. # 26. 2% 48. 9% 16. 7% 3. 6% 4. 6% 3. 88 1. 00 221 23. 5% 42. 9% 28. 2% 3. 4% 2. 1% 3. 82 . 92 238 17. 2% 42. 3% 22. 6% 10. 5% 7. 4% 3. 51 1. 13 239 SA - strongly agree (5 points); A - agree (4); N - neutral (3); D - disagree (2); SD - strongly disagree (1); the mean is out of 5 points; S. D. - standard deviation Bieber et al. , NJIT © 2003 21
Perceived Learning Cronbach’s Alpha=0. 88 Questions SA A N D SD Mean S. D. # I learned from making up questions 17. 9% 42. 5% 21. 3% 13. 8% 4. 5% 3. 55 1. 08 240 I learned from grading other students answers 17. 7% 48. 1% 19. 4% 9. 3% 5. 5% 3. 63 1. 06 237 I learned from reading other people’s answers 15. 8% 45. 0% 22. 1% 11. 3% 5. 8% 3. 54 1. 07 240 13. 6% 50. 2% 22. 6% 10. 9% 2. 7% 3. 61 . 95 221 21. 8% 49. 2% 25. 6% 2. 1% 1. 3% 3. 88 . 83 238 I learned to value other points of view 17. 6% 51. 9% 27. 6% 1. 3% 1. 6% 3. 82 . 81 239 I mastered the course materials 7. 4% 6. 9% 2. 7% 3. 54 . 84 188 I demonstrated what I learned in class My ability to integrate facts and develop generalizations improved Bieber et al. , NJIT © 2003 51. 6% 31. 4% 22
Recommendation: Do Again! Question Would you recommend in the future that this exam process used? SA A N D SD 20. 7% 40. 1% 24. 5% 8. 9% 5. 8% Mean S. D. # 3. 60 1. 10 237 Similar results for CIS 365: undergraduate file structures course using short essay questions (Fall 2002) Bieber et al. , NJIT © 2003 23
Outline • Motivation • PLA: Participatory Learning Approach • A bit of theory • Experimental results • Interesting issues Bieber et al. , NJIT © 2003 24
What students liked best • Active involvement in the exam process • Flexibility • Reduction in tension Bieber et al. , NJIT © 2003 25
Trade-offs • Trade-offs for students (traditional vs. PLA) – Timing: Concentrated vs. drawn-out (2. 5 weeks) – Access to information: limited vs. the Internet – Experimental integrity: we couldn’t justify the process to the students fully • Trade-offs for professors – Fewer solutions to evaluate, but each is different – Timing: Concentrated vs. drawn-out process – Much more administration Bieber et al. , NJIT © 2003 26
Timing • PLA for exams took 2. 5 weeks • For frequent activities PLA processes could overlap – e. g. , quizzes, homeworks – Students could be creating problems for one quiz, while solving problems for the prior quiz, while evaluating solutions from the quiz before that • Benefits to overlapping PLA activities: – working with materials from several classes at the same time – could reinforce class materials – could result in synthesis (combined understanding) Bieber et al. , NJIT © 2003 27
Scope • Which activities? – so far: exams – what about: quizzes, homeworks, larger projects, in-class projects • Which problem types? – so far: short and long essay questions – what about: multiple choice, short answer, computer programs, semester projects – Sub-problems: • computer program design & implementation • semester project outline & execution Bieber et al. , NJIT © 2003 28
Scope, cont. • Course Level – Graduate, undergraduate, secondary school (high school, junior high) • Disciplines – IS/IT, business, science, engineering, humanities, medical, all of secondary school Bieber et al. , NJIT © 2003 29
Scope, cont. • Degree of Evaluation (assigning grades) – Currently: solutions – What about: • quality of problems • quality of evaluations/grades – All could be disputed • Degree of Participation – students could evaluate each – students could arbitrate disputes Bieber et al. , NJIT © 2003 30
Full Collaboration • Groups for: – Problems, solutions, evaluation, dispute arbitration • Requires group process support – Group roles: leader, scheduler, etc. – Process: work on each activity together or separately, internal review – Grading of individual group members – Process Tools: brainstorming, voting, etc. Bieber et al. , NJIT © 2003 32
What can go wrong • Students are late; students drop the course • Entries posted in wrong place • Inadequate critiques – “Good” – “I agree with the other evaluator” • and of course, technical difficulties… Bieber et al. , NJIT © 2003 33
PLA Environment Software • • Guide the process Form groups Assign problem solvers, evaluators, dispute arbitrators On-line templates to ensure full entries Guide people to post entries in correct place Incorporate group process tools Handle problems as much as possible – Remind people who are late – Reallocate who does what • Based on a workflow management tool… Bieber et al. , NJIT © 2003 34
Anonymity/Privacy Issues • Should student entries be anonymous? • Will students reveal their IDs? • Is it fair to post critiques if not anonymous? • Is it fair to post grades if not anonymous? • Will anonymity work in small classes? Bieber et al. , NJIT © 2003 35
Issue: Perceived Fairness • Should students evaluate/grade peers? – But they must evaluate others in the workplace… • It’s the instructor’s job to evaluate and grade – PLA is a (constructivist) learning technique • Students have no training in evaluation – Evaluation is a skill that must be learnt (and taught) • Many evaluators = inconsistent quality – safeguards in the PLA process Bieber et al. , NJIT © 2003 36
Grading Issues • Disputing high grades: – Award bonus points if students dispute (and justify with a critique) grades that are too high • Encouraging honest grading: – For successful disputes, deduct points from evaluators Bieber et al. , NJIT © 2003 37
Grade Inflation • Detailed grading guidelines for sub-criteria: • • • great: 20 points very good: 18 points good: 14 points OK: 10 points poor: 6 points Student does “good” on 5 problems, grade = 70 U. S. students will protest vigorously Evaluators will hesitate to assign “good” Result: pressure for highly skewed grading rubrics Bieber et al. , NJIT © 2003 38
Other Cross-Cultural Issues • In some cultures: – Students are so competitive, they would only give failing grades to peers – Students would not hurt peers’ feelings, and would only give good evaluations • Some systems only have pass/fail, so numeric grades are mostly irrelevant Bieber et al. , NJIT © 2003 39
PLA: Contributions • Systematic technique to increase learning – Constructivist approach, actively engaging students in the entire problem life-cycle – Minimizes overhead for students and instructors • Experimental evaluation • Supporting software Thank you! Questions, please? • PLA liberates learning from its traditional instructor-controlled structure! Bieber et al. , NJIT © 2003 40
- Slides: 39