Calibrated Peer Review TM A Writing and Critical

Calibrated Peer Review. TM A Writing and Critical Thinking Instructional Tool http: //cpr. molsci. ucla. edu Arlene A. Russell, UCLA NLII Focus Session June 15 -16, 2005

http: //cpr. molsci. ucla. edu Calibrated Peer Review (CPR) • was invented at UCLA during the NSF-funded Molecular Science systemic reform project • is an instructional tool that uses writing and peer review to teach higher-order thinking skills and peer collaboration • manages the writing and anonymous review processes of faculty-developed writing assignments • serves any discipline

Tenets of CPR • Expository writing promotes understanding • Clear writing demonstrates clear thinking • Peer review and evaluation require higherorder, critical thinking skills • Writing-to-learn ¹ Learning-to-write – But, writing-to-learn can (and should) include attention to writing skills

Writing-to-Learn • Used in humanities and social sciences • Requires small classes – Science and engineering have put their resources into labs • Imposes intensive grading workload ~~ • Seldom used in large classes, because – Workload too high, teaching writing not the job of scientists, etc. . , not objective, …….

Peer Review • Accepted process for validating academic research • Essential skill required of all professionals ~~ • Seldom taught in science classes • Not generally used in any large courses

Peer Review • “Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks” N. Falchikov and J. Goldfinch, Rev. of Ed. Research, Fall 2000, 70, pp 287 -322 – Peer assessments resembled teacher assessments most closely “when global judgements based on well understood criteria were used. ” – Advanced students not better assessors than beginner students.

Student’s Interaction with CPR (see flowchart handout) Students – carry out a guided study of source material – write a document about what they have studied – are trained as reviewers of this topic by studying a set of “Calibration” documents and evaluating them on specific questions – anonymously review three of their peers’ submissions and provide written feedback on them – apply their evaluation skills to self-review their own document

Instructor’s Interaction with CPR Instructors – (upload the class roster) – select CPR assignments congruent with course goals – set grading criteria and due dates for submission of essays and reviews – monitor student progress and performance!! – resolve disputed reviews (usually fewer than typical exam regrade requests)

What is the impact of CPR? • Has CPR had an impact on teaching? • Does using CPR in a course impact learning?

Central to the CPR Process • Faculty-developed assignments – Enormous national creative effort has occurred • ~1800 assignments have been developed; over 600 in the past year. – Estimating 10 hours/assignment for development = 11 faculty years of effort

So What? Do CPR Assignments Make a Difference? • • Is content knowledge improved? Are evaluation skills developed? Do students learn to collaborate? What do students think about CPR?

So What? • Is content knowledge improved? – Yes, ~10% gains across all levels on both essay and multiple choice exams • Are evaluation skills developed? • Yes • Do students learn to collaborate? – Yes, but they need to understand the process and rationale for peer assessment • What do students think about CPR? – On the whole favorably, if they are supported while they learn to deal with changed instructional methods

Peer Review Feedback

Workload, Value, Reviewing • I thought the idea of CPR was a very good idea because it helped expanding students' knowledge and developing insight into the materials we had studied. However, I didn't really enjoy this project. I didn't like it because there was too much research involved and questions were very hard to answer. This project itself was time consuming as well. It took me a long time to finish the practice before I moved onto the actual project. Another time-consuming part was the peer reviewing. It took me about an average of 10 minutes each to review one paper. It required a lot of concentration as well

Course Value • I thought CPR to be good learning tool. The project made the participants review the course material. As a result a better understanding of the material is attained…. It’s like having two lectures instead of one. Making an environment where students have to think is a good way to make the material understandable.

Student Perceptions

Student Comments Technical Issues • “The chime plugin was a little problem because I too use netscape 6. 1 so I ended up using Internet Explorer 6. 0. It handled the plugin well and did not give me any problems. The web site performed decently with a 56 K modem connection, except the chime illustrations took about 30 sec to 45 sec to load but acceptable. Now that the calibrations are available the calibration screens seem very user friendly but I would tell everyone that it is best to use a screen resolution of at least 800 x 600 because there is so much to info to display. The questions take half the screen and the essay takes up the other half. ”

Timing, Workload • this project was sort of like high school work. it was only assigned to keep me busy. first of all i don't have a computer so i had to go out of my way to use one. we studied cyclohexanes for the first midterm. it would've been a lot better if it was material from the current chapters.

Source Material/Reviewing • “I thought that the CPR was really interesting. I liked the fact that they made us look at 3 D models before we could answer the questions because it gave us a good idea of the concept involved in the question. I think that the process we had to take before writing the essay was great because it took us step by step to the answer, allowing us to thoroughly comprehend the strain (angle and torsional) in cycloalkanes in comparison to their alkane counterparts and to other larger or smaller cycloalkanes…. I also like that we will get to grade other students’ essays and later see if our own essays were of good quality. I believe that it will let us know if we really understand the part that strain plays in cycloalkanes. ”

Reviewing, Course Value • Truthfully, I enjoyed CPR. Writing the essay was a great way to review the specifics of cyclohxane strain, but when I reviewed the peer essays I think I learned most. I was able to see what strong points my essay had, and more importantly where I was lacking. I think CPR can be a very useful tool for students to use if they take it seriously and take the time to explore the website. Also, it is another chance for points, which is always good. By the way, thanks for the extra credit.

Writing • By writing this assignment in a mini-essay form I was able to come at my own conclusions and express the material in my words. I believe that this was very helpful because in order to understand the material it is extremely useful to be able to describe and put the material one learned into his/her own words

Workload, Course Value • I enjoyed the CPR assignment. I liked having another format to think about organic chemistry. If you do this again, I think optical rotation would be a great subject. I thought the writing assignment was a good length--not too much to read six examples of writing

Technical Difficulties and Attitude to CPR Positive Comment Mentioned problems with CHIME (&/or UCR and CPR login) No technical difficulties mentioned Negative Comment 59% 41% 82% 18%

Questions? cpr@nslc. ucla. edu