Creating a Comprehensive Bank of Assessments for Use

Creating a Comprehensive Bank of Assessments for Use in Applied Statistical Methods (Stat 1000) BRIAN LEVENTHAL, PSYCHOLOGY IN EDUCATION PHD EXPECTED 2017 DR. NANCY PFENNING, SENIOR LECTURER, DEPARTMENT OF STATISTICS

Introduction v. Propose a transformation of assessment materials in Stat 1000 in order to consolidate and enhance existing materials • Goal: ensure fair and appropriate content for all instructors – focus on new hires v. Begin: examination of questions of existing assessment materials developed by various faculty to date. • Use assignments, projects that require statistical software, quizzes, midterm and final examinations v. Goal: organize a coherent collection of materials that make it possible for a less experienced lecturer to “hit the ground running” with high-quality assessment tools that are consistent with Stat department’s goals for instruction in that course BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 2

Background v. Introductory statistics classes service a large number of students from various backgrounds and different fields. • Stat 200, Stat 1000, Stat 1100 v. Increased reliance on adjunct instructors • Spring 2016: only 3 out of 15 lectures taught by full-time statistics faculty v. Stat 200 – Dr. Nancy Pfenning has successfully transformed and consolidated assessment materials through use of her text and accompanying slides, etc. v. Next Step: Stat 1000 • Fall – 5 lecture sections offered; Spring – 4 lecture sections; Summer – 2 lecture sections v. Future: Stat 1100 BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 3

Teacher Evaluations – Spring 2016 v. Adjunct 1: OMET • OMET: “The instructor evaluated my work fairly”: mean 2. 62 out of 5 • Overall teaching effectiveness: Mean 1. 90 v. Adjunct 2: Mid-semester survey • “Because the lecturer is new to the university it seems as though the university has not prepared him enough…” BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 4

Teacher Evaluations Spring 2016 cont’d v. Adjunct 3: Mid-semester survey • It seems like he was tossed to the wolves at the last minute and given some class materials and told to go run this class… We have had sporadic quizzes(2 so far) and they have been on material that is several lectures back. We could be on lecture 12 in lecture, talking about lecture 7 in recitation and doing homework on lectures 4 and 5 and a quiz on lectures 7 -10. That seems how this class has been going. It was like that one week for recitation/homework/quiz. Im honestly thinking about sitting down with someone in the STAT department to talk about this class. He's well prepared for class in a sense of knowing the material and teaching it. He is not well prepared for class in regards to the other areas like quizzes, homework, recitation, stat lab, etc v. Adjunct 4: • OMET: “The instructor evaluated my work fairly”: mean 4. 73 out of 5 • Overall teaching effectiveness: Mean 4. 18 BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 5

Step #1 v. Survey of adjunct lecturers about usefulness of materials provided • Provided to three adjunct Spring 2016 Stat 1000 lecturers as well as two adjunct Stat 200 lecturers for comparison • Assess: • Usefulness of materials provided • How easy or difficult it was to adapt various materials • Will be administered at end of Fall 2016 and End of Spring 2017 as well BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 6

Step #2 v. Material has been gathered from experienced lecturers: Dr. Carl Bodenschatz and Dr. Kehui Chen. • • • Midterm examinations Final examinations Quizzes Homework assignments Exam practice material Lab assessments v. Course syllabi: frequency of quizzes, examinations, lab assessments, homework assignments v. Student grades will be collected in order to make a comparison of final grade distributions for students from various lectures of STAT 1000. BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 7

Step #3 v. Ongoing Discussion with experience lecturers: Dr. Carl Bodenschatz and Dr. Kehui Chen v. Discussions will include gaining recommendations for • • • Homework assignments Laboratory assessments Number of examinations Other useful ideas relevant to incoming adjuncts or new full-time faculty What material is considered most important for assessment in final exams BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 8

Step #4 v. Handing off some prepared materials to Fall 2016 Stat 1000 adjuncts • Dr. Carl Bodenschatz will be teaching two lectures • Brian Leventhal will most likely be teaching one lecture • A new adjunct will probably be teaching the final section of STAT 1000. BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 9

Step #5 v. Ongoing review of Fall 2016 Stat 1000 materials from Dr. Carl Bodenschatz v. Engaging in thorough discussions about material on current assignments and assessments to see • Which items work best • What item styles (eg. multiple choice, short-answer, etc. ) are suggested • Other aspects Dr. Bodenschatz thinks are essential for organization of the course BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 10

Step #6 v. Creation of annotated test bank for final exams • Will contain a variety of constructed response and selected response items • Items will range across the taxonomy of educational objectives • Consist of statistical basics, choosing the correct analysis procedure, and interpreting output from statistical software • Items will be classified by topic and type v. After classification of items, topics will be assigned a recommended percentage of final exam coverage, so major topics are given roughly the same weight on various instructors’ final exams BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 11

Step #7 v. Handing off more comprehensive packet of materials to Spring 2017 Stat 1000 adjuncts v. New instructors expected to teach lectures of Stat 1000 will be provided with a list of • • • Final examination assessment items Sample syllabi Sample homework assignments Sample lab assignments Sample midterm examinations Course organization recommendations. BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 12

Overall Progress v. Throughout the pilot study we intend to monitor how much, and, which type of material the instructors use. v. We intend to discuss this with the instructors to understand their thought process so that we can improve and cater to the needs of new instructors v. OMET course evaluation and student grade distributions will be compared • Help inform us which material is most beneficial to the adjuncts. v. This information will be used to make recommendations in our user guide we will provide to the adjuncts. BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 13

Objectives and Assessments #1 v. Easier and more straightforward preparation for new lecturers of Stat 1000 • Assessed through survey (Step #1) v. Example items will include: • The material provided made the development of my STAT 1000 course easier. • (strongly disagree, neutral, agree, strongly agree) • The percentage recommendations for material on the final exam influenced the time I spent on material in lecture. • (strongly disagree, neutral, agree, strongly agree) • The material provided included a wide enough range of items to select from. • (strongly disagree, neutral, agree, strongly agree) • Working with the provided material, I was still able to develop a course that suited my teaching style. • (strongly disagree, neutral, agree, strongly agree) BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 14

Objectives and Assessments #2 v. Better quality of materials used in Stat 1000, especially by less-experienced adjuncts v. This will mainly be assessed through mid-semester lecturer survey and end-of-semester OMETs. v. Example adjunct: • Scored 2. 05 out of 5 on presenting course in organized manner • Scored 2. 62 out of 5 on evaluating students work fairly v. One of our objectives is to ensure that such low levels do not occur again in future Stat 1000 lectures. BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 15

Objectives and Assessments #3 v. More uniform instruction in all Stat 1000 lectures v. Checking for less disparity compared with pre-transformation scores v. Scores on final exams will also be analyzed BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 16

Challenges and Responding Plans v. We recognize that individual instructors have their particular strengths and weaknesses. v Even if the prepared materials are considered optimal, a lecturer with poor organizational skills might not be able to incorporate them effectively. v. Especially during the first two semesters of this project, efforts will be made to develop a user’s guide that anticipates and mitigates as many teaching pitfalls as possible. BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 17

Sustainability and Scalability v. All the materials developed as part of this transformation will be catalogued and stored (using a Box folder) in such a way that they will be as useful and accessible as possible in future semesters. v Access to the Box will be limited to Stat 1000 lecturers. This transformation can also serve as a model to streamline a similar endeavor in the future for our Stat 1100 course. BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 18

Conclusion v. Students and instructors alike will benefit from the implementation of this project made possible by d. B-SERC. We look forward to reporting progress during the coming fall and spring semesters. v. Questions? Tips on Project? Please contact: v. Brian Leventhal – brl 38@pitt. edu v. Dr. Nancy Pfenning – nancyp@pitt. edu BRIAN LEVENTHAL (BRL 38@PITT. EDU); NANCY PFENNING (NANCYP@PITT. EDU) 19