Course Assessment From Start to Finish Kathleen M

Course Assessment From Start to Finish Kathleen M. Doyle, Ph. D. Hostos Community College, CUNY

What Comes to Mind When You think of “Course Assessment”? • That’s why I’m here today. We’re going to try to show you that this isn’t that horrible, allay your fears and demystify the process. • Everything will be online in a week or two including all of the forms as well as a recording of this webinar. • So, curl up and relax and get set to do this!

Why Do Course Assessment? • There are multiple reasons for conducting course assessment • Course assessment can be done independently of any other assessments or it may be part of a larger work • It can seem like a very big, unfamiliar maze with no way out and so many different options you don’t know where to start. There is a lot of jargon. It can be very paralyzing. • That’s why I’m here today. I’m going to show you how I got involved with and completed the assessment of one course; Precalculus, from start to finish.

Why We Needed Course Assessment, Internal Factors • Initially, the need for course assessment came up when my department was doing its Annual Program Review (APR), which is an overall look at the work our department does usually once every 10 -15 years. • These reviews contain the courses offered, degrees awarded, facilities, assessment data and the kitchen sink. Ours is online: http: //www. hostos. cuny. edu/Hostos/media/Office-of-the. President/Mathematics-Department-Academic-Review-Report. pdf • Our college had changed a great deal since the previous review. We had more students and more upper level courses than ever before. • Our course content was dictated by that relationship with the four year institution because students had to be able to move between schools so we didn’t even pick the books we used for Calculus. • Precalculus had a high failure rate and there were complaints that some teachers weren’t teaching all of the material in the departmental syllabus because there wasn’t enough time to teach all of the course content.

Why We Needed Course Assessment, External Factors • During the same academic year, our school was going through its accreditation process with Middle States. • The main criticism of our entire college was that we had a lot of course data but we weren’t doing anything significant with it. Middle States wanted to see us using the data, not just collecting it. • There were courses that we had never assessed because we had never offered them on a routine basis. Especially the upper level courses, starting with Precalculus.

The First Steps • Our Provost and Office of Academic Affairs created a committee on assessment. I was the mathematics department’s liaison to that committee. That’s how I became involved with course assessment. • We were asked for a one year follow up report from Middle States. This committee was formed to help keep assessment uniform across the college. • Specifically we were told we needed to use the data that we had to create a “cycle of continuous improvement”, instead of just collecting data every year. • Collecting data that doesn’t get used is not how to do course assessment. It should be a tool to improve student learning and help us be better teachers.

The Challenges • Well, first obstacle would be: Another committee? • My school can be very bureaucratic and changes can take a very long time because they have to be approved at multiple levels of governance. • Another obstacle is getting faculty involved. Expect some resistance at first. • Where are you supposed to start? There is a lot of information about assessment, general education guidelines, program outcomes, course assessment, student learning outcomes. • Fortunately, we were given a head start by our Provost’s office…

The First Step Came from Google! Our Provost’s Office Googled these three steps and they found them to be good, so they were our starting point! They also gave us a Course Assessment Matrix (CAM) 3 STEPS TO CONTINUOUS IMPROVEMENT OF STUDENT LEARNING AT THE COURSE LEVEL • Identifying SLOs 1)Identify the goals for this course. (The course description found in the college catalog or course syllabus can be used to articulate course goals. ) 2)Establish objectives for this course (Objectives refer to the component parts of the goals. For example an objective is the specific topics that you will cover in your course that help you achieve your course goals) 3)Articulate Student Learning Outcomes (SLOs). (Student Learning Outcomes are what you want students to know at the successful completion of the course. ) 4)Align course goals/objectives with outcomes and with Unit/Department goals. • Collect Data & Analyze Data • Specify Assessment Instruments (How do you know students are getting it? ) Include: a. Assessment Criteria b. Assessment schedule i. What Student learning outcome will you collect information on this semester? ii. How do you plan to analyze data and by whom? (Will you analyze data or will OIR staff help with data analysis? ) iii. How do you plan to use the results to make improvements in teaching and learning? iv. Describe how the department will ensure that results will be used to improve teaching and learning. • Use Data • Review findings. • Determine whether goals were met based on findings. (How well have students demonstrated their learning? ) • Implement changes to the course, as appropriate, based on the findings. • Assess the impact of the changes on subsequent learning over an academic year. (i. e. , begin the Outcomes Assessment Cycle again. ) Additional information about each step is provided in the Assessment Tool Box Power Point Presentation. Adapted from Kent State (2004)

The Second Part Came from the University • In addition to working on our departmental review and this new course assessment, our University was busy implementing a set of new criteria for courses so that our students would be able to transfer to four year colleges without having to repeat courses or lose any academic credits. • Pathways implementation wasn’t pretty. The math department actually had a slight benefit from the new requirements but other departments had to protest or lose class time. • This caused a lot of anger on campus and it made some faculty were very wary about course assessment. • The good part is: The University actually gave us the Student Learning Outcomes (SLOs) that they expected us to use in our Pathways Courses.

The Six Pathways Student Learning Outcomes Mathematical and Quantitative Reasoning Courses must meet ALL of the following learning outcomes: 1. Interpret and draw appropriate inferences from quantitative representations, such as formulas, graphs, or tables. 2. Use algebraic, numerical, graphical, or statistical methods to draw accurate conclusions and solve mathematical problems. 3. Represent quantitative problems expressed in natural language in a suitable mathematical format. 4. Effectively communicate quantitative analysis or solutions to mathematical problems in written or oral form. 5. Evaluate solutions to problems for reasonableness using a variety of means, including informed estimation. 6. Apply mathematical methods to problems in other fields of study.

Getting to Work: Who? How? When? • As you can see, the six SLOs are so incredibly vague, they seem to be useless. • This is where our job truly began! We already had the SLOs and the CAM. We had the tools, we just had to use them. • We also already had a Precalculus/Calculus committee of 6 people who routinely taught Precalculus and the rest of the Calculus sequence. • Adapt the SLOs to fit the course you are going to assess. • SLOs are measurable. These are quantitative, not qualitative. Note the language in the CAM, “Students will be able to ____” • If you have to present this to the whole department, you want the major edits done beforehand in the smaller committee.

Uniform Forms • Our assessment needs came from multiple sources: Our department, our college and our accrediting agency. • Fortunately, the Provost’s office had a single form that they requested all departments use the CAM. We needed to be able to compile data across the college to satisfy our accrediting agency. • Two departments had done their program reviews before us and they shared their completed CAMs so we had an idea of what to do. • Now we have reasons to assess, SLOs and a tool to record the assessment. • Surprise! You’re already done with the first part of course assessment! You have SLOs and a CAM to record them, you can use mine!

Precalculus Student Learning Outcomes Generic Pathways SLOs Precalculus SLOs edited from Pathways • Mathematical and Quantitative Reasoning • Courses must meet ALL of the following learning outcomes: 1. Interpret and draw appropriate inferences from quantitative representations, such as formulas, graphs, or tables. 2. Use algebraic, numerical, graphical, or statistical methods to draw accurate conclusions and solve mathematical problems. 3. Represent quantitative problems expressed in natural language in a suitable mathematical format. 4. Effectively communicate quantitative analysis or solutions to mathematical problems in written or oral form. 5. Evaluate solutions to problems for reasonableness using a variety of means, including informed estimation. 6. Apply mathematical methods to problems in other fields of study. • Approved 3/13/2012 by the Mathematics Department with corrections • MAT 160 Student Learning Outcomes: 1. Interpret and draw appropriate inferences about functions and their properties from quantitative representations such as graphs of polynomials, logarithmic, exponential and trigonometric functions. 2. Use algebraic, numerical and graphical methods to solve mathematical problems including representing functions as graphs and their associated composite and inverse functions. 3. Represent quantitative problems expressed in natural language in suitable algebraic, functional and graphical form. 4. Effectively communicate solutions to mathematical problems in written, graphical or equation form. 5. Evaluate solutions to problems and verify the validity of graphs of functions for reasonableness by inspection. 6. Apply mathematical methods to problems in other fields of study such as Physics, Economics and Chemistry.

Developing the Assessment Instrument • Once the SLOs are approved, the committee has to decide on an assessment instrument. • In Mathematics, we’re very fortunate because there is a right answer and a wrong answer. This makes the assessment process a bit easier. • We elected to have a departmental final exam as our assessment tool. • The exam was based on what students needed to know in Calculus and Physics. What were the topics we needed to explain in later courses? That is how we got our questions for the exam.

What Makes the Instrument Good? • Every question corresponds to an SLO, if it doesn’t, that question doesn’t fit in the course. Why is it there if you’re not teaching it? It shouldn’t be there. • Every SLO is on the exam, although you may not assess every SLO during every cycle. If every SLO isn’t there, your exam doesn’t measure what you said it would in the SLOs. • For example: This is one page of our departmental final with SLOs matched to every question.

Moving Forward with SLOs Once the SLOs are approved by the department, we were ready to go. The committee convened and wrote a final exam. We also adjusted the first draft of the SLOs so we can see each SLO is represented on the final exam. • Approved 3/13/2012 by the Mathematics Department with corrections • Developed 2/2012 to correspond to the Pathways draft 1/31/2012 • MAT 160 Student Learning Outcomes: 1. 2. 3. 4. We decided which questions we would use for assessment 5. 6. Interpret and draw appropriate inferences about functions and their properties from quantitative representations such as graphs of polynomials, logarithmic, exponential and trigonometric functions. a. Final exam questions: 1, 3, 6, 9, 13, 14, 16, 17, 18, 20, 21, 23 and 25. b. Assessment question: 3. Use algebraic, numerical and graphical methods to solve mathematical problems including representing functions as graphs and their associated composite and inverse functions. a. Final exam questions: 2, 4, 6, 7, 8, 10, 11, 12, 15, 17, 21, 23 and 25. b. Assessment questions: 2 and 8. Represent quantitative problems expressed in natural language in suitable algebraic, functional and graphical form. a. Final exam questions: 19, 22 and 24 b. Assessment question: 19 Effectively communicate solutions to mathematical problems in written, graphical or equation form. a. Final exam questions: 6, 9, 16 and 23 b. Assessment question: 9 Evaluate solutions to problems and verify the validity of graphs of functions for reasonableness by inspection. a. Final exam questions: 12, 15, 16, 17, 18, 21, 23, 24 and 25. b. Assessment question: 25 Apply mathematical methods to problems in other fields of study such as Physics, Economics and Chemistry. a. Final exam questions: 19, 22 and 24 b. Assessment question: 22

Precalculus Final Exam • This is only page 1, the entire exam is 25 questions long. • Every question has its SLO listed at the end • Every SLO is on the exam • Questions that we chose for assessment are indicated on this copy of the exam • This is the exact same exam students took, but of course, we removed the SLOs

Faculty Buy In and Timing • In my department, if we have a departmental final, it has to be approved by the department in a curriculum meeting. • This can take a lot of time! It can be very frustrating and demoralizing to realize that you are going to take up to a whole AY to get one course done. • The payoff can be great • The SLOs needed approval, the final needed approval and now, you’re probably two months into the semester. • You have to notify faculty at the start of the semester that there is a departmental final. You might want one or two committee members to give the final exam as a trial run. • However, you don’t want to give the final to the whole department right away. Abruptly announcing that there is a required final turns people off.

Giving the Examand Faculty Buy In • This is usually the easiest part. The exam has been approved by the department which is usually the hard part. • Your committee might want to consider some things that help with buy in • Make it easy for them to grade. Asking faculty if they want multiple exams or just one. • Provide answer keys that are written by the committee. Our final exam is sealed and kept for records because of NY State law so the students never get them back, we don’t have to develop a new exam every year. We’re able to get multiple years of data that’s consistent.

Collecting the Data • This part will depend on the size of your school. If your school is large, you might need your Office of Institutional Research to help handle data collection. • The first time I did this, we had 6 sections of Precalculus and 79 students taking the final exam. • The committee chose 7 questions, one for each SLO and 2 for an SLO we suspected was going to be a problem. • As the committee chair, I did the data collection myself. I took the exams and tallied “Correct, Partial Credit, Incorrect and Omitted”. • We failed 5 of the 6 SLOs and only 1 that was acceptable.

Reporting • Fortunately, our Course Assessment Matrix is very compact and it’s simple to report. • A short memo describing our follow up is the end of the first assessment cycle. • It’s never pleasant to think “Oh no, we failed”, but that’s why you have multiple questions for each SLO. You may wish to go back and choose additional questions to clarify where the issues are. • In my case, I had already gone through the exams by hand I knew, there was no point in choosing different questions, the exams were blank. • I also knew that we needed to change the format of the exam, no blue books! It made data collection a nightmare and it was difficult for faculty too.

Following Up on the Assessment • We worked to find better text books. • After another assessment, we adjusted the questions on the final to make some of them a little easier and we removed a few questions that were redundant. • We are still trying to rearrange material in the course with the pre-Precalculus course. That course passed its assessment. • The Math department saved the day with our accrediting agency. Middle states wanted 35 courses assessed for the 1 year follow up report, we contributed the most. • At this point, we don’t need anymore final exam data, we’re looking at entrance exams, which changed from our first assessment to the Accuplacer. • Then the university changed entrance exams plus other external factors, we’re sort of in a lull, but we managed to get a handle on the course. It’s not our students, it’s the course, it’s too much material.

Common Questions • Are there other models? Of course! Be aware of what’s already been done at your school, what’s happening on your campus and within the University too. This is AMATYC so since we’re community colleges, it might even be appropriate to use the same method as a school where your students transfer. • Does this sound totally overwhelming? My case is somewhat extreme. Sometimes a question measures more than one SLO. Be aware of your time table, if necessary. • Does the assessment instrument have to be a final exam? Absolutely not! • Why are we not discussing remedial? The data there is incredibly disjoint because exams change. (This is in AMATYC Educator, Fall 2015. ) • Can you do this backwards? Yes! If you have student artifacts, you can. There was a lot at the beginning that was really sort of random!

Summary – Assessment in Ten Steps or Less: 1. RELAX! It’s really not that bad! You’re already finished with this step, right? See, I told you. You’re already down to 10 steps at most! 2. Form a small committee for each course if necessary, you may group related courses. This process works best in small groups with classroom experience in the course and the courses that come after. 3. Budget your time appropriately and factor in any outside needs 4. Pick your SLOs and customize them to the course. 5. Choose how to record your SLOs 6. Develop a scale for measuring SLOs 7. Decide on an assessment instrument 8. Write the assessment instrument to include all SLOs 9. Collect the data 10. Analyze results 11. USE THE RESULTS TO IMPROVE TEACHING! Adjust what isn’t working, keep what is.

Conclusion and Thanks • Thank you for being here today. I hope this was productive for you and you’re feeling ready to tackle course assessment. • Congratulations! You’ve already done the first step of assessing your course by being here today! • As part of AMATYC’s professional development series, all slides and forms will be made available online. • Please feel free to contact me. Email is the best way to get in touch with me: kdoyle@hostos. cuny. edu • Thank you to the AMATYC team of Julie Gunkelman, Jon Oaks, Behnaz Rouhani and Lisa Feinman for all of their support and help with developing this webinar • Special thanks to Jon Oaks who hosted webinars that I attended which inspired me to write one of my own. His guidance was invaluable during the proposal stage. • Many thanks to Julie, our fearless coordinator. Her kindness and skill are very much appreciated.