CONTINUOUS IMPROVEMENT LONG AND SHORTTERM PLANNING FOR STUDENTLEARNING

CONTINUOUS IMPROVEMENT: LONG AND SHORT-TERM PLANNING FOR STUDENT-LEARNING ASSESSMENT SUMMER DEPROW, PHD MAY 13, 2021 VIA ZOOM AND AUGUST 16, 2021 HYBRID

WORKSHOP LOGISTICS • Welcome and thank you for inviting me ! • You will be muted upon entry to the workshop, so please introduce yourself via the Chat feature by clicking “Chat" • The window to the right will appear where you can add your introduction: • If you have questions during the session, please add them to the Chat box. I have attempted to add several opportunities to have Q & A. S. DEPROW, PHD 12/24/2021

LEARNING OUTCOMES - AGENDA Goal of Workshop: To establish long and short-term plans for continuous improvement processes that achieves implementation fidelity while adhering to A-State’s Assessment Manual processes and procedures Construct a long-term continuous improvement plan (i. e. , 4 -year plan) Develop a short-term continuous improvement plan (i. e. , 1 semester or academic year) Discuss pitfalls of implementation fidelity S. DEPROW, PHD 12/24/2021

THE BASICS: MODEL OF STUDENT-LEARNING ASSESSMENT Establish PLOs Adapted from JMU S. DEPROW, PHD Use Information for Program Related Decisions Create Curriculum Map Analyze, Report and Document Information Select/Design Measures: Direct & Indirect Collect Outcome Information 12/24/2021

THE BASICS: MODEL OF STUDENT-LEARNING ASSESSMENT Establish PLOs Adapted from JMU S. DEPROW, PHD Use Information for Program Related Decisions Create Curriculum Map Collect Outcome Information Select/Design Measures: Direct & Indirect Implementatio n Fidelity Analyze, Report and Document Information 12/24/2021

WORK TO DATE: LEARNING OUTCOMES Program Learning Outcomes (PLOs) are written Each program (program defined by ADHE and HLC) Tip: Look for ways to collect data that speaks to multiple learning outcomes Additional resources for writing learning outcomes: A-State Workshops University of Hawaii at Manoa S. DEPROW, PHD 12/24/2021

WORK TO DATE: CURRICULUM MAPS Program courses are mapped to PLOs All faculty should review Additional Resources: A-State Workshops U of HA S. DEPROW, PHD 12/24/2021

LONG-TERM ASSESSMENT PLANNING Recommended duration: 3 -5 years Check institutional process A-State Assessment Manual: Assess one PLO every year and all PLOs in four years Assess Gen Ed LO every year Check programmatic accreditation assessment cycles (and plans for new learning standards/outcomes) Have curriculum map in hand Have course rotation schedule in hand Additional Resources: U of HA Program Assessment Plan U of NC at Chapel Hill S. DEPROW, PHD 12/24/2021

LONG-TERM: 5 -COLUMN ASSESSMENT PLAN Expected Program. Level Learning Outcomes Which courses support this outcome? Assessment Measures Make sure to have two types (best practice include a direct and indirect measure) of assessment measures per each outcome Assessment Timetable What academic year will you collect data for this learning outcome? Who is responsible for assessing and reporting on the results? Adapted from Nichols (1995) S. DEPROW, PHD 12/24/2021

LONG-TERM: 5 -COLUMN ASSESSMENT PLAN Expected Program. Level Learning Outcomes Critically assess and remediate issues with data organization and data quality S. DEPROW, PHD Which courses support this outcome? DATA 2004 DATA 3003 DATA 4013 1. 2. Assessment Measures Assessment Timetable Make sure to have two types (best practice include a direct and indirect measure) of assessment measures per each outcome What academic year will you collect data for this learning outcome? Direct measure: A hierarchical final project review panel consisting of peer, stakeholder, and faculty reviews in a rubric format. Indirect measure: Students will complete an exit survey. Assessment data is gathered at the end of each semester in which the DATA 4013 capstone course is offered; review occurs on an annual basis. Who is responsible for assessing and reporting on the results? Instructor of record for DATA 4013 directs final project review, collects rubrics from the panel, and collates the results. Program director reviews results and reports to the Program Steering Committee. Program director may recommend changes, to be approved by the Steering Committee, or changes may be recommended by the Steering Committee directly. 12/24/2021

QUESTIONS? Ask me anything about long-term planning for continuous improvement S. DEPROW, PHD 12/24/2021

SHORT-TERM PLANNING: 1 -YEAR What is this for? To increase implementation fidelity (definition: accuracy in details) To prepare for an academic year When should the planning begin? Depends on the amount of work to be done but it is recommended the spring semester in the prior year Who should be involved? Faculty who teach courses that support the learning outcome (see Curriculum Map) Could be scaffolded with the faculty who are teaching the courses where the data is captured first; then present to all learning outcome faculty S. DEPROW, PHD 12/24/2021

LONG-TERM: 5 -COLUMN ASSESSMENT PLAN Expected Program. Level Learning Outcomes Critically assess and remediate issues with data organization and data quality Which courses support this outcome? DATA 2004 DATA 3003 DATA 4013 1. 2. Assessment Measures Assessment Timetable Make sure to have two types (best practice include a direct and indirect measure) of assessment measures per each outcome What academic year will you collect data for this learning outcome? Direct measure: A hierarchical final project review panel consisting of peer, stakeholder, and faculty reviews in a rubric format. Indirect measure: Students will complete an exit survey. Assessment data is gathered at the end of each semester in which the DATA 4013 capstone course is offered; review occurs on an annual basis. Who is responsible for assessing and reporting on the results? Instructor of record for DATA 4013 directs final project review, collects rubrics from the panel, and collates the results. Program director reviews results and reports to the Program Steering Committee. Program director may recommend changes, to be approved by the Steering Committee, or changes may be recommended by the Steering Committee directly. (Computer Science Department, 2020) S. DEPROW, PHD 12/24/2021

SHORT-TERM PLANNING TEMPLATE Moving to the Word Doc Supporting links: NILOA Article: The Need for Program Theory and Implementation Fidelity in Assessment Practice and Standards Developing assessment measures from James Madison University Research and Practice in Assessment Article: Elevating Program Theory and Implementation Fidelity in Higher Education: Modeling the Process via and Ethical Reasoning Curriculum U of HA Planning and Implementing an Assessment Project S. DEPROW, PHD 12/24/2021

QUESTIONS? Ask me anything about short-term planning for continuous improvement S. DEPROW, PHD 12/24/2021

IMPLEMENTATION FIDELITY OF THE PLANS Pitfalls: Operationalizing the learning or setting KPIs Selecting assessment measures that connect with outcome Selecting or building instruments with enough time to embedded into courses Communication with all faculty: new, part-time, off-campus locations Utilizing technology to capture data and training faculty Running out of time for a pilot JMU Website Implementation Fidelity WEAVE Webinar on Implementation Fidelity Gerstner & Finney (2013) Article S. DEPROW, PHD 12/24/2021

HOMEWORK? Develop long-term plan (probably already available in Taskstream) Develop 2021 -22 short-term plan Late Fall 2021 or early Spring 2022 develop 2022 -23 short-term plan S. DEPROW, PHD 12/24/2021

THANK YOU Please email questions or comments to Summer De. Prow, AVC for Assessment and Accreditation sdeprow@astate. edu Ext. 4311 S. DEPROW, PHD 12/24/2021

REFERENCES AACSB. 2020 Guiding Principles and Standards for Business Accreditation Nichols, O. J. (1995). A practitioner's handbook for institutional effectiveness and student outcomes assessment implementation. Agathon Press. Other references are embedded throughout slide deck. S. DEPROW, PHD 12/24/2021