Closing the Loop on Assessment Improvements that Transform

Closing the Loop on Assessment – Improvements that Transform Student Learning Dr. Evonne Carter, Vice President of Learning Dean Roughton, Dean of Arts and Sciences College of The Albemarle Fall, 2015

Overview of Presentation Sample Student Learning Outcome (SLO) documents – Programs Samples from General Education Analyses of Improvements Process Student learning Instructional Strategies “Improvements that “Transform”

Assessment- Student Learning Outcomes Program-Level Assessment Process for SLO- Outcomes Tools to assess Criteria Results Improvements N=16 over three years

Program Assessment of Student learning Outcomes – Sample from HVAC Learning Outcomes Complete layout drawings for HVAC systems using approved engineering methods Assessment and Criteria 2014 -2015 Results Met / Not Met Lab project- a house drawing 80% of the students Met to develop a duct layout using completed the project approved engineering 20% needed help with methods with 80% accuracy, equipment sizing based on criteria listed Demonstrate competency on Take EPA 608 CFC Test to be given April In progress the concepts and theory for the certification test 90% of 2015 Federal 608 CFC certification students to pass first time. test. Demonstrate knowledge about 90% of students will pass a 90 % of students took the Met refrigeration system operations written and oral lab test with test and passed with 85 85% or higher accuracy on or higher both tests. Troubleshoot and repair 90% of students will 80% of students were able Not met electrical and digital control demonstrate all skills and to pass the oral and systems competencies related to given written parts of the test problems with 85% accuracy. with a score of 90 or better Improvements Use lab time for equipment sizing and add a module to my. Courses with wholesaler info project Assign study groups to practice for the CFC test. Also, rotate the study groups through the semester Introduce commercial chiller operation for commercial refrigeration system operation Develop and use rubrics for the lab assignments: one portion of the rubric will focus on ability to troubleshoot and repair electrical systems

Analysis of Program Improvements – Adjust the Assessment Process Revise test Weekly sheets Practice Tests Rubric Criteria

Analysis of Program Improvements. Adjust student learning strategies Alt delivery Analysies Equipment Individual help Skill remediation Soft skills Software Synthesis Troubleshoot Modules Wriing/ Journal Application lessons 0 1 2 3 4 5 6 7

si te xt Te Te de Vi os y m ar s am ps r SI ao ou gr m su y s ol ol s to ul Si m e lin ud St On tim e s ci se ip tr To La b Ne w e or er ex d el Fi io n at lic pp iti on ri a n tio ite cr ad d b La ta n b La tio uc tr en se n Pr La b M Op in s ta ep nc Co ic al Cl in ed us Fo c Analysis of Program Improvements. Adjust Instructional Delivery 3 2 1 0

Assessment of General Education Student Learning Outcomes Establish the SLOs/competencies Written Communication Oral Communication Information Literacy Quantitative Skills Computer Literacy Student level assessment Pre and post-tests Rubrics Surveys Course/discipline level indicators Course success rates Discipline specific GPA at transfer institutions

General Education Assessment Example – Written Communication Outcome defined: Students will write effective documents that are unified, coherent, well developed, and which adhere to standard grammar and mechanics.

Written Communication Example Assessment Tool - Rubric Competency No/Limited Proficiency 0/1 Thesis is missing OR thesis has no relation to the writing task; paragraphs frequently lose focus. Some Proficiency 2 Thesis and purpose are somewhat vague OR only loosely related to the writing task; thesis may be unimaginative; paragraphs occasionally lose focus. Proficiency 3 Thesis and purpose are fairly clear and match the writing task; thesis is somewhat original; paragraphs are focused. High Proficiency 4 Thesis and purpose are clear to the reader; closely matching the writing task; develops fresh insight that challenges the reader’s thinking; paragraphs are tightly focused. Coherence Unclear organization OR organizational plan is inappropriate to thesis; lacks transitions. Some signs of logical organization; may have abrupt or illogical shifts & ineffective flow of ideas; limited transitions. Organization supports thesis and purpose; transitions are mostly appropriate; sequence of ideas could be improved. Fully & imaginatively supports thesis & purpose; sequence of ideas is effective; transitions are effective. Development Offers simplistic, undeveloped, or cryptic support for the ideas. Inappropriate or off-topic generalizations, faulty assumptions, errors of fact. Offers somewhat obvious support that may be too broad. Details are too general, not interpreted, irrelevant to thesis, or inappropriately repetitive. Offers solid but less original reasoning. Assumptions are not always recognized or made explicit. Contains some appropriate details or examples. Substantial, logical, & concrete development of ideas. Assumptions are made explicit. Details are germane, original, and convincingly interpreted. Grammar & Mechanics Writing contains numerous errors in spelling, grammar, and/or sentence structure which interfere with comprehension. Frequent errors in spelling, grammar (such as subject/verb agreements and tense), sentence structure and/or other writing conventions distract the reader. While there may be minor errors, the paper follows normal conventions of spelling and grammar throughout and has been carefully proofread. The writing is essentially error -free in terms of mechanics. Total score Overall rating = Unity (Rating) ÷ 4

Written Communication – Closing the Loop/Improvements 2009 -10: Implemented ENG 111 A lab course. 2010 -11: Restructured the Writing Center. 2011 -12: Some ENG 111 instructors required Supplemental Instruction via online tutorial service. 2012 -13: Adopted new ENG 111 textbook. 2013 -14: Adjusted rubric outcomes threshold measures. 2014 -15: Removed ENG 111 A lab course and Implemented redesigned ENG 111 course (state driven). Piloted new online tutorial service. 2015 -16: All ENG 111 instructors required to incorporate Supplemental Instruction. Returned to old textbook in new edition.

Improvements – Written Communication 2009 2010 2011 2012 Instruction New course Textbook Student Learning Writing Center Supplemental Instruction SI 2013 2014 2015 New course Online tutorial SI Textbook Assessment process Adjusted rubric/ outcomes

Assessment processes Final Program assessment of Student Learning Valid Reliable One event or culmination Level of Learning is often Application, Analysis, Synthesis or Evaluation Application – Use or apply information in relative situation Analysis- Examine components, compare/contrast, deduce implications Synthesis- Create, connect, make new relationships Evaluation – Make judgments, assess validity, defend

Validity in assessment The degree to which a score is meaningful and appropriate for its intended purpose Tests do not “possess” validity – validity is derived from the analysis of the results The information obtained from a test helps make databased “inferences” and judgments (that is, valid decisions)

Valid decisions can be made if. . Items and tasks are clear Content of assessment matches Course competencies Material is appropriately weighted The level of processing/thinking matches the course expectations The range of test questions is adequate The information derived can be used to improve learning

Reliability Really means “consistency” Each learner is assessed the same way There must be consistency among the “raters” If two teachers give the assessment to the same learner, the scores should be the same

Types of Program Assessments Licensure Exams (written) Essays Research papers Portfolio Clinical skills (practical) Lab projects Shop projects Case Study Internship

Adjustments to Improve Student Learning and Instruction– Application of knowledge Use or apply information in relative situations Handout: Adjustments to Improve Instruction with Higher Order Processes

Adjustments to Improve Student Learning and Instruction– Analysis of knowledge Analysis. Examine components, compare/contrast, deduce implications

Adjustments to Improve Student Learning and Instruction– Synthesis of knowledge Synthesis- Create, connect, make new relationships

Adjustments to Improve Student Learning and Instruction– Evaluation of knowledge Evaluation – Make judgments, assess validity, defend position

Adjustments to Instruction and Delivery Professional Development based on survey results and selfevaluation Handout: COA Faculty Self Evaluation Rubric: Exemplary Indicators

Adjustments to Instruction and Delivery Curriculum evaluation – Outcomes/ sequence Instructional materials Strategies for Active learning (adults often learn by constructing knowledge and building schema, and not as effectively by rote learning) Interactive Lectures Team based projects Problem based learning Simulations Clinical/ Internships