ABET Assessing Program Outcomes Amir Rezaei Outline Context

ABET Assessing Program Outcomes Amir Rezaei

Outline • Context of Assessment • Process of Assessment • Similarities and differences between classroom and program assessment • Developing measurable outcomes • Developing scoring rubrics • Assessment methods • Direct measures/Indirect measures • Data collections • Closing Loops • Writing the report

What comes into the system? What are we doing with the inputs? How many? Context of Assessment Input Progress Output What is the effect? Outcomes Assessment of Outputs What have Student grades; What Assessment provides or rates; students learned; of graduation Outputs serve direct Assessment of what skills have employment rates; indirect measurement inputs and process theyas gained; statistics employment measures for of the only establishes the attitudes effectiveness statistics effectiveness capability or developed? capacity of program Publication Student Background Program & & Program services offered; population served Faculty background Faculty teaching size loads/class size Publication numbers/Faculty y development activities; credit activities; hrs delivered hrs Faculty publication citations data; faculty developement Educational Resources Policies, procedures, governance Statistics on resource availability, participation rate Student learning and growth

Process of Assessment Collect data when they graduate • Can we demonstrate that student have learned outcomes xx to an appropriate level by the time of graduation? • Can we demonstrate that we have added value to student learning of outcome xx to an appropriate level by the time of graduation? Pre and post data collection

ABET Terms Objectives ABET uses these terms, Use same language to reduce the Broad statement that describe the career and professional confusion Definition accomplishments that the program is preparing graduates to achieve Some other terms for same concepts Goals, outcomes, purpose, etc Outcomes Statement that describe what students are expected to know and able to do by the time graduation Objectives, standards, etc Performance Criteria Specific, measurable statements identifying the performance(s) required to meet the outcome; evidence Standards, rubrics, specifications, metrics, outcomes, etc. Assessment Process that identify, collect, use and prepare data that can be used to evaluate achievement Evalautions Evaluation Process of reviewing the results of data collection and analysis and making a determination of the value of finding and action to be taken Assesments

Similarities and differences between classroom and program assessment • Degree of complexity • Time span • Accountability for the assessment process • Cost • Level of faculty buy-in • Level of precision of the measure

Classroom Assessment Context Subject matter Faculty member Pedagogy Student Facility Subject Statics Concepts Forces in 2 d & 3 D Topics Statics of particles Equivalent system of forces Eq. Rigid bodies Equilibrium Structures Friction Assessment Focus: Evaluate individual student performance (grades) Evaluate teaching/learning Moment f a force about. . Eq in 2 D & 3 D FBD Trusses, Frames and Machines Friction Timeline 1 quarter

Program Assessment Coursework & Curricular Patterns Classes chosen; major Student Pre-college Traits Classroom Experience Pedagogy; Facility; Faculty & student Characteristics Institutional Context Out-of-class Experiences Co-curricular; Co-ops; internship Timeline xx years Environmental Factor Educational Objective

Developing measurable outcomes Performance Criteria Outcomes Objetive Work Effectively Ability to function with other on multi-disciplinary team Makes contributions Takes responsibility Value other viewpoints Researches and gathers information Fulfill duties of team roles Shares work equally Listen to other teammates

Developing scoring rubrics • A rubric is a set of categories which define and describe the important components of the work being completed, critiqued, or assessed • Purposes − Information to/about individual student competence (Analytic) − Overall examination of the status of the performance of a group of students (Holistic)

Developing scoring rubrics • Generic − Big picture approach − Element of subjectivity • Task-specific − Single task − Focused approach − Less subjective • You don’t have to develop a rubric for every outcomes -Note Rubric template

Assessment methods • Written surveys and questionnaires No rubric • Exit and other interviews Rubric • Commercial, standardized exams Rubric • Locally developed exam No rubric • Archival records No rubric • Focus group No rubric • Portfolios Rubric • Simulations Rubric • Performance Appraisal Rubric • External examiner Rubric • Oral exam Rubric • Behavioral observations Rubric

Direct and Indirect Measures • Direct measures: − Direct measures provide for the direct examination or observations of student knowledge or skills against measurable learning outcomes • Indirect Measures: − Indirect measures are those that ascertain the opinion of self-report of the extent or value of learning experiences

Direct • Exit and other interviews Indirect • Standardized exams • Written surveys and questionnaires • Locally developed exams • Exit and other interviews • Portfolios • Archival records • Simulations • Focus groups • Performance Appraisal • External examiner • Oral exam • Behavioral observation

Sampling And Data collection • For program assessment, sampling is acceptable and desirable for programs of sufficient size Year 1 Define Outcom es/ Map curr. Year 2 Data Collection Year 3 Evaluation & Design of implement ation Year 4 Implement Improvem ents & Data collec.

Closing the Loop Evaluation committee receive and evaluate all data; makes report and refers recommendations to appropriate areas Institute assessment comm. Prepares reports for submission to dept. Heads of the collected data (e. g. survey, eportfolio rating) Institute acts on the recommendations of the Eval. Comm. Report of actions taken by the Institute and the targeted areas are returned to the Eval Comm. For iterative evaluation