USING PERFORMANCE REVIEWS IN CAPSTONE DESIGN COURSES FOR

USING PERFORMANCE REVIEWS IN CAPSTONE DESIGN COURSES FOR DEVELOPMENT AND ASSESSMENT OF PROFESSIONAL SKILLS (Paper 1041, Session 3525) Greg Kremer Associate Professor and Chair, Mechanical Engineering (kremer@ohio. edu) David Burnette MSME, 2007 Ohio University, Athens, OH Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 1

OR -----------------Expanding ME program outcomes from "How-2 -Do" engineering to "How-2 -Be" a good engineer Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 2

HOLD PARAMOUNT… Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 3

ST 1 DO NO HARM HOLD PARAMOUNT… Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 4

T ENGINEERS MAKE A X E T N WORLD OF O C L DIFFERENCE A I C O S WILL IT BE FOR GOOD OR BAD? Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 5

CASTL 2005 -2006 Integrative Learning Project Overall Goal My true passion is to craft a capstone design experience that is truly an “education of the whole person” in order to encourage engineering students to appreciate the importance of integrity and wholeness in life. I want our graduates to see the profession of engineering as an opportunity to serve the world and support social justice, and I want them as individuals and as groups to “care as well as know”. My integrative learning project deals with integrating the non -technical but critically important "professional skills" into student conceptions of what it means to be a good engineer. Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 6

CASTL 2005 -2006 Integrative Learning Project Initial research tasks 1. Develop a taxonomy of professional skills outcomes for engineers that are understandable to undergraduate engineering students and that are linked to the ABET professional skills outcomes. 2. Develop an authentic way to get undergraduate engineering students to value and develop professional skills. 3. Begin initial assessments of the effectiveness of purposeful identity development activities (integrated throughout the Mechanical Engineering program and organized around a team-based capstone design project) on student achievement of an integral engineering identity. Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 7

OU ME Program Educational Objectives Technical Skills L A I R T S RY U IND VISO AD ARD BO Skills to perform in the work environment (Professional Skills) Informed & aware of contemporary issues and the impact of engineering on society Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 8

Skills to perform in the work environment (Professional Skills) n e t t i wr g n h i d t t l n ge ea al a d h u d b n nd n: Or , a a g in ent rds, ty atio l e u f Sa unic work ched ovem anda et k t s r r m m s , p ity a m Tea ing m i m o l ( c s nn g to qua s al us a l c e o i n , p i n u n i s s ntin t: lead ard. ). ch u n. s l e b e t , i a d o , T t s h m n c n t n e ue men / io l sta of u n wi lity, g l t a an valu iona ems atio qua re v viron nce m r o en t , t e nat a c g s c n f / l e y e o m t j i r r e s t in fac sion ving rfo S inte g Pro n f i is vi nvol y, pe t r o / a e ss ne er s sion ng, i ualit. i e g n n tom mis aki q / life re e , a f e s , o m tim ility u ofit c Aw n n , o r , o i s t, p i t s s s ab a i i c i l i e c c e n n pre ware eme ive de onom and r p c A v ect a , e o ABET y r , t i f l p y i f t im nd e safe ab r e PROFESSIONAL a nd p l o a ic lth a h t SKILLS E hea Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 9

OU ME CAPSTONE DESIGN Student behavior (responding to what the instructor asks) Professional behavior (responding to open-ended problems, dealing with uncertainty, making “good” decisions based on informed engineering judgment) Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 10

Professional Skills – Prior Work Lewis and Bonollo (2002) Identified 5 professional skills that professional design clinic supervisors valued highly in students: 1. Negotiation with clients 2. Problem solving 3. Acceptance of responsibility for outcomes (professional behavior), 4. Interpersonal skills 5. Project management Shuman et al. (2005) Claim that “portfolios, along with performance appraisals and behavioral observations, offer the most comprehensive information for measuring many outcomes and are conducive to evaluating professional skills. ” Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 11

Professional skills must be presented and developed in an authentic way DECISION: ACTIVITY WITHIN CAPSTONE PROJECT -------- VS Add-on requirement ACTIVITY CONSTRUCTED FOR vs ASSESSMENT Integral to engineering identity Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 12

OU ME CAPSTONE DESIGN ~ENERGY EFFICIENT CAMPUS TRANSPORTATION, NISH ~50 STUDENTS/YEAR Large (6 -8) and diverse teams Develop a production plan Need statement based on contemporary issues Identify a customer & market Develop target specs based on customer needs Year-long Capstone Design Project Create and maintain a project schedule Build and test a prototype Select a concept and refine the design Develop feasible concepts Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 13

OU ME CAPSTONE DESIGN Large (6 -8) and diverse teams. Need statement based on contemporary issues l a n o i s s e ills f o k r P S Identify a customer & market Develop target specs based on customer needs Year-long Capstone Design Project Develop a production plan. Select a concept Build and test and refine the a prototype design Create and maintain a project schedule Develop feasible concepts Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 14

Learning Needs Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 15

OU ME CAPSTONE DESIGN Create list of Professional Skills statement Important Need to their team with explanations based on Large (6 -8) and Identify a and examples and use it contemporary diverse teams customer & market issues for peer ratings Performance Reviews with self and peer rating of professional skills Develop a production plan Build and test a prototype Develop target specs based on face-to-face customer needs Participate in performance reviews with Year-long Capstone Design Project “supervisor” to review accomplishments and Create set project and maintain a project schedule personal development goals Select a concept and refine the design Develop feasible concepts Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 16

DECISION: ASSESSMENT RUBRIC DECISION: STUDENT DEVELOPED DECISION: VS DEVELOPMENT LEVEL VS TAXONOMY VS FACULTY DEVELOPED Levels of Development # (correlated with Bloom's affective levels ^), and what the characteristics look like in practice for the various levels No Evidence (Unrated) Attitude and character Ignorance (Receiving) 0 Awareness (Responding) 1 Doesn’t want to do Sometimes is any work & doesn’t optimistic about doing care about others; work but would usually always looks at the negative side of things. prefer to avoid it. Motivated by external factors ($, boss) and personal gain. Usually takes on responsibilities only when asked to. Only acts when prodded. Expects others to make no mistakes. Selfish motives, seeks personal gain. Importance (Valuing / Organizing) 2 Embodiment (Internalizing) 3 Usually is optimistic about doing work and takes on tasks willingly, but sometimes is stressed and wants to avoid extra work. Thrilled to face a challenge & do work; always in a good mood and looks at the positive side of things. Is not unrealistic in expectations of others. Often motivated internally and for the common good. Creates a foundation of honor. Tackles tasks wholeheartedly and works diligently to see the task completed fully. Allows others room to "not know" [Feito], and has a helpful rather than critical attitude. Acts selflessly for the benefit of others. I always bring positive energy to the group. An example is when we don't do as well as we expected on one of our presentations or reports, I am the one that will make a joke to lighten up the tension in the room. I usually make fun of Phil or someone. EXPECTED BEHAVIOR Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 17

PROFESSIONAL SKILLS RATING SHEET Developed by students in 2006 -2007 Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 18

PROFESSIONAL SKILLS RATING SHEET Developed by students in 2006 -2007 Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 19

PROFESSIONAL SKILLS RATING SHEET Developed by students in 2006 -2007 Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 20

PROFESSIONAL SKILLS RATING SHEET Developed by students in 2006 -2007 Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 21

PROFESSIONAL SKILLS RATING SHEET Developed by students in 2006 -2007 Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 22

Professional Skills Performance Review: PURPOSE IS DEVELOPMENT AND ASSESSMENT Adequate example / experience that demonstrates use of a range of professional skills Honest / valid rating and explanation of the example situation relative to the team / professional expectation Ability to identify influences or temptations that prevent professional-level performance Ability to design and implement a plan to improve individual performance in specific areas (to meet or exceed the expectations of the profession) Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 23

Assessment Results from 2005 -2006 Class (Developmental Taxonomy, Development Project Defined in Winter and Assessed in Spring) Adequate example / experience Low/Marginal: 9/48 that demonstrates use of the Acceptable: 26/48 professional skill Above Expectations: 13/48 > 81% met outcome * Plan to bring individual performance up to an > 75% met outcome** acceptable professional level * Students gave good examples but tended to over-rate their performance but truth emerges in face-to-face dialogue as part of the performance review **Senioritis prevented some from documenting the results from their development plans Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 24

Assessment Results from 2006 -2007 Class (Expected behavior Taxonomy, Development Project Defined in Fall, 1 st assessment in Winter, Follow-up Assessment in Spring) Adequate example / experience that demonstrates use of the 100%* met outcome professional skill Plan to bring individual performance up to an 100%* met outcome acceptable professional level Started the process earlier and was more clear in describing why we were doing performance reviews and how the would impact grades Performance level approach was more effective than development level Being part of Carnegie Scholar project made it seem important to students * Treated as mastery outcome with recycling Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 25

Observations and Lessons Learned Context Matters: >95% self report awareness of social impact of engineering, also high for the importance of professional skills • When asked to discuss and describe professional skills that are important for being a good engineer, students come up with impressive lists and examples Ø“They know professionalism when they see it”. Ø Student involvement in creating the list is essential • Although students can identify areas for personal improvement and come up with improvement plans, most need reminders and assignments or else the plan will get “lost in the shuffle. ” ØMost students are not self-motivated for “self evaluation, leading to improvement” Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 26

Observations and Lessons Learned • Although it takes a lot of time, face-to-face performance review meetings are very effective and allow a more authentic and professional discussion of goals, accomplishments, behaviors, etc. ØIncreases student engagement and accountability • There are some advantages to using team consensus rather than class consensus or consensus of the profession for establishing the professional skills list Ø Universal consensus is difficult (especially with language and terms that are understandable by all), and most behavioral expectations and norms are set and evaluated locally Ø Most behavioral expectations can be integrated into the “team contract” to model “effective teamwork” practices Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 27

Thanks! Greg Kremer (kremer@ohio. edu) Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 28

ABET and Professional Skills Outcomes • A sense of professional and ethical responsibility • An appreciation for the impact of engineering solutions in a global and societal context • A knowledge of contemporary issues • An awareness of the integral nature of learning and engineering practice (lifelong learning) • An ability to function effectively on multi-disciplinary teams (including an understanding of the positive role of diversity and specialization) • An ability to communicate effectively Performance Reviews for Prof Skills, Greg Kremer, Ohio University, Mechanical Engineering, June 2008, 29