GOALORIENTED ASSESSMENT OF LEARNING CREATION OF AN INDIVIDUALIZED

GOAL-ORIENTED ASSESSMENT OF LEARNING: CREATION OF AN INDIVIDUALIZED LEARNING PLAN FOR RESIDENTS Patrick Huffer MD, Thomas Hahn MD, Jensena Carlson MD University of Wisconsin, Madison Department of Family Medicine and Community Health May 2, 2016

OBJECTIVES • At the end of this presentation the participants will be able to: • 1. Appreciate background of individualized learning plans (ILPs) in medical education • 2. Introduce the Goal-Oriented Assessment of Learning (GOAL) or similar ILP to learners • 3. Assess the efficacy of GOAL in aiding residents with future planning using results of a pilot study • 4. Apply the GOAL to the Practice-Based Learning and Improvement-2 milestone regarding self-directed learning.

BACKGROUND: ILPS • Self-directed learning, self-assessment, and goal-setting are important skills for lifelong improvement as a physician • ILPs: § Encourage users to develop measurable goals and track progress (Li et al. , 2010) § Increase motivation and sharing of ideas among learners (Chan & Chein, 2000)

CHALLENGES OF ILPS • Thinking of concrete goals can be difficult (Stuart et al. , 2005) • Difficult to apply ILP over a short time period (like a clinical rotation) § Greater utility over the span of residency?

ILPS IN FAMILY MEDICINE • Little research in family medicine • Most research involves pediatric residents, nursing and medical students § Pedialink

WHAT IS THE GOAL? • Goal-Oriented Assessment of Learning § ILP for residents to help plan/organize residency and future learning through self-assessment and goal setting § Created using guidance from learning plans described in the literature § For use by University of Wisconsin Madison Family Medicine residents (n=43)

GOAL Complete GOAL at start of residency and update every 6 months (Supplement form) Upload/store in New Innovations Review every 6 months with APD and clinicbased faculty advisor

SPECIFIC AIMS OF THE GOAL • Tailor residency learning to long-term career goals o Set learning goals every 6 months to shape residency experience to help reach career goals • Identify strengths and deficiencies • Establish plans for developing areas of interest • Select a "team" of mentors and others to help meet goals • Provide a uniform framework to track educational development • Help facilitate development of curriculum vitae

WHAT DOES THE GOAL LOOK LIKE? • 7 main sections § A. Career goals § B. Goals to meet by the end of residency § C. Self-assessment of areas of confidence and challenge § D. Pursuits (residency activities) § E. 3 SMART (Specific, Measurable, Attainable, Realistic, Timebound) goals for the next 6 months § F. 3 personal goals for the next 6 months § G. Mentorship team

STUDY TIMELINE/METHODS • Summer 2015: IRB application, exempted as not research • September 2015: Introduced the GOAL form via seminar, email o Pre-survey (Qualtrics) o Residents review GOAL with APDs • March 2016: Residents update the GOAL form using supplement o Post-survey (Qualtrics) o GOALS reviewed with APD during Spring meeting • Ongoing: Every 6 months, residents update GOAL using the GOAL supplement and save on online platform

RESPONSE DEMOGRAPHICS Total residents: 43 Pre-survey participants: 33 (77%) Post-survey participants: 14 (33%) Residents who completed both surveys: 10 (23%)

RESIDENCY GOALS I have goals for my residency experience. 6 Likert Scale 5 4 4. 42 4. 73 4. 27 All Classes 3 2 Year 1 Year 2 Year 3 1 0 5 = Strongly Agree 4 = Agree 3 = Neutral 2= Disagree 1 = Strongly Disagree

MOTIVATION I am motivated to reach my learning goals 6 Likert Scale 5 4 3 2 4. 21 4. 36 4. 45 3. 82 All Classes Year 1 Year 2 Year 3 1 0 5 = Strongly Agree 4 = Agree 3 = Neutral 2= Disagree 1 = Strongly Disagree

CONFIDENCE I am confident that I am reaching my residency learning goals. 5 Likert Scale 4 3 3. 42 3. 18 3. 36 3. 73 All Classes Year 1 2 1 Year 2 Year 3 0 5 = Strongly Agree 4 = Agree 3 = Neutral 2= Disagree 1 = Strongly Disagree

POST-RESIDENCY CAREER I know what I want to do in my career after residency. 6 Likert Scale 5 4 3 2 3. 58 4. 00 3. 27 3. 45 All Classes Year 1 Year 2 Year 3 1 0 5 = Strongly Agree 4 = Agree 3 = Neutral 2= Disagree 1 = Strongly Disagree

SURVEY QUESTIONS AND RESPONSES The following questions showed no significant difference between pre- and post-test: § I have goals for my residency experience. § I have an organized plan for my learning throughout residency. § I am motivated to reach my learning goals. § I am confident that I am reaching my residency learning goals. § I believe my residency program has a system in place to help me reach my learning goals. § I know what I want to do in my career after residency. § I am aware of my strengths and weaknesses.

SURVEY QUESTIONS AND RESPONSES Though not statistically significant, there was a potential trend showing that use of the GOAL may help with reaching residency goals § I am confident I am reaching my residency goals (p = 0. 081)

POSTTEST SPECIFIC QUESTIONS To what extent did completing the GOAL tool help you set residency goals? 2. 82 (0. 40) To what extent did completing the GOAL tool help you design/complete a resume/curriculum vita? 2. 40 (0. 84) To what extent did completing the GOAL tool help you identify mentors? 2. 64 (0. 67) To what extent did completing the GOAL tool help you plan post-residency career plans? 2. 55 (0. 52) To what extent did completing the GOAL tool help you feel that you are reaching your goals? 2. 27 (0. 47) How useful was discussing the GOAL with your APD/faculty mentor? 2. 00 (0. 50) Responses based on Likert Scale, with 1 = Not at all, 2 = Neutral/Not Sure, 3 = A little, 4 = Very much/a lot

LIMITATIONS Difficult to draw meaningful conclusions between residency class averages § 5 -point Likert Scale not optimal § Low n, standard deviations overlap • Small sample size (total n=43) • Low survey response rate (77% for pre-survey, 33% for post-survey, and only 23% filled out both) • Short time period between assessments (6 months)

FUTURE DIRECTIONS • Adapt GOAL for use by faculty, medical students • Solicit subjective feedback on utility of the tool • Make the tool shorter/easier vs. longer/more comprehensive • Compare pre-residency responses with postresidency responses

REFERENCES Chan SW, Chien WT. Implementing contract learning in a clinical context: report on a study. Journal of Advanced Nursing. 2000; 3: 7. Li ST, Burke AE. Individualized learning plans: basics and beyond. Academic Pediatrics. 2010; 10: 289 -292. Li ST, et al. (2011). Is resident’s progress on individualized learning plans related to type of learning goal set? Academic Medicine, 86, 1293 -1299. Li ST, et al. Factors associated with successful self-directed learning using individualized learning plans during pediatric residency. Academic Pediatrics. 2010; 10: 124 -130. Lockspeiser TM, et al. (2013). Assessing residents’ written learning goals and goal writing skill: Validity evidence for the learning goal scoring rubric. Academic Medicine, 88. Mc. Dermott MM, Curry RH, Stille FC, Martin GJ. Use of learning contracts in an office-based primary care clerkship. Medical education. 1999; 33: 374 -381. Sectish T, Floriani V, Badat MC, Perelman R, Bernstein H. Continuous professional development: raising the bar for pediatricians. Pediatrics. 2002; 110: 152 -156. Shebard ME. (2012). Use of individualized learning plans among fourth-year sub-interns in pediatrics and internal medicine. Medical Teacher, 34, 46 -51. Stuart E, Sectish T, Huffman L. Are residents ready for self directed learning? A pilot program of individualized learning plans in continuity clinic. Ambulatory Pediatrics. 2005; 5: 298 -301.