Locus of Control and SelfAssessment Measures as Predictors

Locus of Control and Self-Assessment Measures as Predictors of Family Medicine Resident’s Academic Performance: Year Two Cohort Results Joseph Brocato, Ph. D, Erik Solberg, MA, and Shailendra Prasad, MD, MPH Department of Family Medicine and Community Health

Presentation Objectives – Define locus of control (LOC) and self-assessment (SA) and their potential relationship to academic performance – State the research design used for a study of LOC and SE at the UMN Dept of Family Medicine and Community Health – Identify the elements of the instruments to measure LOC & SA – Describe the next steps in the five-year study of UMN Family Medicine Resident’s LOC & SA

Global Challenges • Residents coming in from diverse background and different learning styles • Patients don’t check board scores • Current academic measures may not be indicative of future academic performance • Ill-defined, inconsistently applied, non-existent residency selection criteria

Current methods of measuring a resident’s performance • In training examination (ITE) • Board Scores • Rotation based evaluations of medical knowledge • Other evaluations – Structured & Unstructured

Key Concept: Locus of Control • Originally described by Rotter in 1950 s • Individual’s perception of underlying causes of events in life • Conceptualized as a unidimensional continuum from an internal to an external

Internal LOC • “Actions are within ones control” • General acceptance that internal LOC is desirable • Studies in the 1980 s that less people higher internal LOC going into primary care

Key Concept: Self Assessment • Y. Trope, (1982, 1975) – Model to predict task selection – Self assessment is narrowly predictive: it is limited to specific conditions – Based on the premise people strive to attain a realistic understanding of weaknesses and strengths to better predict and effectively cope with their environment – The attractiveness of a task depends on the extent to which it can reduce uncertainty about one’s ability level – All outcomes can have positive values. The scale of a success or failure determines the net impact on the individual

Research Questions from our Study • What is the LOC & SA of FM residents? • Change over time? • Differences in LOC & SA by demographic variables? • Are LOC & SA strong predictors of traditional academic performance during residency?

Research Design: Overview & Timeline • 5 year study • LOC & SA assessment – PGY 1: During first month & during last month of internship/PGY 1 – PGY 2: During last month of PGY 2 – PGY 3: During last month of PGY 3 • Resident Academic Performance – In Residency: Board Scores, ITE Exams – Post-residency: Board Certification Exams

About our residents • There a total of 55 first year residents (166 total) within eight sites across MN • Residents from the Duluth program (10 residents) did not participate in the study as they are a new residency in the system.

Resident sample demographics: Cohort 1 • 33 of 48 agreed to participate in the study (69%) • 33 residents completed the locus of control survey (33/33=100%)* – 27 MDs, 6 DOs – 14 men and 19 women – Includes 12 IMG residents • 4 men and 8 women » *During year 2, one resident opted out of the study and one has not completed the instrument for a total response rate of 31/33 (94%) • 29 residents completed the self-assessment survey (29/33=88%) – 23 MDs, 6 DOs – 10 men and 19 women – Includes 12 IMG residents • 4 men and 8 women

Resident sample demographics: Cohort 2 • 39 of 45 agreed to participate in the study (87%) • 37 residents completed the locus of control instrument (37/40=95%) – 31 MDs, 5 DOs, 1 MBBS – 20 women and 17 men – Includes 6 IMG residents • 4 men and 2 women • 36 residents completed the self-assessment instrument (36/39=92%) – 30 MDs, 5 DOs, 1 MBBS – 19 women and 17 men – Includes 6 IMG residents • 4 men and 2 women

Research Design: Instrumentation • Survey Monkey Longitudinal Resident Self-Assessments – Resident Self-Assessment of Knowledge and Skills – Duttweiler Internal Locus of Control Index

Resident Self Assessment of Knowledge and Skills • Demographics &Tracking information (confidential, not anonymous) • Knowledge areas based on FM program requirement subject matter • Scale for knowledge based upon Bloom’s Taxonomy (recall, comprehension, application, analytical, synthesis, evaluation) • Skills: Core departmental inpatient and outpatient procedures • Skills Scale: Dreyfus Model of Skill Acquisition • Overall self-assessment: ACGME general competencies

Duttweiler Locus of Control Index (1984) • Previously validated instrument of LOC • 28 question survey instrument • Complete questions using scaled responses • high internal LOC ~ score of 140. • low internal LOC ~ score of 28.

Locus of Control results: Cohort 1, Year 1 and Year 2 – Relatively high amount of internal LOC – Low degree of variability in class Cohort 1, Year 1 Cohort 1, Year 2 Mean 105. 15 103. 7 Median 106 104 Mode 99 99 Std. Deviation 8. 68 8. 54

Locus of control results: Cohort 1, Year 1 Maximum score of 140 Mean: 105. 15 Median: 106 Mode: 99 Std. Deviation: 8. 68 Minimum score of 28

LOC Results: Cohort 1, Year 2 Maximum score of 140 Mean: 103. 7 Median: 104 Mode: 99 Std. Deviation: 8. 54 Minimum score of 28

Self Assessment Results: Cohort 1, Year 1 and 2 • Low scores - participants have not encountered training or knowledge in these areas to this point • Scoring: – – 0 – no prior exposure 1 - rotation in medical 2 - Rotation in residency 3 - Rotation in both medical school and residency

Medical school/residency exposure/training results • Top three areas with highest reported training/exposure Cohort 1, Year 1 Cohort 1, Year 2 Maternity Care (mean=1. 79, SD=0. 94) Inpatient Pediatrics (mean=2. 39, SD=0. 76) Inpatient Pediatrics (mean=1. 72, SD=1. 0) Emergency Care (mean=2. 35, SD=0. 71) Surgical Care (mean=1. 72, SD=1. 0) Maternity Care (mean=2. 32, SD=0. 83)

Medical school/residency exposure/training results • Top three areas with least training/exposure Cohort 1, Year 1 Cohort 1, Year 2 Management of health systems (mean=0. 17, SD=0. 60) Oral health (mean=0. 35, SD=0. 75) Oral health (mean=0. 38, SD=0. 78) Endoscopy (mean=0. 52, SD=0. 57) Hematology (mean=0. 41, SD=0. 56) Infectious diseases (mean=0. 52, SD=0. 68)

Self Assessment: Current medical knowledge • Scoring: 7 point scale – 0 – little or nothing – 1 - “recall some basic facts” – 2 - “can understand pathophysiology” – 3 - “used knowledge in the care of some of my patients” – 4 - “can recognize the signs and symptoms consistently” – 5 - “can generalize from one patient to another” – 6 - “can compare to best practices”

Results: knowledge areas • 3 areas of highest reported knowledge/training (calculated by mean) Cohort 1, Year 1 Cohort 1, Year 2 Maternity care (mean=4. 03, SD=1. 05) Maternity care (mean=4. 83, SD=1. 00) Outpatient pediatrics (mean=3. 93, SD=1. 19) Outpatient pediatrics (mean=4. 72, SD=0. 96) Cardiology (mean=3. 79, SD=0. 91) Emergency Care (mean=4. 66, SD=. 86)

Results: knowledge areas • 3 areas of lowest reported knowledge/training (calculated by mean) Cohort 1, Year 1 Cohort 1, Year 2 Management of health systems (mean=1. 72, SD=1. 56) Oral health (mean=2. 93, SD=1. 58) Oral health (mean=2. 14, SD=1. 46) Opthamology (mean=3. 17, SD=1. 42) Hematology (mean=2. 52, SD=1. 15) Management of health systems (mean=3. 21, SD=1. 54)

Skill Assessment: outpatient/officebased procedures • Scoring: Participants reported their skills in common outpatient procedures – – – 0 - “novice” 1 - “advanced beginner” 2 - “competent” 3 -“proficient” 4 - “expert”

Results: outpatient/office based procedures – 3 areas of highest reported skill level (calculated by mean) Cohort 1, Year 1 Cohort 1, Year 2 Pap smear (mean=2. 45, SD=0. 95) Pap smear (mean=3. 41, SD=0. 68) Incision and drainage of abscess (mean=2. 17, SD=0. 89) Incision and drainage of abscess (mean=2. 97, SD=0. 82) EKG interpretation (mean=1. 69, SD=1. 17) Low-risk obstetrics (mean=2. 76, SD=0. 83)

Results: outpatient/office based procedures – 3 areas of lowest reported skills (calculated by mean) Cohort 1, Year 1 Cohort 1, Year 2 Vasectomy (mean=0. 20, SD=0. 41) Vasectomy (mean=0. 38, SD=0. 62) Thrombosed hemorrhoid incision (mean=0. 41, SD=0. 80) Aspiration of breast cyst (mean=0. 66, SD=0. 86) Reduction of uncomplicated dislocations (mean=0. 52, SD=0. 69) Thrombosed hemorrhoid incision (mean=0. 72, SD=0. 88)

Results: inpatient procedures – 3 areas of highest reported skill level (calculated by mean) Cohort 1, Year 1 Cohort 1, Year 2 Venipuncture (mean=1. 86, SD=1. 30) Vertex Delivery (mean=3. 0, SD=0. 71) Fetal Monitoring (mean=1. 83, SD=1. 00) Low Risk Obstetrics (mean=2. 72, SD=0. 80) Vertex Delivery (mean=1. 72, SD=1. 07) Fetal Monitoring (mean=2. 69, SD=0. 71)

Results: inpatient procedures – 3 areas of lowest reported skill level (calculated by mean) Cohort 1, Year 1 Cohort 1, Year 2 Forceps assisted vaginal delivery (mean=0. 38, SD=0. 86) Forceps assisted vaginal delivery (mean=0. 59, SD=0. 98) Central venous lines (mean=0. 55, SD=0. 74) Central venous lines (mean=0. 86, SD=0. 83) Chest tube placement (mean=0. 59, SD=0. 63) Chest tube placement (mean=0. 90, SD=0. 77)

Results: Overall ACGME Competency Assessment • Top two highest and bottom two lowest reported scores: • Highest: Cohort 1, Year 1 Cohort 1, Year 2 Communicating with patients and families (mean=2. 48, SD=0. 78) Communicating with patients and families (mean=3. 00, SD=0. 71) Engage other professionals (mean=2. 24, SD=0. 64) Engage other professionals (mean=2. 76, SD=0. 69) • Lowest: Cohort 1, Year 1 Cohort 1, Year 2 Practicing cost effective health care (mean=1. 59, SD=0. 63) Facilitating learning of students/other professionals (mean=2. 31, SD=0. 76) Practicing continuous quality improvement (mean=1. 72, SD=0. 80) Practicing cost effective health care (mean=2. 31, SD=0. 89)

Next Steps • Now-July 1, 2011: • Cohort 2 participants will at year end retake the self assessment and locus instruments • We will collect USMLE/AOA Board Scores and in-training scores • July 1, 2011: Assessment of new incoming class of approximately 45 residents begins

Questions?