Standardized Letters of Recommendation Anna H Messner MD

Standardized Letters of Recommendation Anna H. Messner, MD Professor & Vice Chair Program Director Otolaryngology/Head & Neck Surgery Stanford University

Application season

Why have letters? • Review of 966 LOR to General surgery program • 24% were helpful in evaluation process Fortune JB. The content and value of letters of recommendation in the resident candidate evaluative process. Curr Surg 2002

Why have letters? The theory. • Information on past performance which may be a good predictor of future success • Unique features • Insight into typical performance • (as opposed to maximal performance) • Opportunity to explain a less than perfect application • Opportunity to promote students he/she envisions as becoming future leaders in field.

Medicine “seems to have taken up permanent residence along the shores of Lake Wobegon. All of the applicants are above average—way above. ” All are outstanding Schneider A. Why you can’t trust letters of recommendation. Chron High Educ 2000

Writing letters- The basics 1. Introduction 2. Relationship to applicant Length of time he/she has known subject Context of relationship Right to view waived 3. Describe subject’s “record, ” Brief summary of subject’s academic performance Summary of clinical performance 4. Subject’s traits and performance are evaluated and compared to peers (+ unique features). 5. Summary sentence

OHNS Words/letter • • 763 letters • Chair of dept (290) vs others (350) (p<. 001) Letters Mean = 329 (82 -1297) Messner A, Shimahara E Laryngoscope 2009

What we currently do… • Look for Standout adjectives (average 2. 6 in OHNS letters) • star, excellent, superb, gifted, extraordinary, outstanding, unique, exceptional, unparalleled, superstar, finest, exquisite, stellar, sterling, phenomenal, ideal, exemplary • Grindstone adjectives: • Hardworking, resolve, conscientious, dependable, meticulous, thorough, diligent, dedicated, careful, responsible, reliable, tireless, solid • “Gets along with others”, “Team player” • “Compassionate, Caring, Kind, Empathy” • “Intelligent”

Features of letters • Leadership potential • High match rank • Sense of humor • Final sentence: Intentionally ambiguous • Recommend him without hesitation vs recommend highly vs highest recommendation vs absolute highest recommendation • Strong recommendation vs very strong recommendation vs strongest recommendation • Compare letters from same letter writer

Watch for doubt raisers • • • “Technical skills were average. ” “I have discussed alternative career options” “Made an effort to be an effective team member. ” “Average fund of knowledge” “His score of 210 accurately reflects his knowledge level. ”

Doubt Raisers Doubt raisers • “It absolutely amazes me how he can work with others. ” • “He’s always trying. ” • “I assure you that no person would be better for the job. ”

Typical Letter analysis: beginning & end Dear Program Director: I have been asked to submit a letter of recommendation on behalf of Jane Doe who has waived her right to see this letter. I have known Jane for the past 2 years since she first became interested in Otolaryngology. Jane is a fourth year medical student at X School of Medicine. She graduated from Y University in 2002 with a Bachelor of Arts and History. She has achieved several awards during her medical school tenure including “Outstanding Service” in April 2006. This is an award elected annually by current X medical students to the one third year student who contributes significantly to improving the lives of fellow students. Additionally, Jane obtained honors in pediatrics, a high pass in surgery, a high pass in obstetrics and gynecology. Jane rotated on the Head and Neck Surgery Service in the Department of Otolaryngology Head and Neck Surgery at S School of Medicine. During her rotation Jane performed in an outstanding manner. She is a bright and intelligent student who has an excellent knowledge of the head and neck anatomy and diseases that afflict the head and neck. She was actively engaged throughout her rotation in caring for the head and neck cancer patients in the ward. Additionally, she played an active role in the operating room where she clearly demonstrated an affinity for surgery. She has excellent hand-eye coordination and an excellent sense of the anatomy. Jane was well liked by her peers, the residents and the attending staff. She is a pleasant person to work with. She is hard working and while somewhat shy Jane is clearly well informed and well read. In general I was extremely impressed with this young student who I feel will make an outstanding contribution to any otolaryngology head and neck surgery training program. As a result, I would like to submit a very strong letter of recommendation on behalf of Jane Doe. If I can be of further assistance please feel free to contact me. Sincerely,

Typical Letter- ignore duplication Dear Program Director: I have been asked to submit a letter of recommendation on behalf of Jane Doe who has waived her right to see this letter. I have known Jane for the past 2 years since she first became interested in Otolaryngology. Jane is a fourth year medical student at X School of Medicine. She graduated from Y University in 2002 with a Bachelor of Arts and History. She has achieved several awards during her medical school tenure including “Outstanding Service” in April 2006. This is an award elected annually by current X medical students to the one third year student who contributes significantly to improving the lives of fellow students. Additionally, Jane obtained honors in pediatrics, a high pass in surgery, a high pass in obstetrics and gynecology. Jane rotated on the Head and Neck Surgery Service in the Department of Otolaryngology Head and Neck Surgery at S School of Medicine. During her rotation Jane performed in an outstanding manner. She is a bright and intelligent student who has an excellent knowledge of the head and neck anatomy and diseases that afflict the head and neck. She was actively engaged throughout her rotation in caring for the head and neck cancer patients in the ward. Additionally, she played an active role in the operating room where she clearly demonstrated an affinity for surgery. She has excellent hand-eye coordination and an excellent sense of the anatomy. Jane was well liked by her peers, the residents and the attending staff. She is a pleasant person to work with. She is hard working and while somewhat shy Jane is clearly well informed and well read. In general I was extremely impressed with this young student who I feel will make an outstanding contribution to any otolaryngology head and neck surgery training program. As a result, I would like to submit a very strong letter of recommendation on behalf of Jane Doe. If I can be of further assistance please feel free to contact me. Sincerely,

Typical Letter analysis: standouts, grindstone, team player Dear Program Director: I have been asked to submit a letter of recommendation on behalf of Jane Doe who has waived her right to see this letter. I have known Jane for the past 2 years since she first became interested in Otolaryngology. Jane is a fourth year medical student at X School of Medicine. She graduated from Y University in 2002 with a Bachelor of Arts and History. She has achieved several awards during her medical school tenure including “Outstanding Service” in April 2006. This is an award elected annually by current X medical students to the one third year student who contributes significantly to improving the lives of fellow students. Additionally, Jane obtained honors in pediatrics, a high pass in surgery, a high pass in obstetrics and gynecology. Jane rotated on the Head and Neck Surgery Service in the Department of Otolaryngology Head and Neck Surgery at S School of Medicine. During her rotation Jane performed in an outstanding manner. She is a bright and intelligent student who has an excellent knowledge of the head and neck anatomy and diseases that afflict the head and neck. She was actively engaged throughout her rotation in caring for the head and neck cancer patients in the ward. Additionally, she played an active role in the operating room where she clearly demonstrated an affinity for surgery. She has excellent hand-eye coordination and an excellent sense of the anatomy. Jane was well liked by her peers, the residents and the attending staff. She is a pleasant person to work with. She is hard working and while somewhat shy Jane is clearly well informed and well read. In general I was extremely impressed with this young student who I feel will make an outstanding contribution to any otolaryngology head and neck surgery training program. As a result, I would like to submit a very strong letter of recommendation on behalf of Jane Doe. If I can be of further assistance please feel free to contact me. Sincerely,

Typical Letter analysis: doubt raisers Dear Program Director: I have been asked to submit a letter of recommendation on behalf of Jane Doe who has waived her right to see this letter. I have known Jane for the past 2 years since she first became interested in Otolaryngology. Jane is a fourth year medical student at X School of Medicine. She graduated from Y University in 2002 with a Bachelor of Arts and History. She has achieved several awards during her medical school tenure including “Outstanding Service” in April 2006. This is an award elected annually by current X medical students to the one third year student who contributes significantly to improving the lives of fellow students. Additionally, Jane obtained honors in pediatrics, a high pass in surgery, a high pass in obstetrics and gynecology. Jane rotated on the Head and Neck Surgery Service in the Department of Otolaryngology Head and Neck Surgery at S School of Medicine. During her rotation Jane performed in an outstanding manner. She is a bright and intelligent student who has an excellent knowledge of the head and neck anatomy and diseases that afflict the head and neck. She was actively engaged throughout her rotation in caring for the head and neck cancer patients in the ward. Additionally, she played an active role in the operating room where she clearly demonstrated an affinity for surgery. She has excellent hand-eye coordination and an excellent sense of the anatomy. Jane was well liked by her peers, the residents and the attending staff. She is a pleasant person to work with. She is hard working and while somewhat shy Jane is clearly well informed and well read. In general I was extremely impressed with this young student who I feel will make an outstanding contribution to any otolaryngology head and neck surgery training program. As a result, I would like to submit a very strong letter of recommendation on behalf of Jane Doe. If I can be of further assistance please feel free to contact me. Sincerely,

Typical Letter Dear Program Director: I have been asked to submit a letter of recommendation on behalf of Jane Doe who has waived her right to see this letter. I have known Jane for the past 2 years since she first became interested in Otolaryngology. Jane is a fourth year medical student at X School of Medicine. She graduated from Y University in 2002 with a Bachelor of Arts and History. She has achieved several awards during her medical school tenure including “Outstanding Service” in April 2006. This is an award elected annually by current X medical students to the one third year student who contributes significantly to improving the lives of fellow students. Additionally, Jane obtained honors in pediatrics, a high pass in surgery, a high pass in obstetrics and gynecology. Jane rotated on the Head and Neck Surgery Service in the Department of Otolaryngology Head and Neck Surgery at S School of Medicine. During her rotation Jane performed in an outstanding manner. She is a bright and intelligent student who has an excellent knowledge of the head and neck anatomy and diseases that afflict the head and neck. She was actively engaged throughout her rotation in caring for the head and neck cancer patients in the ward. Additionally, she played an active role in the operating room where she clearly demonstrated an affinity for surgery. She has excellent hand-eye coordination and an excellent sense of the anatomy. Jane was well liked by her peers, the residents and the attending staff. She is a pleasant person to work with. She is hard working and while somewhat shy Jane is clearly well informed and well read. In general I was extremely impressed with this young student who I feel will make an outstanding contribution to any otolaryngology head and neck surgery training program. As a result, I would like to submit a very strong letter of recommendation on behalf of Jane Doe. If I can be of further assistance please feel free to contact me. Sincerely, NOTE: nothing unique, interesting, no comparison to peers, no mention leadership potential, no humor, no match potential

Letter of Minimal Assurance “I believe that Joe is a strong candidate for a residency position and that he will ultimately be an excellent otolaryngologist and contributor to medicine. He is passionately committed to this goal and he is driven by that commitment. He is a unique person, and one whom you will enjoy meeting and interviewing. I would of course welcome a direct phone call regarding this letter. Thank you for considering Joe for an interview position. ”

Current Status • • Average program: 250 applications x 4 letters = 1000 letters Average letter = 329 words Time = 1 min/letter (without analysis) 1000 min = 17 hours. • Result: Letters often not used to make 1 st cut • Increased reliance on USMLE scores • And… many not read at all.

Standardized letter of recommendation (SLOR) • Council of Emergency Medicine Residency Directors created SLOR • Successfully used since 1996

Comparison of SLOR & NLOR • Part I: Compared 20 SLOR and 20 NLOR • 7 - point Likert-type scale • 4 raters (2 experienced, 2 inexperienced) • Results • Interrater reliability of SLOR = 0. 97, NLOR 0. 78 • Average time to interpret SLOR = 16 sec, NLOR = 90 sec Girzadas DV Jr, Harwood RC, Dearie J, Garrett S. Acad Emerg Med. 1998 Nov; 5(11): 1101 -4

Comparison of SLOR & NLOR • Part I: Compared 20 SLOR and 20 NLOR • 7 - point Likert-type scale • 4 raters (2 experienced, 2 inexperienced) • Results • Interrater reliability of SLOR = 0. 97, NLOR 0. 78 • Average time to interpret SLOR = 16 sec, NLOR = 90 sec • Part II: 207 SLOR/NLOR pairs • Results • 54% given same numeric value, 39% differed by one point, 6% differed by 2 points. Overall correlation was 0. 58. Girzadas DV Jr, Harwood RC, Dearie J, Garrett S. Acad Emerg Med. 1998 Nov; 5(11): 1101 -4

EM SLOR • 173 surveys of PD organization (70% response rate) • “Compared with the narrative LOR, please rate the SLOR in terms of: ” • Ability to discriminate differences between candidates • Better 75%, Same 18%, Worse 7% • Ease of reading and incorporating into ranking scheme • Better 84%, Same 12% Worse 4% • Ease of completion • Better 83%, Same 9%, Worse 4% • Would you like to continue using the SLOR? • Yes 90%, No 7% Keim SM, et al. Acad Emerg Med. 1999 Nov; 6(11): 1141 -6

Proposed Standardized Letter of Recommendation

Standardized LOR

Standardized LOR

Compare to other medical students

Standardized LOR

Standardized LOR

Standardized LOR

Standardized LOR

Standardized LOR

Standardized LOR

Advantages of SLOR • • More consistent information Less duplication- of contents of application and letters Less time to complete & interpret Information less dependent on letter writer (more fair) • (A superb letter writer may portray a “good” candidate in stratospheric terms, whereas a mediocre letter writer may portray a “superb” candidate less positively) • Still have opportunity to tell about unique features of applicant

Disadvantages of SLOR • • Potential loss of nuances Much harder to make the average student look stupendous Potential attending MD guilt Don’t get to show off fancy Department stationary • But OK to add Logo, or put form on stationary

Next Steps • Open for comments until February 1, 2012 • Send to amessner@ohns. stanford. edu • OPDO council will review all comments and revise proposed SLOR • Assuming response is positive: • Will post SLOR on SUO-AADO website • Email distribution to PDs and Chairs • Need Program director assistance to get the word out to faculty and applicants • Investigate information re SLOR in Oto journal • Start using in spring/summer 2012 • Survey SUO/AADO/PD re use, then revise.

Conclusions • Traditional letters have the potential to help with identifying top residency candidates– but often do not. • A SLOR has the potential to increase the value of LOR while saving time. • Please send all comments to: amessner@ohns. stanford. edu