ONLINE POINTOFCARE FEEDBACK TO IMPROVE MILESTONESPECIFIC EVALUATIONS IN

ONLINE POINT-OF-CARE FEEDBACK TO IMPROVE MILESTONE-SPECIFIC EVALUATIONS IN FAMILY MEDICINE RESIDENTS Stephen Warnick Jr. , M. D. Clinical Assistant Professor, Wayne State University DMC-Sinai Grace/WSU Family Medicine Residency Assistant Program Director, WSU/DMC Sinai-Grace Psychiatry Residency

Seminar Goals & Objectives At the end of this seminar, participants will be able to… List common websites and systems that can be used to format point-of-care evaluations for residents Identify an area in your residency where point-of-care evaluations will improve resident feedback and education Use point-of-care evaluations to form individualized reports to improve milestone-specific feedback on formal evaluations

Intended Audience Behavioral Scientists Family Medicine Core Faculty Family Medicine Residents Family Medicine Staff involved in resident evaluations (i. e. Program Coordinators)

TECHNOLOGY ENCOURAGED If you have a laptop, take it out! If you have a tablet, take it out! If you have a smartphone, take it out! We will use technology to engage today. Partner up with somebody if you do not have a web-ready device.

Using Technology to Engage Learners On-line pre-tests during lectures Audience response monitoring (remotes or texting) Web modules for learning during rotations You. Tube videos to demonstrate points Blogging for reflection Point-of-care evaluations YOUR IDEAS?

PRE-TEST https: //docs. google. com/form s/d/1 Kw. TSb 0 Xq. MQz. I 6 Neev. Irx. WBu 4 hy. Ln. Oun GKb. RLq. Vy. J 8 Q/viewform Copy and paste this into your browser OR

Website for Seminar www. wsubehavioral. com/feedback Password: feedback All of the evaluations we will discuss will be linked to on this website

Point-of-Care Evaluations INTENT Provide QUICK, EASY, USEFUL feedback to learners. Provide feedback that learner can access IMMEDIATELY Provide faculty with concrete examples of resident’s strengths and areas to improve

The Evaluation Doldrums How often have you been faced with a resident or student evaluation and not been sure how to rate the learners? OR How often have you wanted to rate a learner as deficient or strong but could not provide evidence to support your evaluation?

How is Point-of-Care Different? Every rotation has an evaluation at the end-of-rotation Good rotations do a mid-rotation feedback Does this happen in continuity clinic always? Point-of-care allows direct, concrete feedback and areas to improve No delay in recall for evaluator, no delay in obtaining for recipient

Evidence for Online Evals Literature search with scant evidence base Virtua FM program with POC evaluations correlated with ITE scores Literature base for audience evaluation improving education Nursing studies demonstrating increased knowledge with POC

Where Can We Do Pointof-Care? PRECEPTING Signing off notes Commenting on quality of notes Commenting on professionalism - notes done on time, labs signed off, phone calls answered Inpatient rounds Behavioral Medicine Clinic 360 Degree Evaluations

Point-of-Care for Milestone Education Evaluations can be mapped to the milestones Evaluation on a particular milestone Can help faculty track which milestones they need to evaluate on a resident

Residency Management Systems Which programs does your residency use?

Residency Management Systems - Sample New Innovations Med. Hub E*Value My. Evaluations Residency Partner Verinform RM Hospital or Residency Designed Systems AND MANY MORE!

Residency Management Systems You may be able to do point-of-care evaluations in your RMS Sometimes artificial barriers (do not have access to create - need program coordinator) May not be able to e-mail directly to resident The stigma of logging into your RMS, both faculty and resident

New Innovations Sample Program Director inspired by FM PDW presentation by Timothy P. Graham, M. D. of Mount Carmel FM Residency in Westerville, OH Works well to track milestones Our limitation was this was difficult to use to give immediate feedback (could not e-mail directly to resident once submitted; needed to log onto New Innovations)

Internet Sites for Evaluations I have no vested interest in one RMS or internet site I personally use wufoo because that’s what I worked on in residency but each site has it’s positive and negatives You will see a variety of platforms but this is NOT all inclusive

Sample Sites for Internet Evaluations Survey Monkey Wufoo Type Form Google Forms Zoho Survey Gizmo Survey Planet Etc, etc

Sample Sites for Internet Evaluations Important things to consider: Cost How many users? Will your residents use this to make their own? Will faculty have access? Your program coordinator?

Sample Sites for Internet Evaluations Important things to consider: How do evaluations get sent to the learner? Is there a way to e-mail? Is this automatic? REPORTING Collecting info does not good if you cannot access it EASILY Does it export to Excel only? Does it make a report that can be easily printed or viewed?

Sample Sites for Internet Evaluations Important things to consider: How does it look? If aesthetically pleasing more likely to be used If it looks cumbersome, it will be skipped Can you customize URL? How will you communicate address to faculty/staff/learners?

Sample Sites for Internet Evaluations Important things to consider: Does it work well with mobile devices? WHAT OTHER CONSIDERATIONS ARE IMPORTANT TO YOU?

Tool for Evaluating Sites Linked to website and on FMDRL

Tool for Evaluating Sites Evaluation Tool for Online Point-of-Care Survey Sites • Can you easily do this with your residency management software? • How much does the site cost? Do you have access to all features for free? • Can you easily make reports? Do reports cost extra? • How do reports appear? Graphic? Excel export?

Tool for Evaluating Sites • Can results be easily sent to learners? • Is the form aesthetically attractive online and on mobile devices? • How easy can you make surveys? Is there a steep learning curve? • Can you add extra users? • Does it cost money to have extra users?

SAMPLE FORMS Let’s look at sample evaluation forms

Reporting https: //wsufm. wufoo. com/reports/anonymousresident-20152016 -report-precepting/

Resident Feedback Faculty evaluated anonymously twice a year Residents can comment on ANYTHING

Resident Feedback My feedback: 5/10 evaluations (only 12 residents in program) mentioned quality of feedback or timeliness of feedback 3 Other faculty: 2/10 evaluations state constructive feedback This faculty uses our forms but not consistently 1/10 mentioned would like more feedback

YOUR TURN! Pick an area where your program needs to improve resident feedback Pick a platform Create user account Use evaluation tool to see if this works well for your program

When Making Evaluations Try to minimize questions if you want others to use this Incorporate the milestones as much as possible Find a way to make url easy for users (may not be possible)

Resources Post RE, Jamena GP, Gamble JD. Using Precept-Assist to Predict Performance on the American Board of Family Medicine In-Training Examination. Fam Med 2014; 46(8): 603 -607. Looks at point-of-care evaluation form that correlates to ABFM ITE scores Wojcikowski K, Kirk. L. Immediate detailed feedback to testenchanted learning: an effective online educational tool. Med Teach 2013; 35(11): 915 -919. Online testing, and then giving answers, improves learner’s knowledge.

Resources Grossman S, Conelius J. Simulation pedagogy with nurse practitioner students: impact of receiving immediate individualized faculty feedback. Creat Nurs 2015; 21(2): 100 -109. Students receiving immediate feedback after clinical simulations showed better test scores and preceptor evals. de la Cruz MS, Kopec MT, Wimsatt LA. Resident perceptions of giving and receiving peer-to-peer feedback. J Grad Med Educ 2015 Jun; 7(2): 208 -213. Residents filling out an online peer feedback survey found the experience useful.

Resources Rush S, Ooms A, Marks-Maran D, Firth T. Students’ perceptions of practice assessment in the skills laboratory: an evaluation study of OSCAs with immediate feedback. Nurse Educ Pract 2014 Nov: 14(6): 627 -634. Students liked immediate feedback after observed clinical simulation.