Implementation of MEMS Standards for Outcomes Assessment of

Implementation of MEMS Standards for Outcomes Assessment of Continuing Education: Lessons & Recommendations for Educational Providers Alliance for CME Annual Conference January 23, 2012 Orlando, Florida

Presenters and Disclosures • Sean Hayes, Psy. D Vice President of AXDEV Group Does have for sale a technology, program, product, and/or service for CME/CPD professionals. • Jack Kues, Ph. D. University of Cincinnati, Associate Dean for Continuing Medical Education Does not have for sale a service for CME/CPD professionals. • Andrew Rabin, BS Chief Technology Officer, CECity Does have for sale a technology, program, product, and/or service for CME/CPD professionals. 2

Objectives of Today’s Session • Review the MEMS and PARS implementation • Discuss potential value to their organization • Explore issues and challenges based upon ‘early adopters’ experiences regarding MEMS implementation 3

Discussion • Your background • What you hope to get out of this session

Med. Biquitous Mission To advance healthcare education through technology standards that promote professional competence, collaboration, and better patient care. Not-for-profit, member-driven, standards development organization

Med. Biquitous Goals • Better tracking and evaluation of professional education and certification activities • Easier discovery of relevant education and information resources when and where needed • Interoperability and sharing of high quality online education • Coordination and tracking of competence assessment data

Med. Biquitous Metrics Working Group Charter To develop XML standards & Web services requirements and descriptions for the exchange of aggregate evaluation data and other key metrics for health professions education. 7

MEMS: Medical Education Metrics • Technology standard for core evaluation data • Users – Educators want best practices, ability to compare – Funders want to measure reach and efficacy – Accreditors want to measure success of activity and provider

Discussion • How are you currently managing data for your CME activities? – Evaluation data – Participation/CE Certificates – Assessment results

MEMS Data • Activity Description – What’s being evaluated • Participant Activity Evaluation – What did participants think • Participation Metrics – How many people participated • Learner Demographics

MEMS is XML • Structured data • Strong validation built in error checking • Designed for automation – computer to computer communications • Can also use in file upload or FTP Upload

Discussion • How are you using the following kinds of data? – Evaluation data – Participation data – Assessment results

Discussion • Do you see opportunities to do more with your data? • What are the barriers you see to making the most of your data?

ACCME PARS Survey Results • After submitting their 2010 activity and program data, ACCME-accredited providers were invited to complete an online survey asking for their feedback about their experience using PARS. • Responses from 222 (32%) out of 694 ACCMEaccredited providers. http: //www. accme. org/dir_docs/doc_upload/1156 c 8 b 63 d 2 e-401 b-b 0 bf-829 e 49028741_uploaddocument. pdf

Results • • 161 providers used the web form 77 used tab delimited upload 5 used XML upload Uploads more likely from providers with more activities (>100) • Entering data perceived as easy overall (> 20%)

Key challenges • Learning the system (> 60%) • New processes (> 40%) • Coordinating with IT for year-end reporting (> 20%) • Expending additional resources (> 20%)

Discussion • Did you upload data to PARS in CSV or XML? • Do you have any insights to share?

From Andy’s outline • Current usage of MEMS – ACCME • ACCME PARS Process Overview • ACCME PARS Survey results – XXXXX • ACCME PARS Support – CECity • ACCME PARS Support • Opportunities for expanded adoption – ACME Survey – Healthcare Reform – Open Brainstorming / Engage Audience

Case Study • Implemented MEMS for PARS upload • Experience implementing Med. Biquitous standards • Integrated new data into user interface and workflow • Provided user training

Opportunities for Expanded Adoption • Multi-site CME Effectiveness Research • ? ?

Lessons Learned • Standards implementation can bring about efficiencies when integrated into workflow and properly supported • Validation tools needed • Web services needed to realize full benefits, one click data transfer 24