HRPP METRICS Cynthia Monahan MBA CIP IRB Director

HRPP METRICS Cynthia Monahan, MBA, CIP IRB Director Boston University Charles River Campus IRB

Table of Contents l l l Overview of Metrics IRB Metrics QI Program Metrics AAHRPP Metrics for Quality Improvement

What are Metrics? l Standards of measurement by which efficiency, performance, progress, or quality of a plan, process, or product can be assessed http: //www. businessdictionary. com/definition/m etrics. html#ixzz 2 a 9 m 0 c. Hou

Why collect metrics? l Improve IRB processes l l l l Efficiency Quality Provide researchers with information that will assist them in planning research activities Determine adequacy of resources Identify gaps/problems Assess new policies/forms Target education Prevention

Define Metrics l Define parameters and measures • • What data will be collected Calendar days vs. business days Analysis (e. g. Mean vs. Median) Starting and stopping points for review times • Protocol received vs. protocol reviewed at meeting

IRB Metrics l Turn-around time l l l l IRB review category (expedited, exempt, full board) Submission type (new, CR, amendment, etc. ) Volume submitted Volume approved Submissions by department Submissions by IRB staff Workload Time in IRB Office vs. Time in PI office

Review Time Metrics 120 Average Days to Approva l - Turnaround Time 100 80 60 40 20 0 Period 1 Period 2 New Expedited New Full Board New Exempt 69 65 82 100 43 41 New Not Human Sbj 69 31 Amendments 22 18 ROS/Final Reports 47 38 Deviation Reports 64 12 Clarifications Other 3, 5 16 5 13

Volume Metrics

Metrics and the QI Program l Audits l l l For cause audit vs. not-for-cause audits l l IRB Research site Compare findings/observations Random audits l Define criteria

QI Program Metrics l IRB l l Minutes Categorization of reviews Waiver/Alteration of consent Research Sites l l l Compliance with protocol Compliance with institutional policies Prevention

Association for the Accreditation of Human Research Protection Programs (AAHRPP) l l l An independent, non-profit accrediting body AAHRPP uses a voluntary, peer-driven, educational model to ensure that Human Research Protection Programs (HRPPs) meet rigorous standards for quality and protection. To earn accreditation, organizations must provide tangible evidence—through policies, procedures, and practices—of their commitment to scientifically and ethically sound research and to continuous improvement

Goals, Principles, and Standards l l AAHRPP has adopted nine overarching principles for protecting human research participants, which serve as the foundation for the AAHRPP Accreditation Standards for Human Research Protection Programs. The AAHRPP Accreditation Standards themselves describe what Organizations can do to consistently act on these principles by applying them to the diverse organizational and cultural contexts in which research is conducted and reviewed

Accreditation Standards l STANDARD I-5: The Organization measures and improves, when necessary, compliance with organizational policies and procedures and applicable laws, regulations, codes, and guidance. The Organization also measures and improves, when necessary, the quality, effectiveness, and efficiency of the Human Research Protection Program.

Elements Element I. 5. A. The Organization conducts audits or surveys or uses other methods to assess compliance with organizational policies and procedures and applicable laws, regulations, codes, and guidance. The Organization makes improvements to increase compliance, when necessary. Element I. 5. B. The Organization conducts audits or surveys or uses other methods to assess the quality, efficiency, and effectiveness of the Human Research Protection Program. The Organization identifies strengths and weaknesses of the Human Research Protection Program and makes improvements, when necessary, to increase the quality, efficiency, and effectiveness of the program.

Element I. 5. A • An Organization’s quality improvement program should include measures of compliance with organizational policies and procedures and applicable laws, regulations, codes, and guidance • • • Audits and reviews to determine compliance Audits should be systematic and consistent The Organization’s quality improvement program should include an evaluation of the HRPP to determine whether it is effective in achieving compliance • • Review of minutes Review of categories (exempt, expedited, waivers, etc. )

Element 1. 5 A (cont) l The organization should collect objective data through audits, surveys, or other methods and use the data to make improvements and monitor compliance on an ongoing bases l l l Share data with stakeholders Revise programs/processes based on objective data The number and breadth of audits should be determined by the organization and should be sufficient enough to provide data that inform the quality improvement program l Identify measures and volume in advance

Element 1. 5 A (cont) Essential Requirements for Written Materials: l The Organization has a quality improvement plan that periodically assess compliance with the HRPP: l l The plan states the goal of the quality improvement plan The plan defines at least on objective to achieve or maintain compliance The plan defines at least one measure of compliance The plan describes the methods to assess compliance and make improvements

Element I. 5. B l An Organization’s quality improvement program should include measures of quality, efficiency, and effectiveness to evaluate the performance of the HRPP l l Performance metrics The Organization should use the results from the quality improvement program to design and implement improvements l How are metrics used

Element I. 5. B (cont) l The Organization should collect objective data through audits, surveys, or other methods and use the data to make improvements and monitor quality, efficiency, and effectiveness on an ongoing basis l l Share data with stakeholders Revise programs/processes based on objective data

Element I. 5. B (cont) Essential Requirements for Written Materials: l Quality Improvement Plan l l States the goals of the quality improvement plan with respect to achieving targeted levels of quality, efficiency, and effectiveness of the HRPP Defines at least one objective of quality, efficiency, or effectiveness Defines at least one measure of quality, efficiency, or effectiveness Describes the methods to assess quality, efficiency, and effectiveness and makes improvements

How do you meet the Elements? l l Compliance Plan Audits, surveys, or data collection tools Quality Improvement Plan Evaluation Reports

Metrics for Quality Improvement l l l Define responsibilities and roles Collect what you measure and measure what you collect Identify measures and goals in advance Continuous process that should be re-assessed and revised as appropriate (Annual basis) Periodic reviews External audits/inspections (FDA, etc. )

Metrics for Quality Improvement l l l l Track feedback Analyze results (who, what, when) Review outliers Share results Training and education based on results (individual vs. trends) Process for implementing changes Benchmark—identify peer institutions

Resources l l AAHRPP: http: //www. aahrpp. org/ University of Washington: http: //www. washington. edu/research/hsd/topi cs/HSD+metrics