RANDUCLA Appropriateness Method Study To Identify Quality Measurement

RAND/UCLA Appropriateness Method Study To Identify Quality Measurement Indicators Matthew Byrne, Academic Clinical Fellow in Primary Dental Care University of Manchester

Background Dimensions of Quality Domains of Quality Safe Structure Process Outcome Effective Patientcentred Indicator Timely Efficient Equitable ? ? ?

Aims 1) To gain consensus on the dimensions of quality that are important for Primary dental care to inform a definition of quality for dentistry; 2) To assess the currently available quality measures and to aim to gain consensus on quality measures that are clear, necessary and feasible for construction a core set of quality indicators 3) To identify the dimensions and domains of quality where further development of quality measures is required. 4) To identify measures that require further development to allow them to be used as quality indicators.

Identifying core measures - RAND • Consensus forming exercise to gain face validity on measures. • Previously used in development of the GP QOF • 2 Rounds of rating (Online, then face to face) • Quantifies how necessary, clear and feasible measures are and gives indication of agreement • Existing Measures from grey and peer reviewed literature identified in systematic review • Dimensions of quality outside of Io. M definition identified through literature search Clarity Indicators describe a clear target population Necessity Indicators derived from robust evidence base and describe aspects of care necessary to Feasibility Indicators that are feasible to measure in general practice and do not detract from the normal working practices of the GDP

Methods – Round 1 • Panel Comprising GDPs, secondary care and Public Health consultants • First round: Online rating of 247 quality measures and 19 Dimensions of quality • Clarity and Necessity rated in online questionnaire 1 -9 scale. • Rating considering an average dentist providing care in an average practice to average patients in England. • Scores compiled and Median rating and Disagreement index calculated (Interpercentile range adjusted for symmetry) Rating 1 -3 4 -6 7 -9 Clarity Indicator was unclear and could not be developed Further development of clarity the indicator was required. Indicator was clear and needed little or no development Necessity Actions measured unnecessary for the provision of quality care Necessity of the action measured by indicator was unclear Actions measured are absolutely necessary in the provision of quality care Feasibility In current circumstances, the indicator was infeasible to measure Indicator may be feasible to introduce but would require changes to systems and processes. Measure could be easily implemented with little modification to systems and process Disagreement index >1 = Disagreement <1 = Agreement Median ≥ 7 and DI<1 = Agreement of clarity, necessity or feasibility

Methods - Round 2 Excerpt of Participant sheet – Frequency of other participants responses from first round Excerpt of moderator sheet – Frequency, median, disagreement index • Face to Face meeting • Each participant received a personalised rating sheet to re rate indicators • Their own 1 st round rating • Distribution of other participants rating • Moderator had moderation sheet • Median Score, • Disagreement index • Each participants first round response • The role of the moderator was to guide discussion and highlight points of disagreement from first round to aid the formation of consensus. • Rewording of indicators • Data collected into Excel • Median scores and disagreement index (IPRAS technique) was calculated.

Methods: Possible outcomes Median Score Disagreement index Interpretation 1 -3 <1 Consensus that measure is unclear, unnecessary or infeasible 4 -6 <1 Consensus that clarity, necessity or feasibility is uncertain 7 -9 <1 Consensus on clarity, necessity or feasibility Any score >1 Disagreement • For a measure to be included Agreement must be met on Clarity, necessity and Feasibility • I. e. : Median ≥ 7 and DI<1

Further Dimensions from literature Io. M Dimensions Results - Dimensions of Quality Dimension Equity Efficiency Timeliness Safety Effectiveness Patient Centeredness Caring function Appropriateness Relevance Optimality Legitimacy Acceptability Technical Quality Comprehensive Continuity of care Empathetic Responsive Tangibility Accessibility Clarity Median Disagreement Index 7 0. 11 7 0. 39 7 0. 24 9 0. 11 7 0. 24 5 0. 24 7 0. 11 2 0. 39 3 0. 58 2 0. 24 5 0. 24 8 0. 24 7 1. 12 7 0. 58 7 0. 39 8 0. 11 5 0. 24 7 0. 11 Agreement Clear Clear Uncertain Clear Unclear Uncertain Clear Disagreement Clear Uncertain Clear Median 8 9 8 8 7 8 5 6 4 6 8 8 8 7 7 Necessity Disagreement Index 0. 24 0. 11 0. 24 0 0. 39 0. 11 0. 24 0. 11 Agreement Necessary Necessary Uncertain Necessary Necessary Include? YES YES YES NO NO YES YES NO YES

Results – Quality measures Ratings of Indicators • Panel: 6 GDPs with experience ranging from 3 -34 yrs • • • 2 practice principals 4 associates 1 consultant in restorative dentistry 1 consultant in dental public health 1 regional dental dean • 260 quality indicators were rated in the second round • 21 rewordings offered by the panel at the face to face meeting. • 101 were indicators of structure, • 85 of process • 74 of outcomes. • 79 Indicators Rated Median ≥ 7 with DI <1 for Clarity, Necessity and Feasibility were included – • 45 Composite measures constructed though iterative discussion of the measures between the authors in order to avoid unnecessary overlap between the measures. Indicators rated as Unneccesary 29 45 107 79 Indicators Rated as Clear, Necessary and Feasible Indicators Rated as Clear, Necessary but with Uncertain Feasiblity

Indicators Dimensions of Quality Domains of Quality Safe Effective Patientcentred Timely Efficient Equitable Structure n=14 5 0 1 1 4 3 Process n=26 4 15 2 2 4 0 Outcome n=5 0 0 0 0

Composite Indicators Rated Clear, Necessary and Feasible • Structure n=14– Focus on safety and efficiency • Safety of buildings, accessibility, infection control • appropriate training of staff • Process n=26– Focus on Effectiveness • Correct prescription of prevention – Fluoride • Reflects clinical guidance – Detailed pocket charts with BPE 3 and 4 • Proper recording of risk assessments • Clinical records • Outcome n=5 – All regarding effectiveness • Failure of restorations • Adaptation of margins

Conclusions • The RAND/UCLA process is a feasible method for rating quality measures for PDC that may be repeated in a number of different contexts. • Consensus on the appropriateness of the dimensions of quality described by the Io. M definition was reached. The high rating of dimension of Technical quality suggests this may be required in a definition of quality for PDC. • Few measures of clinical outcomes emerged from this process. Most measures of outcome addressed restoration survival. Further measure development is required to populate a comprehensive measurement suite. • A number of process measures were rated as clear, necessary and feasible that may be derived from inspection of an electronic health records, suggesting potential for automatic quality measurement. • Many current indicators were deemed to be unnecessary