Measurement for improvement Workshop Whakakotahi Learning Session 1

  • Slides: 56
Download presentation
Measurement for improvement Workshop Whakakotahi Learning Session 1 Sue Wells A/Prof Quality Improvement University

Measurement for improvement Workshop Whakakotahi Learning Session 1 Sue Wells A/Prof Quality Improvement University of Auckland

Clinician-led cycle of care improvement (Prof Ian Scott)

Clinician-led cycle of care improvement (Prof Ian Scott)

Clinician-led cycle of care improvement (Prof Ian Scott) How will we know a change

Clinician-led cycle of care improvement (Prof Ian Scott) How will we know a change is an improvement? What are we trying to accomplish? How will we know a change is an improvement? What changes can we make?

Clinician-led cycle of care improvement (Prof Ian Scott)

Clinician-led cycle of care improvement (Prof Ian Scott)

Components of a system Structure Process Culture Outcome

Components of a system Structure Process Culture Outcome

Categorising measures Structure Process Outcome Balancing measures- check for unanticipated consequences Measures which allow

Categorising measures Structure Process Outcome Balancing measures- check for unanticipated consequences Measures which allow us to make inferences about the quality of health care Qualitative information Quantitative information

Why Measure? • To understand current situation • Move away from anecdote • To

Why Measure? • To understand current situation • Move away from anecdote • To manage by fact- need to quantify and verify possible causes (need evidence) • To baseline current performance • Get everyone on the same page • To develop solutions – need data to determine most effective approach • To show impact of changes • To prompt need for further improvement

Example Improving patient experience: A patient requiring long term medication has complained about the

Example Improving patient experience: A patient requiring long term medication has complained about the difficulty about getting a prescription filled on the same day. You ask… how easy it is for our patients to request and receive a repeat prescription?

Understand your current state • You need to measure “In God we trust. All

Understand your current state • You need to measure “In God we trust. All others bring data. ” Deming

Gather and review relevant data • How many scripts/week requested? • What day do

Gather and review relevant data • How many scripts/week requested? • What day do they generally come in? • Morning or afternoon? • What route do they come in? • • Phone Portal Walk-in Pharmacy

Gather feedback from patients and staff © NHS Institute for Innovation and Improvement 2011

Gather feedback from patients and staff © NHS Institute for Innovation and Improvement 2011

What are the major barriers and challenges to achieving same day service?

What are the major barriers and challenges to achieving same day service?

Map your current process for prescriptions © NHS Institute for Innovation and Improvement 2011

Map your current process for prescriptions © NHS Institute for Innovation and Improvement 2011

Check Sheet: Delays in getting prescriptions in last week Causes of delay in getting

Check Sheet: Delays in getting prescriptions in last week Causes of delay in getting scripts filled Problem Tally Subtotal Doctor busy 39 Nurse busy 4 Receptionist busy 5 Patient loses script 3 Script gets lost by pharmacist 3 Script request gets lost in practice 13

Bar Chart from Check Sheet 45 40 35 30 25 20 15 10 5

Bar Chart from Check Sheet 45 40 35 30 25 20 15 10 5 0 Doctor busy nurse busy receptionist busy patient loses scripts pharmacy loses script practice loses script request

Pareto Chart- prescription delays 45 100. 00%100% 95. 50% 91. 00% 40 85. 10%

Pareto Chart- prescription delays 45 100. 00%100% 95. 50% 91. 00% 40 85. 10% 35 80% 77. 60% 30 25 60% 58. 20% 20 40% 15 10 20% 5 0 Doctor busy practice loses script request receptionist busy Count nurse busy cum percent patient loses scripts pharmacy loses script 0%

Things to think about BEFORE you collect data 1) Your question/s- well defined 2)

Things to think about BEFORE you collect data 1) Your question/s- well defined 2) Stratification needed? separation of data according to strata or factors that might influence care patterns • Time of day or day of week • Season • Type of order (urgent vs routine) • Big or small practice • Type of worker • Patient factors eg; severity scores patient illness 3) Type of data 4) Sampling frame (e. g. PMS), sample size, sample strategy

Understanding types of data • Continuous data- measured on a ‘continuous scale’ • Discrete-

Understanding types of data • Continuous data- measured on a ‘continuous scale’ • Discrete- categories • Discrete- count data • Qualitative

Sampling in QI • NOT research…. . QI is simply asking “what is happening

Sampling in QI • NOT research…. . QI is simply asking “what is happening here? ” • no intention to generalise the results beyond the local setting • critical question is…. “does this sample represent the care/processes so that the team will accept results and act upon the findings? ”

Minimum sample sizes (rule of thumb) • Estimating mean • Standard deviation • Proportion

Minimum sample sizes (rule of thumb) • Estimating mean • Standard deviation • Proportion • Errors • Histogram/pareto chart • Scatter graph • Run/Control chart 5 -10 25 100 sample until 5 errors 50 25 25 -30 Thornley Group Lean Six Sigma Training 2010

Simple sampling strategies • Block sampling – straight sequence in a single time frame

Simple sampling strategies • Block sampling – straight sequence in a single time frame eg consecutive patients first 2 weeks of this month • Random sampling – simple, stratified Systematic or Purposive sampling- regular selection every 10 th patient, every hour on the hour or set time of day, day of week R Lloyd 2011

Variation variation should be viewed in one of two ways special or common cause.

Variation variation should be viewed in one of two ways special or common cause. • Common cause • Inherent part of every process • Random fluctuation • Stable, predictable or “in control”

Special Cause Variation • Indicates something has changed • Unstable or “out of control”

Special Cause Variation • Indicates something has changed • Unstable or “out of control” • “systematic” change or shift from the usual process • Maybe a purposeful change • Or indicates something is not right

Run Charts & Control Charts plotting measurements of process or outcomes over time useful

Run Charts & Control Charts plotting measurements of process or outcomes over time useful for understanding variation & demonstrating impact of interventions

Run charts • Allows a team to study data over specific period of time

Run charts • Allows a team to study data over specific period of time • Performance of the process/system • Used to detect shifts, trends or cycles • Measure performance before and after an intervention • X axis always time • Y-axis- whatever process/measure • Simple rules for interpretation

How do I plot/interpret a Run Chart? Median Brassard and Ritter, 1994

How do I plot/interpret a Run Chart? Median Brassard and Ritter, 1994

Run chart exercise

Run chart exercise

Prescription process • Plot % of patients who get same day prescriptions: Week Percent

Prescription process • Plot % of patients who get same day prescriptions: Week Percent Week Percent 1 2 3 4 5 6 7 8 9 60% 55% 70% 66% 58% 72% 62% 75% 10 11 12 13 14 15 16 17 18 71% 66% 58% 57% 64% 55% 64% 74%

Fill in the template and join the dots Percent of patients with same day

Fill in the template and join the dots Percent of patients with same day prescriptions 100% 90% 80% 70% 60% 50% 40% 1 2 3 4 5 6 7 8 9 10 week 11 12 13 14 15 16 17 18

Do you get something like this? Percent of patients with same day prescriptions 100%

Do you get something like this? Percent of patients with same day prescriptions 100% 90% 80% 70% 60% 50% 40% 1 2 3 4 5 6 7 8 9 10 week 11 12 13 14 15 16 17 18

Work out the median 1. Rewrite each data point (percentages) so that they are

Work out the median 1. Rewrite each data point (percentages) so that they are in rank order, going from highest to lowest (or lowest to highest) 2. Identify the median (middle) value (if an even number of values take the average of the 2 middle values)

Do you get something like this? Percent (in rank order, descending) 75 74 72

Do you get something like this? Percent (in rank order, descending) 75 74 72 71 70 66 66 66 64 64 62 61 60 58 58 57 55 55 Median

Draw the median line on the graph Percent of patients with same day prescriptions

Draw the median line on the graph Percent of patients with same day prescriptions 100% 90% 80% 70% 60% 50% 40% 1 2 3 4 5 6 7 8 9 10 week 11 12 13 14 15 16 17 18

Do you get something like this? Percent of patients with same day prescriptions 100%

Do you get something like this? Percent of patients with same day prescriptions 100% 90% 80% 70% 60% 50% 40% 1 2 3 4 5 6 7 8 9 10 week 11 12 13 14 15 16 17 18

Before you can apply the 4 rules • Count the number of “useful observations”

Before you can apply the 4 rules • Count the number of “useful observations” • Count the number of “runs”

Useful observations • All observations apart from any that fall on the median

Useful observations • All observations apart from any that fall on the median

How many useful observations? Percent of patients with same day prescriptions 100% 90% 80%

How many useful observations? Percent of patients with same day prescriptions 100% 90% 80% 70% 60% 50% 40% 1 2 3 4 5 6 7 8 9 10 week 11 12 13 14 15 16 17 18

How many useful observations did you get? Percent of patients with same day prescriptions

How many useful observations did you get? Percent of patients with same day prescriptions 100% 90% 80% 70% 60% 50% 40% 1 2 3 4 5 6 7 8 9 10 week 11 12 13 Two points on the line have same value as median Useful observations = 18 - 2 =16 14 15 16 17 18

Runs • >1 data points on same side of median • Exclude data on

Runs • >1 data points on same side of median • Exclude data on the median

How many runs? Percent of patients with same day prescriptions 100% 90% 80% 70%

How many runs? Percent of patients with same day prescriptions 100% 90% 80% 70% 60% 50% 40% 1 2 3 4 5 6 7 8 9 10 week 11 12 13 14 15 16 17 18

How many runs did you get? Percent of patients with same day prescriptions 100%

How many runs did you get? Percent of patients with same day prescriptions 100% 90% 80% 70% 60% 50% 40% 1 2 3 4 5 Number of runs= 10 6 7 8 9 10 week 11 12 13 14 15 16 17 18

Four run rules A special cause of variation may be signalled by meeting one

Four run rules A special cause of variation may be signalled by meeting one of more of the following 4 run rules: 1. Shift in the process (too many data points in a run: >6 consecutive points above or below median) 2. Trend (>5 consecutive points all increasing or decreasing) 3. Too many or too few runs (use table to work out) 4. An “astronomical” data point (the interocular test)

Examples

Examples

Are any of the run chart rules met for our example? Percent of patients

Are any of the run chart rules met for our example? Percent of patients with same day prescriptions 100% 90% 80% 70% 60% 50% 40% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 week Rule 1: Shift (>6 above/below median)? Rule 2: Trend (>5 all increasing or decreasing)? Rule 3: Too many or too few? (see table next slide – need to know number of useful observations [16] & number of runs [10]) Rule 4: Astronomical data point?

Rule 3: Too many or too few? Number of useful observations Minimum number of

Rule 3: Too many or too few? Number of useful observations Minimum number of runs Maximum number of runs 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 5 5 5 6 6 6 7 7 7 8 8 8 10 10 10 11 12 13 13 14 15 16 16 17 17 18 18 19 19 20 20 21

Clinician-led cycle of care improvement (Prof Ian Scott)

Clinician-led cycle of care improvement (Prof Ian Scott)

What was the impact of the changes? Useful obs 32 weeks- 3 points on

What was the impact of the changes? Useful obs 32 weeks- 3 points on median =29 useful observations percentage of patients with same day prescriptions 100 80 60 40 20 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

What was the impact of the changes? Useful obs 32 weeks- 3 points on

What was the impact of the changes? Useful obs 32 weeks- 3 points on median =29 10 + 2 = 12 runs percentage of patients with same day prescriptions 100 80 60 40 20 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Four run rules A special cause of variation may be signalled by meeting one

Four run rules A special cause of variation may be signalled by meeting one of more of the following 4 run rules: 1. Shift in the process (too many data points in a run: >6 consecutive points above or below median) 2. Trend (>5 consecutive points all increasing or decreasing) 3. Too many or too few runs (use table to work out) 4. An “astronomical” data point (the interocular test)

Rule 3: Too many or too few? Number of useful observations Minimum number of

Rule 3: Too many or too few? Number of useful observations Minimum number of runs Maximum number of runs 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 5 5 5 6 6 6 7 7 7 8 8 8 10 10 10 11 12 13 13 14 15 16 16 17 17 18 18 19 19 20 20 21

Control charts

Control charts

Control charts simplified • Run charts and control charts same purpose • Central line

Control charts simplified • Run charts and control charts same purpose • Central line mean (run chart median) • Limits (UCL and LCL) provide additional tests to identify special cause • More statistically robust • Better able to identify special cause variation (more sensitive) • Type of control chart depends on the type of data (many types vs only one type of run chart) • Control chart rules different

Control chart type determined by data type “Variables” (continuous) data • Quantitative data that

Control chart type determined by data type “Variables” (continuous) data • Quantitative data that can be measured • Infinite number of possible values depending on the precision of measurement • Does not have to be collected as a whole number • E. g. weight, height, BP, volume of workload “Attribute” (non-continuous) data Count (“non-conformities”, “defects”) • • • Occurrences only (don’t count those that don’t occur) Numerator only E. g. falls, medication errors, CLAB infections Classification (“nonconforming units”, “defectives”) • • Can count occurrences and non-occurrences Numerator and denominator E. g. % mortality, % readmitted, % c-sections

Which control chart should I use? Continuous? No Yes Classification? 1 observation per subgroup?

Which control chart should I use? Continuous? No Yes Classification? 1 observation per subgroup? (usually: is there 1 measurement per time period? ) No Yes I P Do you need to convert to a rate? No C Yes

Key points about measurement • Purpose: learning; x judgement • Be aware of the

Key points about measurement • Purpose: learning; x judgement • Be aware of the limitations of the measure • Balanced set: process, outcome, balance • Plot over time • Report regularly (has the process improved, stayed the same or become worse? ) • Measures should be: • linked to the team’s aim • Used to guide improvement and test changes • Integrated into the team’s daily routine • Focus on the vital few (Pareto principle / 80: 20 rule)