Measurement and analysis of provider behaviour change in

Measurement and analysis of provider behaviour change in knowledge translation research: Is there a best practice? 2 nd National Knowledge Translation Conference in Rehabilitation, April 19, 2018 Nancy Salbach, PT, Ph. D CIHR New Investigator in Knowledge Translation Associate Professor, Department of Physical Therapy, University of Toronto

Presentation Objectives: 1. To consider the advantages and limitations of methods for measuring provider behaviour change following a KT intervention 2. To understand which quantitative approaches to measuring provider behaviour change may lead to clustered data and how to evaluate clustering in data analysis 2

Outcomes of KT studies in Rehabilitation Systematic review: Jones et al 2015 3

More than 75% of KT intervention studies* in rehab target professional or process outcomes Outcome Patient function Professional/process Economic N (%) 5 (19) 20 (77) 1 (4) *provider targeted: PT (13 studies); OT (7 studies); SLP (3 studies); OT&PT (3 studies) Systematic review: Jones et al 2015 (n=26 studies) 4

Why professional or process outcomes? • Ultimate aim = improved health of Canadians = improved patient function in rehabilitation setting • KT interventions targeting multidisciplinary team • • Patient function = primary outcome Provider practice = intermediate ‘process’ outcome • KT interventions targeting uniprofessional practices • Provider practice = primary outcome as patient function influenced by practices of “other” providers 5

Most professional-level outcomes measured by provider self-report Measurement of PT behaviour in 8 studies N (%) Provider self-report Patient self-report Provider interviews/focus groups Observation Health record review Technology-based usage 7 (88) 2 (25) 0 0 0 Systematic review: Jones et al 2015 6

Warning! Brain squeeze https: //imgflip. com/meme/58388563/ Thinking-hard What are the pros and cons of different methods for measuring behaviour change in rehabilitation providers following KT interventions? 7

Provider Self-report Pros • Feasible • Inexpensive if online • Data on large group Cons • Measurement at one point in time • Attrition over time • Social desirability bias • Item reliability? 8

Patient Self-report Pros Cons • Inexpensive if online • Privacy - Difficulty obtaining patient emails • Can capture practice • Recall over time • Expense - phone/mail survey • Low response rate • Is response linked to health? 9

Interviews & Focus Groups Pros Cons • Deep understanding • Interviews=resource of mechanisms of intensive behaviour change • Focus groups: scheduling and opportunity to speak • Limited for large groups 10

Observing Provider Practice Pros • Observer unbiased • Range of patients https: //www. smartcherry. in/your. Cons observation-power-smart-cherry/ • Expensive, resource intensive • Representativeness depends on duration of observation • Observation may influence practice 11

Health record review Pros • Retrospective: Less vulnerable to bias • Range of patients Cons • Inconsistent documentation • Expensive, resource intensive • If prospective, providers can change documentation behaviour http: //www. bcm j. org/mdsbe/merit-andagonyretrospectivechart-reviewsmedicalstudent%E 2%8 0%99 sperspective 12

Now a word about clustering • Clustering is when people or data in your study form natural groupings because the people or data within a group are more related to each other than they are to people or data outside the group • Can you think of examples of clustering in rehabilitation? 13

Clustering of Providers within Hospitals Eg. Guideline implementation study, examine provider use of recommendations in 3 rehab hospitals Hospital 1 Hospital 2 Hospital 3 Clusters 14

Clustering of Patients within Providers Eg. Guideline implementation study, examine provider use of recommendations with each patient Provider 1 Provider 2 Provider 3 Clusters Patients 15

Clustering of Measurements within Patients Eg. Guideline implementation study, examine provider use of recommendations across sessions with same patients Patient 1 Patient 2 Patient 3 Clusters Data points 16

Multi-level Clustering Cluster Level Hospital Provider Patients Data points Hospital 2 Hospital 3

What happens when you don’t account for clustering? • It depends on whether clustering has an influence • If clustering influences your study outcome, and you don’t account for clustering in the analysis, you will likely get the wrong answer • Let’s take a look at which level of clustering was important in a guideline implementation trial 18

Stroke Canada Optimization of Rehabilitation through Evidence-Implementation Trial (SCORE-IT) • National cluster randomized guideline implementation trial • 20 stroke inpatient rehabilitation centres randomized to a multi-component facilitated or passive KT intervention for guideline implementation • Primary outcome was patient function: Effect on walking capacity (6 -minute walk test) but no effect on UE function (Box & Block test) observed (n=777) • Examining implementation of recommendations was planned to help explain results related to patient outcomes 19

Hypothesis • The rate of implementing 18 selected treatment recommendations would improve from pre- to postintervention to a greater extent in the active, facilitated KT intervention group than in the passive KT intervention group 20

Measurement of Implementation of 18 SCORE Treatment Recommendations • Self-reported implementation (yes/no) using checklists • PTs, OTs, nurses asked to complete after every patient seen in a 2 -week period, pre- & post-intervention • Intervention period: 16 months 21

Warning – This is a test! In the SCORE-IT study… Present Level of Clustering Clusters of providers within hospitals? ü Clusters of patients within providers? ü Clusters of measurements within ü patients? 22

Analysis plan for each of 18 outcomes • Logistic Regression with random effects modeling (can account for clustering) • Tested for clustering effect at site, provider and patient levels • Estimated the rate of adherence to each treatment recommendation pre- and post-intervention • Compared the change from pre- to post-intervention between groups using an interaction term: group x time • Analysis conducted with and without adjustment for clustering and covariates (eg patient motor function) 23

Significant clustering effects at Provider (67%) and Patient (39%) levels only Treatment 1. Sit-to-stand 2. LE ROM and/or stretching 3. Use of LE external support 4. Task-specific training (i. e. , stairs) 5. Training for sitting balance 6. Training for standing balance 7. FES for the LE 8. Walking practice 9. Treadmill walking practice 10. UE ROM and/or stretching 11. Interventions to prevent shoulder pain 12. Task-specific training (i. e. , self-care tasks) 13. Techniques to reduce hand edema 14. Ice/heat or soft tissue massage for shoulder 15. FES for wrist/ arm/shoulder 16. Educate patient or caregiver on how to handle arm or shoulder 17. UE constraint-induced therapy 18. Visual imagery to enhance arm recovery Site Provider Yes Yes Yes Patient Yes Yes Yes 24 Yes

Between-group analysis – UE treatments: No effect before/after adjustment for clustering Treatment Effect (Change. E-Change. C) % (95% CI) Effect Regression Estimate (p-value) Unadjusted Adjusted for Clustering and Covariates UE ROM/Stretching 5. 2 (-4. 4, 14. 7) 0. 1 (0. 876) 0 (0. 968) Interventions to prevent shoulder pain 4. 9 (-5. 1, 14. 9) 0. 8 (0. 343) 0. 8 (0. 269) Task-specific UE training 6. 0 (-5. 4, 17. 3) 0. 5 (0. 237) Techniques to reduce hand edema 0 (-9. 6, 8. 8) -0. 1 (0. 734) -0. 1 (0. 759) Shoulder ice/heat/massage 4. 8 (-2. 3, 11. 8) 0. 8 (0. 487) 0. 9 (0. 432) FES for UE 0. 5 (-4. 1, 5. 1) DNC 1. 7 (0. 382) Educate patient/caregiver 3. 4 (-3. 9, 10. 6) 0. 5 (0. 718) 0. 2 (0. 916) Constraint-induced therapy 1. 8 (-5. 6, 9. 2) -0. 8 (0. 596) -0. 9 (0. 606) Visual imagery 3. 5 (-3. 7, 10. 8) 0. 4 (0. 381) 0. 5 (0. 338) 25

Between-group analysis – LE-Posture-Mobility Treatments: 2 of 5 effects no longer significant after adjustment for clustering Treatment Effect (Change. E-Change. C) % (95% CI) Effect Regression Estimate (p-value) Unadjusted Adjusted for Clustering and Covariates Sit to stand 21. 4 (10. 4, 32. 4) 1. 1 (0. 035) 1. 0 (0. 028) Walking practice 22. 0 (11. 2, 32. 8) 1. 0 (0. 029) 1. 0 (0. 043) LE ROM/Stretching -14. 7 (-26. 0, -3. 5) -1. 1 (0. 093) -1. 1 (0. 106) Training for sitting balance -14. 6 (-28. 2, -1. 0) -0. 8 (0. 125) -0. 9 (0. 123) Training for standing balance 26 -22. 8 (-39. 4, -6. 2) -1. 0 (0. 040) -1. 1 (0. 037)

Take home messages • Self-report methods to evaluate provider behaviour are common in KT intervention studies • Many methods to evaluate provider behaviour: • Consider whether method captures practice in a range patients over time, vulnerability to bias, feasibility, and cost • Understand how limitations influence interpretation of results • Plan to adjust for important covariates (patient, provider, hospital) based on theoretical or empirical evidence • Qualitative and quantitative research helps us understand what influences provider practice 27 Campbell et al 2012; Campbell et al 2007

Take home messages • Identify levels of clustering in design phase; review reporting guidelines for cluster RCTs • In context of stroke rehabilitation guideline adoption: • Effect of patients clustered within provider has strongest impact on estimated use of best practices, followed by data clustered within patients • Hospital site does not appear influential • Consult with a statistician re: impact of clustering and covariate adjustment on sample size estimation; plan to evaluate clustering in analysis 28

SCORE-IT Research Team Mark Bayley, MD, Toronto Rehabilitation Institute. University Health Network Sharon Wood-Dauphinee, Ph. D, Mc. Gill University Susan Barreca, PT, MSc Sandra Black, MD, Sunnybrook Health Sciences Centre Lucie Brosseau, Ph. D, University of Ottawa Johanne Desrosiers, Ph. D, Université de Sherbrooke Janice Eng, Ph. D, University of British Columbia Ian Graham, Ph. D, University of Ottawa Jeremy Grimshaw, Ph. D, University of Ottawa Margaret Harrison, Ph. D, Queen’s University Jeffery Hoch, Ph. D, University of Toronto Susan Jaglal, Ph. D, University of Toronto Nicol Korner-Bitensky, Ph. D, Mc. Gill University Marilyn Mac. Kay-Lyons, Ph. D, Dalhousie University Nancy Mayo, Ph. D, Mc. Gill University Carol Richards, Ph. D, Laval University Monica Taljaard, Ph. D, Ottawa Health Research Institute Robert Teasell, MD, Western University Merrick Zwarenstein, MBBCh, Ph. D, Western University SCORE-IT Coordinators: Amanda Hurdowar Jennifer Mokry Olga Yaroslavtseva Marion Hau 29

SCORE-IT Acknowledgements Funding Participants We would like to acknowledge the hospitals and thank the many healthcare professionals and people with stroke who participated

Thank you! Nancy Salbach, PT, Ph. D nancy. salbach@utoronto. ca @nancysalbach Knowledge to Action Lab: http: //www. physicaltherapy. utoronto. ca/research/ktalab/

32

CONSORT Diagram 33

Clustering Effects Treatment 1. Sit-to-stand 2. LE ROM and/or stretching 3. Use of LE external support 4. Task-specific training (i. e. , stairs) 5. Training for sitting balance 6. Training for standing balance 7. FES for the LE 8. Walking practice 9. Treadmill walking practice 10. UE ROM and/or stretching 11. Interventions to prevent shoulder pain 12. Task-specific training (i. e. , self-care tasks) 13. Techniques to reduce hand edema 14. Ice/heat or soft tissue massage for shoulder 15. FES for wrist/ arm/shoulder 16. Educate patient or caregiver on how to handle arm or shoulder 17. UE constraint-induced therapy 18. Visual imagery to enhance arm recovery Intracluster Correlation Coefficient Site Provider Patient 0. 06 0. 26* 0. 10* 0. 00 0. 30* 0. 04 0. 11 0. 10* 0. 23* 0. 00 0. 35* 0. 00 0. 01 0. 16* 0. 09 0. 02 0. 23* 0. 08 0. 12 0. 00 0. 02 0. 31* 0. 11* 0. 03 0. 19 0. 17 0. 08 0. 32* 0. 06* 0. 08 0. 16* 0. 21* 0. 05 0. 28* 0. 06 0. 22* 0. 14 0. 24 0. 16 0. 20 0. 04 0. 05 0. 11 0. 00 0. 25* 0. 19 0. 31* 0. 07 0. 14 0. 04 *Significant clustering effect at Provider (67%) and Patient (39%) levels only.

Cluster sizes at the site, provider and patient level by study group and sampling time point Cluster Level Cluster Size Mean (SD) 7 Treatments Implemented by 11 Treatments Implemented by RNs, OTs, and PTs OTs and PTs Facilitated Passive Pre Post Site: Providers 15 (8) 13 (6) 10 (7) 13 (7) 6 (3)* per site Provider: Patients per 3 (3) 2 (1) provider Patient: Forms per 13 (13) 7 (3) 6 (4) 6 (3) 4 (2) patient 6 (3)* 5 (4)* 7 (4)* 2 (1) 4 (2) 3 (1) *Smaller provider clusters per site for treatments that fewer health professional groups were asked to implement. 35

Baseline Implementation of Recommended Treatments for LE/Mobility Function (unadjusted) Treatment % of Times Implemented (95% CI) n Facilitated KT n Level Passive KT Sit to stand 647 20. 4 (17. 3, 23. 5) 193 36. 3 (29. 5, 43. 1) >15% LE ROM/Stretching 151 15. 9 (10. 1, 21. 7) 118 8. 5 (3. 4, 13. 5) >15% Use of external LE support 647 193 15. 0 (10. 0, 20. 1) <15% Task-specific LE training 151 31. 8 (24. 3, 39. 2) 118 26. 3 (18. 3, 34. 2) >15% Training for sitting balance 151 23. 8 (17. 0, 30. 7) 118 17. 0 (10. 2, 23. 7) >15% Training for standing balance 151 51. 7 (43. 7, 59. 6) 118 36. 4 (27. 7, 45. 1) >15% FES for LE 151 0 (0, 0) <15% Walking practice 647 31. 6 (25. 0, 38. 2) >15% Treadmill walking 151 6. 8 (2. 2, 11. 3) <15% 7. 3 (5. 3, 9. 3) 0. 7 (-0. 6, 2. 0) 118 15. 9 (13. 1, 18. 7) 193 2. 7 (0. 1, 5. 2) 118 36

Within-Group Unadjusted Change: Improved (green) and reduced (red) for LE/Mobility Treatment No. forms Pre to Post Change in % of Times Implemented (95% CI) Facilitated KT Passive KT Sit to stand 1381 18. 7 (12. 2, 25. 3) -2. 7 (-11. 5, 6. 2) LE ROM/Stretching 547 -5. 4 (-13. 1, 2. 3) 9. 3 (1. 1, 17. 5) Use of external LE support 1381 1. 4 (-2. 5, 5. 3) 2. 3 (-4. 5, 9. 1) Task-specific LE training 547 6. 7 (-4. 3, 17. 6) 11. 5 (0. 1, 22. 9) Training for sitting balance 547 -6. 4 (-15. 6, 2. 9) 8. 2 (-1. 8, 18. 2) Training for standing balance 547 0. 8 (-10. 7, 12. 2) 23. 6 (11. 6, 35. 6) FES for LE 547 0 (-1. 9, 1. 9) 0. 7 (-0. 7, 2. 2) Walking practice 1381 23. 2 (16. 8, 29. 6) 1. 2 (-7. 4, 9. 9) Treadmill walking 547 -1. 3 (-4. 5, 2. 0) -1. 6 (-7. 5, 4. 3) 37

Baseline Implementation of Recommended Treatments for UE Function (unadjusted) Treatment % of Times Implemented (95% CI) n Facilitated KT n Level Passive KT UE ROM/Stretching 647 12. 7 (10. 1, 15. 2) 193 21. 8 (15. 9, 27. 6) >15% Interventions to prevent shoulder pain 647 25. 0 (21. 7, 28. 4) 193 25. 4 (19. 2, 31. 5) >15% Task-specific UE training 647 28. 9 (25. 4, 32. 4) 193 37. 3 (30. 5, 44. 1) >15% Techniques to reduce hand edema 151 7. 3 (3. 1, 11. 4) 118 10. 2 (4. 7, 15. 6) <15% Ice/heat or soft tissue massage for shoulder 151 1. 3 (-0. 5, 3. 2) 118 8. 5 (3. 4, 13. 5) <15% FES for UE 151 2. 0 (-0. 2, 4. 2) 118 2. 5 (-0. 3, 5. 4) <15% Educate patient/caregiver how to handle UE 647 8. 8 (6. 6, 11. 0) 193 13. 0 (8. 2, 17. 7) <15% Constraint-induced therapy 151 4. 6 (1. 3, 8. 0) 118 10. 2 (4. 7, 15. 6) <15% Visual imagery 151 2. 7 (0. 1, 5. 2) 118 5. 1 (1. 1, 9. 1) <15% 38

Within-Group Unadjusted Change: Improved (green) and reduced (red) implementation for UE Treatment No. forms UE ROM/Stretching Interventions to prevent shoulder pain 1381 Pre to Post Change in % of Times Implemented (95% CI) Facilitated KT Passive KT 8. 7 (3. 2, 14. 2) 3. 5 (-4. 3, 11. 4) 1381 0. 7 (-5. 5, 6. 8) -4. 3 (-12. 1, 3. 6) Task-specific UE training 1381 6. 1 (-3. 0, 15. 2) Techniques to reduce hand edema 547 12. 0 (5. 3, 18. 8) -1. 7 (-7. 3, 3. 9) Ice/heat or soft tissue massage for shoulder 547 1. 5 (-1. 8, 4. 7) FES for UE 547 -0. 6 (-3. 5, 2. 4) -1. 1 (-4. 6, 2. 4) Educate patient/caregiver how to handle UE 1381 0. 6 (-3. 5, 4. 7) -2. 8 (-8. 8, 3. 2) -1. 3 (-8. 6, 6. 0) -3. 3 (-9. 6, 3. 0) Constraint-induced therapy 547 -3. 9 (-7. 6, -0. 3) -5. 7 (-12. 2, 0. 8) Visual imagery 547 3. 6 (-1. 1, 8. 4) 0. 1 (-5. 4, 5. 6) 39