The Interrater Reliability and Intrarater Reliability of Bedside

The Inter-rater Reliability and Intra-rater Reliability of Bedside Ultrasounds of the Femoral Muscle Thickness Daren K. Heyland, MD, MSc, FRCPC Professor of Medicine Queen’s University, Kingston General Hospital Kingston, Ontario

A Randomized Trial of Supplemental Parenteral Nutrition in Under and Over Weight Critically Ill Patients: The TOP UP Trial Hypothesis: Increased energy and protein delivery to underweight and overweight critically ill patients (Body Mass Index [BMI] <25 or >35) will result in improved 60 day survival compared to usual care. • Multicenter pilot study • Randomized trial of 160 critically ill adult patients from 8 tertiary care ICU’s in Canada, United States, and Europe. • Patients randomized to one of 2 interventions: enteral nutrition (EN) alone or enteral nutrition plus parenteral nutrition (supplemental PN group). • Patients stratified on the basis of admission BMI: <25 or >35, medical or surgical admission diagnosis, by site and if EN was administered between ICU admission and randomization Primary outcome: 60 day mortality. Secondary outcomes: 28 day mortality, hospital mortality, duration of stay (ICU and hospital), multiple organ dysfunction (SOFA and PODS), duration of mechanical ventilation, development of ICU acquired infections, functional status at hospital discharge, and 3 and 6 month survival and health-related quality of life.

Weekly Ultrasounds • There is emerging evidence that muscle mass and muscle function predict morbidity in surviving patients and that muscle mass at ICU admission may predict length of hospital stay. • We propose to evaluate the effect of differential amounts of protein and energy provided to study patients on muscle mass and function. • We can postulate that the beneficial effect of enhanced energy and protein provision is mediated by the preservation (or attenuated deterioration) of muscle mass and increased function in these better fed patients, which would ultimately result in positive outcomes. • We will evaluate muscle mass in all study patients using non-invasive bedside ultrasound of the femoral muscle

Testing Ultrasound Reliability before TOP-UP Before we performed weekly ultrasounds on the study population, we conducted a trial of the ultrasound protocol to allow us to: • standardize the training of all Study Investigators performing the ultrasound assessment • test the feasibility of the procedures • determine ‘normal’ values to which we can compare our measures in the study population • determine intra-rater (trainer) and the inter-rater (trainee) reliability. The ultrasound reliability trial involved: • the 7 participating TOP UP sites • healthy volunteers • a standardized protocol: o ultrasound trainers were to perform ultrasounds twice on each patient o a trainee repeated the measurement on the same patient The objective was to evaluate the inter-rater reliability and intra-rater reliability of bedside ultrasounds of the femoral muscle measuring muscle thickness in healthy volunteers before using this tool in detailing overall muscle mass in ICU patients.

Reliability Protocol 1) Trainer to complete a 2/3 rd and midpoint ultrasound on each leg 2) Trainer to repeat a 2/3 rd and midpoint ultrasound on each leg (intra-rater reliability test) 3) Trainee to complete a 2/3 rd and midpoint ultrasound on each leg (inter-rater reliability test) The overall muscle thickness was calculated as the average of the readings measured at the border between the lower third and upper two-thirds between Anterior Superior Iliac Spine (ASIS) and upper pole of the patella as well as the reading at the midpoint between the ASIS and the upper pole of the patella averaged over the right and left legs.

Volunteers’ Demographics Variable N Mean Std Dev Minimum Maximum Age 64 30. 6 8. 4 21. 0 55. 0 Height 78 171. 4 11. 5 135. 0 196. 0 Weight 78 71. 2 16. 4 46. 0 136. 4 BMI 78 24. 1 4. 4 16. 9 40. 7

Intra-rater Reliability Results Site Erasme University Hospital, Brussels, Belgium Subjects Between Subject Variance Within Subject Variance 10 0. 2307 0. 01380 NA ICC 0. 94 Grey Nuns Hospital, Edmonton, Alberta, Canada 0 NA Royal Alexandra Hospital, Edmonton, Alberta, Canada 10 0. 2425 0. 001018 >0. 99 Nouvel Hôpital Civil, Strasbourg, France 4 0. 2567 0. 000199 >0. 99 University of Alberta Hospital Edmonton, Alberta, Canada 12 0. 05866 University of Colorado, Aurora, Colorado, USA 5 0. 2869 0. 000613 >0. 99 University of Texas Health Science Center, Houston, Texas, USA 5 0. 1774 0. 003837 0. 98 Pooled 46 0. 2648 0. 004554 0. 98 0. 003180 NA 0. 95 ICC: Intra-class correlation coefficient, ICC = between subject variance / (between subject variance +within subject variance)

Intra-rater Reliability Results (Continued) • The paired t-test was used examine the average difference between the first and second trainer measurements. • The paired profile plot visualizes the difference in the paired measures. Mean difference between trainer measurement 1 and trainer measurement 2 Trainer measurement 1 Trainer measurement 2 The mean difference between trainer measurements (95% CI) = 0. 037 (0. 010 to 0. 063) p=0. 0077

Inter-rater Reliability Results Site Erasme University Hospital, Brussels, Belgium Subjects Between Subject Variance Within Subject Variance 10 0. 2194 0. 02900 ICC 0. 88 Grey Nuns Hospital, Edmonton, Alberta, Canada 13 0. 3217 0. 000769 >0. 99 Royal Alexandra Hospital, Edmonton, Alberta, Canada 10 0. 2305 0. 02072 0. 92 Nouvel Hôpital Civil, Strasbourg, France 18 0. 2623 0. 009972 0. 96 University of Alberta Hospital Edmonton, Alberta, Canada 12 0. 03587 0. 01360 0. 73 University of Colorado, Aurora, Colorado, USA 5 0. 1714 0. 02746 0. 86 University of Texas Health Science Center, Houston, Texas, USA 5 0. 1704 0. 03334 0. 84 Pooled 73 0. 2584 0. 01580 0. 94 ICC: Intra-class correlation coefficient, ICC = between subject variance / (between subject variance +within subject variance)

Inter-rater Reliability Results (Continued) • The paired t-test was used examine the average difference between the first trainer and the trainee measurement. • Paired profile plots are provided to visualize the difference in the paired measures. Mean difference between trainer measurement and trainee measurement Trainer Trainee There was a small but statistically significant difference between the trainer and trainee results: Mean (95% CI) = -0. 061 cm (-0. 100 to -0. 022), p= 0. 0028

Conclusion • There is excellent inter and intra-rater reliability for ultrasound measurements of the femoral muscle to determine overall muscle thickness in healthy volunteers. • A sample of ‘normal’ values is now available to compare measures from a study population. • Further evaluation of this technique must be validated in critically ill patients. • Efforts to link the ultrasound measurements to ICU outcomes should be undertaken.

Questions