Singlecase designs Methodology and data analysis Analyses via

Single-case designs: Methodology and data analysis Analyses via web-based applications

Outline v. Visual analysis: 6 aspects [15] o Level o Trend o Overlap o Variability o Immediacy of effects o Consistency of data patterns

Outline v. Visual aids [18] o Trend stability envelope o Standard deviation bands o Projecting split-middle trend with limits o Conservative Dual Criterion

Outline v. Mean difference: [24] o Within-case standardized mean difference o Log Response Ratio o Percentage change (all data or last three per phase) o (Percentage Zero Data) o Slope and Level Change o Mean Phase Difference (projected vs. actual)

Outline v. Nonoverlap indices: [37] o Percentage of Nonoverlapping Data o Percentage of data points Exceeding the Median o Nonoverlap of All Pairs o Improvement Rate Difference Ø Tau: without and with trend correction Ø Baseline corrected Tau: conditional correction Ø PEM-Trend Ø PNCorrected. D

Outline o Regression analyses (taking trend into account; descriptive function only in the present document) [51] o Overall: Ordinary least squares o Overall + taking autocorrelation into account: Generalized least squares o Immediate effect + change in trend: Piecewise regression

Outline o Several comparisons beyond AB [56] o Slope and level change o Mean phase difference o Two-level models: (HLM / Multilevel / Mixed effects) o Between-cases standardized mean difference § Multiple-baseline design § Reversal design (e. g. , ABAB) replicated

Outline o Alternating treatments design [80] o Mean difference o Quantifications of variability o Nonoverlap of All Pairs o PND according to Wolery et al. (2010) o Average Difference between Successive Observations o Comparing Actual and Linearly Interpolated Values

Outline: R-Cmdr plug-in v. Randomization tests [97] üUsing R, R-Commander, plug-ins o AB design o Multiple-baseline design o Alternating treatments design

Example: Baldwin & Powell (2015) Protocol design • “Multiple baseline” across behaviors (not staggered) preferred over ABAB due to ethical reasons • Baseline (A) compared with Google Calendar text alerts delivered to a mobile phone as a memory aid • 1 patient with traumatic brain injury + severe problems in memory and executive functioning • Outcome: number of events forgotten + a subjective measure (Everyday Memory Questionnaire) • Original analysis: Nonoverlap of all pairs

Example: Baldwin & Powell (2015) A or B? Target behaviors Control behaviors

Web-based Visual aids: Loading & summarizing the data https: //manolov. shinyapps. io/Overlap/

Web-based Visual aids: WWC Standards https: //manolov. shinyapps. io/Overlap/

Web-based Visual aids: WWC Standards consistency Meaningfulness of the combination? Data loss due to different metric https: //manolov. shinyapps. io/Overlap/

Web-based Visual aids: Trend stability envelope https: //manolov. shinyapps. io/Overlap/

Web-based Visual aids: Standard deviation bands https: //manolov. shinyapps. io/Overlap/

Web-based Visual aids: Split-middle trend & Md-envelope https: //manolov. shinyapps. io/Overlap/

Web-based Visual aids: Split-middle trend & IQR-interval https: //manolov. shinyapps. io/Overlap/

Web-based Visual aids: (Conservative) Dual Criterion https: //manolov. shinyapps. io/Overlap/

Web-based Within-case SMD (/SA) https: //jepusto. shinyapps. io/SCD-effect-sizes/

Web-based Within-case SMD (/Spooled) https: //jepusto. shinyapps. io/SCD-effect-sizes/

Web-based Log response ratio https: //jepusto. shinyapps. io/SCD-effect-sizes/

Web-based Percentage change: Loading & summarizing the data https: //manolov. shinyapps. io/Change/

Web-based Percentage change: Percentage change index https: //manolov. shinyapps. io/Change/

Web-based Percentage change: Percentage zero data https: //manolov. shinyapps. io/Change/

Web-based Slope & Level Change http: //manolov. shinyapps. io/Change/

Web-based Slope & Level Change http: //manolov. shinyapps. io/Change/

Web-based Mean phase difference Unlikely projections of improving trend: understimate effect http: //manolov. shinyapps. io/Change/ Unlikely projections of deteriorating trend: overstimate effect

Web-based Nonoverlap indices : Percentage of nonoverlapping data https: //jepusto. shinyapps. io/SCD-effect-sizes/

Web-based Nonoverlap indices : Percentage of nonoverlapping data http: //ktarlow. com/stats/pnd/

Web-based Nonoverlap indices : % data points exceeding the Md https: //jepusto. shinyapps. io/SCD-effect-sizes/

Web-based Nonoverlap indices : Nonoverlap of all pairs https: //jepusto. shinyapps. io/SCD-effect-sizes/

Web-based Nonoverlap indices : Improvement rate difference https: //jepusto. shinyapps. io/SCD-effect-sizes/

Web-based Nonoverlap indices : Tau without trend correction https: //jepusto. shinyapps. io/SCD-effect-sizes/

Web-based Nonoverlap indices : Tau-U with trend correction https: //jepusto. shinyapps. io/SCD-effect-sizes/

Web-based Nonoverlap indices : Baseline corrected Tau http: //ktarlow. com/stats/tau/

Web-based Nonoverlap indices: Loading & summarizing the data https: //manolov. shinyapps. io/Overlap/

Web-based Nonoverlap indices: Baseline corrected Tau https: //manolov. shinyapps. io/Overlap/

Web-based Nonoverlap indices: % data points exceeding median trend https: //manolov. shinyapps. io/Overlap/

Web-based Nonoverlap indices: PNCorrected. D https: //manolov. shinyapps. io/Overlap/

https: //manolov. shinyapps. io/Regression/ Web-based Regression-based ES: Loading & summarizing the data

https: //manolov. shinyapps. io/Regression/ Web-based Regression-based ES: Ordinary least squares

https: //manolov. shinyapps. io/Regression/ Web-based Regression-based ES: Generalized least squares (“directly”)

https: //manolov. shinyapps. io/Regression/ Web-based Regression-based ES: Piecewise regression

Example: Baldwin & Powell (2014)

http: //manolov. shinyapps. io/Several. AB/ Web-based SLC & MPD for multiple baseline: Loading & summarizing the data

http: //manolov. shinyapps. io/Several. AB/ Web-based SLC for multiple baseline: Graphing the data Data loss due to different metric

http: //manolov. shinyapps. io/Several. AB/ Web-based SLC for multiple baseline: Graphing the differences

http: //manolov. shinyapps. io/Several. AB/ Web-based SLC for multiple baseline: Numerical results

http: //manolov. shinyapps. io/Several. AB/ Web-based MPD for multiple baseline: Graphing the differences

http: //manolov. shinyapps. io/Several. AB/ Web-based MPD for multiple baseline: Numerical results

http: //manolov. shinyapps. io/Several. AB/ Web-based Two-level HLM: Loading & summarizing the data

http: //manolov. shinyapps. io/Several. AB/ Web-based Two-level HLM: Graphical results Data loss due to different metric ? Theoretically this should have been a multiple baseline across participants

http: //manolov. shinyapps. io/Several. AB/ Web-based Two-level HLM: Numerical results

Web-based Between-cases SMD https: //jepusto. shinyapps. io/scdhlm/

Web-based Between-cases SMD Data loss due to different metric https: //jepusto. shinyapps. io/scdhlm/

Theoretically this should have been a multiple baseline across participants Large intra-class correlation: the between-behaviors (ideally, cases) variability is large inconsistencies a way of detecting whether the intergration is meaningful Web-based Between-cases SMD https: //jepusto. shinyapps. io/scdhlm/

Example: Coker et al. (2009) Protocol design • ABAB design • Intervention: m. CIMT (1 hour /day) • Participant: One child aged 5 months in the beginning; 9 months at B 1; 18 months at follow up. Maturation a factor? • Outcome measures: video analysis of affected limb use for: – Reaching an object: reported – Stabilizing weight: reported – Approaching midline: reported – Sensory exploration: not reported – Bimanual association: not reported

Example: Coker et al. (2009) ØWhat did the authors do? ØReported graphically observed affected limb use from videotapes: main SCED outcome ØNo visual aids provided. No statistics. ØSimple description of graphs in results section. ØAdditional outcome: Peabody developmental motor scale: percentile & age normative groups - comparison Ø Is it appropriate? Yes, but insufficient. ØOnly 2 A 1 points, due to “no improvement” (ethics, clinical) ØGreat variability in each phase Difficult to conclude whether the intervention is effective

Example: Coker et al. (2009) Ø What are we doing? Apply the d-statistic by Hedges, Pustejovsky, and Shadish (2012). Ø Why? Obtain a summary quantification; SCEDspecific; applicable to ABAB designs replicated across participants or behaviors; get an overall summary, given that visual inspection is unclear. Ø What more can be done? Not much, too short baseline data not meeting current Standards. Maybe, nonoverlap indices. Theoretically this should have been a replication across participants!!! Once again: applying statistics requires thinking!!!

https: //jepusto. shinyapps. io/scdhlm/ Web-based Between-cases SMD

Web-based Between-cases SMD https: //jepusto. shinyapps. io/scdhlm/

Web-based Between-cases SMD https: //jepusto. shinyapps. io/scdhlm/

Example: Kirsch et al. (2004) Protocol design • ABA design (Study 1); Alternating treatments design (Study 2) • Intervention: Assistive technology for cognition • Participants: Study 1: 19 -year-old man, with topographical disorientation after traumatic brain injury (TBI). Study 2: 71 -year -old woman with cognitive declines associated with TBI and a pre-injury history of chronic ischemic changes. • Outcome measures: recorded – Navigation task: average number of errors per route (Study 1) – Setting an alarm clock: Average number of errors per task substep (Study 2) – Setting an alarm clock: number of substeps attempted (Study 2)

Example: Kirsch et al. (2004) Specific characteristics of the Alternating treatments design • Referred to as “modified ABAB” by Kirsch et al. (2004) • «One or two trials were conducted each day, depending on the participant’s availability. However, time of day and order of conditions within and across days were counterbalanced. » • Deemed not optimal due to learning MBL better • Other possibilities for designs in which frequent alternation of conditions is possible: – Completely randomized sequence of conditions (limiting n. A=n. B) – Randomized block design: for each pair of measurement occasions, decide at random whether A or B is taking place – «ATD» : random sequence of conditions with a restriction of a maximum of two consecutive administrations of the same conditions.

Web-based ATD data analysis: PND as a quantification of superiority learning anxiety

Web-based ATD data analysis: Basic quantifications https: //manolov. shinyapps. io/ATDesign/

Web-based ATD data analysis: PND as a quantification of superiority Same day: meaningful comparison https: //manolov. shinyapps. io/ATDesign/

Web-based ATD data analysis: Regression analyses https: //manolov. shinyapps. io/ATDesign/

Web-based ATD data analysis: Regression analyses https: //manolov. shinyapps. io/ATDesign/

Web-based ATD data analysis: Regression analyses https: //manolov. shinyapps. io/ATDesign/

Web-based ATD data analysis: Average DIfference between Successive Observations https: //manolov. shinyapps. io/ATDesign/

Web-based ATD data analysis: Actual and Linearly Interpolated Values: Difference https: //manolov. shinyapps. io/ATDesign/

References for the examples Baldwin, V. N. , & Powell, T. (2015): Google Calendar: A single case experimental design study of a man with severe memory problems. Neuropsychological Rehabilitation, 25, 617 -636. Coker, P. , Lebkicher, C. , Harris, L. , & Snape, J. (2009). The effects of constraint-induced movement therapy for a child less than one year of age. Neuro. Rehabilitation, 24, 199– 208. Kirsch, N. L. , Shenton, M. , Spirl, E. , Rowan, J. , Simpson, R. , Schreckenghost, D. , & Lo. Presti, E. F. (2004). Web-Based assistive technology interventions for cognitive impairments after traumatic brain injury: A selective review and two case studies. Rehabilitation Psychology, 49, 200 -2012.

References for the analyses • Nonoverlap indices Parker, R. I. , Vannest, K. J. , & Davis, J. L. (2011). Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification, 35, 303 -322. Tarlow, K. (2016, November 8). An improved rank correlation effect size statistic for singlecase designs: Baseline corrected Tau. Behavior Modification. Advance online publication. doi: 10. 1177/0145445516676750 Tarlow, K. R. , & Penland, A. (2016, September 26). Outcome assessment and inference with the Percentage of Nonoverlapping Data (PND) single-case statistic. Practice Innovations. Advance online publication. doi: 10. 1037/pri 0000029 Vannest, K. J. , & Ninci, J. (2015). Evaluating intervention effects in single-case research designs. Journal of Counseling & Development, 93, 403 -411.

References for the analyses • Within-case standardized mean difference (SMD) Beretvas, S. N. , & Chung, H. (2008). A review of meta-analyses of single-subject experimental designs: Methodological issues and practice. Evidence-Based Communication Assessment and Intervention, 2, 129 -141. • Between-cases standardized mean difference (SMD) Shadish, W. R. , Hedges, L. V. , Horner, R. H. , & Odom, S. L. (2015). The role of between-case effect size in conducting, interpreting, and summarizing single-case research (NCER-2015 -02). Washington, DC: National Center for Education Research, Institute of Education Sciences, U. S. Department of Education. Retrieved on October 12, 2016 from http: //ies. ed. gov/ncser/pubs/2015002/. • Log response ratio Pustejovsky, J. E. (2015). Measurement-comparable effect sizes for single-case studies of freeoperant behavior. Psychological Methods, 20, 342 -359. • Slope and level change (SLC) and Mean phase difference (MPD) Manolov, R. , & Rochat, L. (2015). Further developments in summarising and meta-analysingle-case data: An illustration with neurobehavioural interventions in acquired brain injury. Neuropsychological Rehabilitation, 25, 637 -662.

References for the analyses • Alternating treatments designs Jacoby, W. G. (2000). Loess: A nonparametric, graphical tool for depicting relationships between variables. Electoral Studies, 19, 577– 613. Manolov, R. , & Onghena, P. (in press). Analyzing data from single-case alternating treatments designs. Psychological Methods. Moeyaert, M. , Ugille, M. , Ferron, J. , Beretvas, S. N. , & Van Den Noortgate, W. (2014). The influence of the design matrix on treatment effect estimates in the quantitative analyses of single-case experimental designs research. Behavior Modification, 38, 665– 704. Solmi, F. , Onghena, P. , Salmaso, L. , & Bulté, I. (2014). A permutation solution to test for treatment effects in alternation design single-case experiments. Communications in Statistics Simulation and Computation, 43, 1094– 1111. Wolery, M. , Gast, D. L. , & Hammond, D. (2010). Comparative intervention designs. In D. L. Gast (Ed. ), Single subject research methodology in behavioral sciences (pp. 329– 381). London, UK: Routledge.

Example: Winkens et al. (2014) Protocol design • AB design. The moment of change in phase determined at random • Baseline (A) compared with the ABC method: a simplified form of behavioral modification therapy especially designed for nursing staff (B) • 1 patient with acquired brain injury: olivo-ponto-cerebellar ataxia; 52 years-old, wheelchair-bound • Outcome: daily score on verbal aggressiveness (2 x 0: not at all to 2 x 4: continuous yelling, screaming, cursing, or threatening) during helping with Activities of Daily Living • Original analyses: Visual analysis (excessive variability) + Randomization test + NAP

Example: Winkens et al. (2014)

R-Commander data analysis: Websites

R-Commander data analysis: Installing the package once

R-Commander data analysis: Loading the package every time

R-Commander data analysis: Using the package for design Minimum 30 for p ≤. 05 Changed from 7

R-Commander data analysis: Using the package for design

R-Commander data analysis: Using the package for analysis

R-Commander data analysis: Using the package for analysis

R-Commander data analysis: Using the package for analysis

R-Commander data analysis: Using the package for analysis

Example: Fictitious

R-Commander data analysis: Using the package for analysis Plug-in only works for equal n

R-Commander data analysis: Using the package for analysis

R-Commander data analysis: Using the package for analysis Additional file, separated by tabs

Example: Sil et al. (2013)

Example: Sil et al. (2013) Protocol design • Several Alternating treatments designs with initial baseline. Alternation determined at random with a maximum of 2 consecutive per condition (restricted randomization; semi-random) • Baseline (A) compared with passive distraction (B) and interactive distraction (C) • 4 -year-old girl receiving repeated burn dressing changes • Outcome: child cooperation (to increase) and distress (to reduce) as reported by a parent and a nurse on a 100 mm visual analogue scale • Original analysis: Randomization test

Example: Sil et al. (2013) itching Without it no long sig. , as per original analysis

R-Commander data analysis: Using the package for analysis

R-Commander data analysis: Using the package for analysis

R-Commander data analysis: Using the package for analysis

R-Commander data analysis: Using the package for analysis

Additional references Heyvaert, M. , & Onghena, P. (2014). Analysis of single-case data: Randomisation tests for measures of effect size. Neuropsychological Rehabilitation, 24, 507 -527. Heyvaert, M. , & Onghena, P. (2014). Randomization tests for single-case experiments: State of the art, state of the science, and state of the application. Journal of Contextual Behavioral Science, 3, 51– 64. Levin, J. R. , Ferron, J. M. , & Gafurov, B. S. (2016, July 1). Comparison of randomization-test procedures for single-case multiple-baseline designs. Developmental Neurorehabilitation. Advance online publication. doi: 10. 1080/17518423. 2016. 1197708. Sil, S. , Dahlquist, L. M. , & Burns, A. J. (2013). Case study: videogame distraction reduces behavioral distress in a preschool-aged child undergoing repeated burn dressing changes: A single-subject design. Journal of Pediatric Psychology, 38, 330– 341. Winkens, I. , Ponds, R. , Pouwels-van den Nieuwenhof, C. , Eilander, H. , & van Heugten, C. (2014). Usingle-case experimental design methodology to evaluate the effects of the ABC method for nursing staff on verbal aggressive behaviour after acquired brain injury. Neuropsychological Rehabilitation, 24, 349 -364.