Chapter 10 Planning and Evaluating Applied Behavior Analysis

Chapter 10: Planning and Evaluating Applied Behavior Analysis Research Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 1

Importance of Individual Subject • Enables applied behavior analysts to discover and refine effective interventions for socially significant behaviors • Contrasted with groups-comparison approach Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 2

Groups-Comparison Experiment • Randomly selected pool of subjects from relevant population • Divided into experimental and control groups • Pretest, application of independent variable to experimental group, and posttest Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 3

Group Data Not Representative of Individual Performance • Individuals within a group could stay the same or decrease, while the improvement of others could make it appear as overall average improvement • To be most useful, treatment must be understood at an individual level Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 4

Group Data Masks Variability • Hides variability that occurs within and between subjects • Statistical control should not be a substitute for experimental control • To control effects of any variable, must either hold it constant or manipulate it as an independent variable Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 5

Absence of Intrasubject Replication • Power of replicating effects with individuals is lost • Many applied situations in which overall performance of group is socially significant • When group results don’t represent individuals, should supplement the data with individual results Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 6

Importance of Flexibility in Design • An effective researcher must actively design each experiment so that it achieves its own unique design • Good experimental design is any independent variable manipulation that produces data that convincingly addresses the research question • The book presents analytic tactics in design form Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 7

Experimental Designs • Often designs entail a combination of analytic tactics • Component analysis of elements • Infinite number of possible designs with different combinations • Most effective use ongoing evaluation of data from individuals to employ baseline logic of prediction, verification, and replication Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 8

Internal Validity • Experiments that demonstrate clear functional relations have high degree of internal validity • Experimental control refers to all relevant variables • Steady state responding as evidence • Confounding variables are threats to internal validity Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 9

Subject Confounds • Maturation: changes in subject over course of experiment • Repeated measurement controls and detects uncontrolled variables Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 10

Setting Confounds • Studies in natural settings are more prone to confounding variables than in controlled laboratories • If change in setting occurs, should then hold new conditions constant until steady state responding is observed Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 11

Measurement Confounds • Observer drift or bias • Keeping observers naïve to expected outcomes can reduce observer bias • Must maintain baseline conditions long enough for reactive effects to run their course and then obtain stable responding • Could use intermittent probes except when practice effects would be expected Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 12

Independent Variable Confounds • Placebo control separates effects produced by subject’s perceived expectations • Double-blind control eliminates confounding by subject expectations, teacher and parent expectations, differential treatment by others, and observer bias Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 13

Treatment Integrity • Similar to procedural fidelity • Extent to which the independent variable is implemented or carried out as planned • Low treatment integrity makes it very difficult to confidently interpret experimental results • Treatment drift: when application of independent variable in later phases differs from original application Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 14

Precise Operational Definition • A high level of treatment integrity requires a complete, precise operational definition of treatment procedures • Define in 4 dimensions: verbal, physical, spatial, and temporal Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 15

Simplify, Standardize, and Automate • Simple, precise treatments are more likely to be consistently delivered • Simple, easy-to-implement techniques are more likely to be used and socially validated • Experimenters should standardize as many aspects as possible and practical • If possible without compromise, could use an automated device to deliver independent variable Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 16

Training and Practice • Train or provide practice for individual who will conduct the experimental sessions • Could provide a detailed script, verbal instructions, modeling, or performance feedback Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 17

Assessing Treatment Integrity • Collect treatment integrity data to measure how the actual implementation of the conditions matches the written methods • Observation and calibration give the researcher the ongoing ability to use retraining and practice to ensure high treatment integrity • Reduce, eliminate, or identify the influence of as many potentially confounding variables as possible Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 18

Social Validity • Includes the social significance of the target behavior, the appropriateness of the procedures, and the social importance of the results • Usually assessed by asking direct and indirect consumers • Consumer satisfaction Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 19

Social Importance of Behavior Change Goals • To determine socially valid goals: – Assess the performance of persons considered competent – Experimentally manipulate different levels of performance to determine which produces optimal results Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 20

Social Importance of Interventions • Rating scales and questionnaires for obtaining consumers’ opinions on acceptability of interventions • Examples: – Intervention Rating Profile – Treatment Acceptability Rating Form Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 21

Social Importance of Behavior Changes • Methods for assessing outcomes: – Compare subject’s performance to a normative sample – Use standardized assessment instrument – Ask consumers to rate social validity of performance – Ask experts to evaluate subject’s performance – Test subject’s new performance in natural environment Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 22

Normative Sample • Not limited to posttreatment comparisons • Compare subject’s behavior to ongoing probes of behavior of normative sample to provide ongoing measure of improvement and how much is still needed Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 23

Consumers and Experts • Most frequently used method for assessing social validity is to ask consumers • Experts can be called upon to judge the social validity of some behavior changes Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 24

Standardized and Real-World Tests • Example of standardized test: Self-Injury Trauma Scale (SITS) • Real-world test in the natural environment provides direct assessment of social validity • Also exposes subject to naturally occurring reinforcement, which may promote maintenance and generalization Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 25

External Validity • Degree to which a functional relation in an experiment will hold under different conditions • A matter of degree, not all-or-nothing • Those with greater degrees of generality, make greater contribution to applied behavior analysis Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 26

External Validity and Groups. Design Research • There is nothing in the results of a groupsdesign experiment that can have external validity • Unable to provide data that lead to improved practice in education • Groups-design is effective in large-scale evaluations Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 27

External Validity and Applied Behavior Analysis • Generality of findings in ABA is assessed, established, and specified through replication of experiments • Two major types of scientific replication: direct and systematic Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 28

Direct Replication • Duplicates exactly the conditions of an earlier experiment • Intrasubject direct replication: uses same subject to establish reliability of functional relation • Intersubject direct replication: uses different but similar subjects to determine generality Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 29

Systematic Replication • Researcher purposefully varies one or more aspects of earlier experiment • Can demonstrate reliability and external validity of earlier findings • Can alter any aspect: subjects, setting, administration of independent variable, or target behaviors Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 30

Evaluating Applied Behavior Analysis Research • Questions to ask in evaluating the quality of research in applied behavior analysis fall under 4 categories: – Internal validity – Social validity – External validity – Scientific and theoretical significance Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 31

Internal Validity • Must decide whether functional relation has been demonstrated • Requires close examination of measurement system, experimental design, and the researcher’s control of potential confounds Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 32

Evaluating Internal Validity • Definition and measurement of dependent variable • Graphic display • Meaningfulness of baseline conditions • Experimental design • Visual analysis and interpretation Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 33

Visual Analysis and Interpretation • Factors that favor visual analysis over tests of statistical significance in ABA: – Want to see socially significant behavior change, not statistically significant – Good for identifying variables that produce strong, large, and reliable effects – Accepting statistical analysis as evidence of functional relation may cause researcher not to experiment further – Tests of statistical significance may cause data sets to conform, losing flexibility in design Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 34

Errors • Type I error: when researcher concludes that independent variable had effect on dependent variable, when it did not • Type II error: when researcher concludes that independent variable did not have effect on dependent variable, when it did • Visual analysis leads to less Type I and more Type II errors • Statistical analysis leads to more Type I and less Type II errors Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 35

Social Validity • Independent variable should be assessed in terms of its effects on dependent variable, as well as social acceptability, complexity, practicality, and cost • Consider maintenance and generalization of behavior change in evaluation of a study Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 36

External Validity • To effectively judge external validity, compare a study’s results with those of other relevant published research Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 37

Theoretical Significance and Conceptual Sense • Evaluate a study in terms of its scientific merit • Look at its contribution to the advancement of the field • “knowledgeable reproducibility” Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 38

Need for More Thorough Analyses • Need for more conceptual understanding of the principles that underlie successful demonstrations of behavior change • Readers should consider the technological description, the interpretation of results, and the level of conceptual integrity in experimental reports Cooper, Heron, and Heward Applied Behavior Analysis, Second Edition 39