Developing a ContextSensitive Approach to AT Outcomes Measurement

Developing a Context-Sensitive Approach to AT Outcomes Measurement OT 541 Fall 2003 James A. Lenker, MS, OTR/L, ATP

What are Outcomes? z. Outcomes are the result of an intervention (Scherer, 1998) z. Outcomes answer the question, "Does it work? " (Oldridge, 1996) z. Outcomes are the impact produced by AT devices in the lives of users and their environments (Fuhrer, 2001)

What is Outcomes Measurement? z"The evaluation process in the service delivery system that is designed to measure and establish a baseline of what works; how well something works; for which clients it works; and at what level of economic efficiency it works” (De. Ruyter, 1997)

Purposes of Outcomes Measurement z. Funding justification for equipment and services (assessment & training) z. Measure individual's change over time z. Program evaluation: "Who do we serve? How well do we serve them? "

Domains of Outcomes Measurement z. Many domains of inquiry can be considered an "outcome" y Medical status y Activity performance y Quality of Life (QOL) impact y Community impact y AT usage & usability

AT Outcomes Research – Developmentally Speaking z Blissful ignorance (Lenker, 2000) z Denial ( “ ) z Exhortative (Fuhrer, 1999) z Sober appraisal ( “ ) z Infrastructure building ( “ ) z Model demonstration ( “ ) z Getting on with it ( “ )

AT Outcomes Research: Challenges Heterogeneity of…. y. End users: age, abilities, needs, support y. AT devices y. Contexts of use y. Reimbursement models y. Practitioner skills

Limitations of Previous Studies z Incomplete information about reasons for device abandonment z Inconsistent definitions for prevalence of use that have been difficult to compare across studies because and abandonment are often defined differently z Changes in functional status when evaluating longterm device use and disuse, which bias the results toward high rates of ‘abandonment’.

Limitations of Previous Studies z Incompletely defined outcomes indicators (e. g. comfort, safety, and function), hindering replication by others z Incomplete information about device success z Measurement tools that were not sensitive to improvements in function z Low numbers of participants

Research Needs v v Measurement of impact on social participation and on work (Fuhrer, 1999) Consideration of ATD usage within context of use service delivery model, and user characteristics (Fuhrer, 1999) Methods that embrace the user's perspective and assess impact on quality of life (Fuhrer, 1999; Jutai et al. , 1996; Scherer, 1998) Approaches that evaluate underlying factors contributing to overall success/failure of AT interventions (De Ruyter, 1994; Scherer, 1996)

Research Needs (con’t. ) z z z Measurement of the hassle experienced by end users, comparing trade-offs against the functional gains (Sprigle & Abdelhamied, 1998) Measurement for reduced dependence (Fuhrer, 1999) Description of devices that are achieving desirable outcomes, which would provide the rationale for funding ATD devices and/or training services (Scherer, 1996).

The Chain of Outcomes Research (Keith, 1995) z Conceptual Models => relationships among domains z Domains => measurement tools z Measurement tools => data z Research design links treatment to outcome, i. e. interpretation of data, i. e. “outcomes” z Outcomes influence practice and help build theory

So, Where Do I Start? z Know the context of AT use z Identify all relevant stakeholders z Develop specific research questions z Identify indicators that will answer your questions z Identify measurement tools

Useful Conceptual Perspectives z. Quality of Life z. Product Usability z. Functioning and Health z. HAAT: Human, Activity, Assistive Technology z. MPT: Matching Person & Technology

Quality of Life: The Bottom Line z. Definition: “(Quality of Life is) the degree to which the person enjoys the important possibilities of his or her life” (Renwick & Brown, 1996).

Quality of Life (Renwick & Brown, 1996) Quality of Life Physical • Personal safety • Health • Fitness • Mobility Material • Finance • Housing • Transport • Security & tenure Development & Activity • Competence independence, choice & control • Productivity job, education home life, leisure Emotional Well Being • Mental health • Self-esteem • Faith • Sexuality • Fulfillment Social • Relationships family, friends • Community

Product Usability Perspective z. Effectiveness y. Does it achieve the desired goal? z. Efficiency y. What are the costs associated with achieving the desired goal? Ex: $$, time, speed, errors effort, comfort, pain, hassle z. User satisfaction International Standards Organization. (1993). ISO CD 9241 -11. 2

Smith’s Parallel Interventions Model Reduce the Impairment (Therapy) Change technique / Increase Skill Use AT Devices and Services Redesign the Activity Redesign the Environment Use Personal Assistance

ICF – AT Model (Lenker & Jutai, 2002)

HAAT: Human, Activity, Assistive Technology (Cook & Hussey, 2002) CONTEXT Social Context Family Peers Strangers Setting Home School Employment Community Physical Context Space Light Sound Temperature

MPT: Matching Person & Technology (Scherer, 1998)

Getting Down to Business z What question(s) do you want to answer? y. This the most important consideration when designing your study & picking a tool z Who are the stakeholders? y. Consumers, clinicians, administrators, manufacturers, policy makers z What is your conceptual orientation? z What are your resources?

Develop Specific Research Question(s) z. Must be a match between y. Stakeholder interests y. Reimbursement model y. User characteristics y. AT device y. Context of use

Identify Stakeholders z Who is your target audience? y. End users y. Practitioners / direct service providers y. Program administrators y. Manufacturers of AT devices y. Funding / policy agencies x. State VR agencies x. Federal agencies that sponsor rehabilitation R&D programs x. State, local departments of Education

Describe Context of Use z Reimbursement Model y. Educational, vocational, medical y. Practitioners involved in service delivery y. User involvement in service process z AT User y. Physical, cognitive, learning, sensory y. Ages and environments of use y. Needs / goals / roles

Context of Use (con’t. ) z Assistive Technology Devices y. Wheelchair mobility, postural support, seating y. Adaptive computer software and hardware y. Alternative & augmentative communication aids y. Job accommodations, often ‘ergonomic’ y. Aids for daily living activities z Important!! AT devices are uniformly defined but they should not be uniformly studied

What data will answer your research question? z The domain of data may include y. Functional performance, QOL impact, pain, usability, user satisfaction, etc. z The perspective of data may be yobjective (by observation) ysubjective (self-reported) z The nature of data may be yquantitative (data are numeric) yqualitative (data are language-based)

Outcomes research requires measurement tools! z. The most important consideration when picking a tool? y. Knowing what question (s) you want to answer

What other considerations matter when choosing a measurement tool? z Does it measure what you want it to measure? z Does it measure what it purports to measure? z Is it easy for participants to answer consistently? y Test-retest reliability z Is it easy for multiple observers to rate consistently? y Inter-rater reliability z What demands does it place on respondents? z What demands does it place on administrators? y Is tool-specific training required? y Is it time consuming to administer? y How difficult is it to score the results? z Accessibility of format for people with disability?

A Useful Model: ICF z Presents a functional continuum of human performance in life situations z Provides framework for outcomes research y. Interventions y. Reimbursement / service delivery models y. Assessment tools y. Outcomes questions

ICIDH-2 / WHO Model in the Context of Reimbursement Models and Assistive Technology Interventions ICIDH-2 Domain Body Functions and Structures (Body level): Activities (Individual level): Participation (Role level): Contextual Factors (Societal level): Domainrelevant examples Functional and structural integrity vs. impairment. Normal physiologic processes and organ systems function: Cognition, sensory, perception, motor, physiologic and psychological processes Involvement in life situations and performance in roles (family, work, and educational; self-care; leisure) that take place in specific contexts (physical, social, cognitive). Features of the physical and social world that affect the individual but are not specific to the individual's ability or disability; e. g. physical barriers in public spaces, employer attitudes; funding for social service benefits; public policy; Domainrelevant Assistive Technology Heart pacemaker, hip/knee replacement, cochlear implant, baclofen pump Performance of action or activity (cognitive, sensory, motoric, and/or behavioral) that facilitates performance in multiple roles: - initiate, organize, sequence, judge, attend - sit, lift, stand, climb, reach, pinch, grasp, hold, release - relate, interact, cope - hear, speak, see, taste move - read, write, learn, understand Mobility aids: wheelchairs, crutches, canes, walkers; UE orthotics; eyeglasses, hearing aid, AAC device Job modifications of-task/device/ environment; home modification; vehicle modification; computerbased AT Accessible public venues: buildings, transportation, communications, recreation; 3 rd party reimbursement for AT services & equipment Funding Model Medical Rehabilitation Vocational Rehab Educational Independent Living Employer Independent Living Public Policy Adapted from: Baum, C. M. , & Law, M. (1997). Occupational therapy practice: Focusing on occupational performance. American Journal of Occupational Therapy, 51, 286. Gray, D. B. , Quatrano, L. A. , & Lieberman, M. L. (1998). Moving to the next stage of assistive technology development. In D. B. Gray & L. A. Quatrano & M. L. Lieberman (Eds. ), Designing and Using Assistive Technology: The Human Perspective (pp. 299 -309). Baltimore, MD: Paul H. Brookes. Moyers, P. A. (1999). The guide to occupational therapy practice. American Journal of Occupational Therapy, 53, 247 -322. World Health Organization. (1999). ICIDH-2: International classification of functioning and disability. (Beta-2 draft). Geneva: Author.

Example Research Questions z Do computer-based ATDs improve the psychological and emotional well-being of end-users? z Do computer-based ATDs increase end-user participation in work, school, and/or leisure activities? z Do users with clearly defined goals become more successful user of computer-based ATDs? z Does impact of ATD use vary across disability populations?

Research Questions (con’t. ) z Are there societal barriers that preclude or interfere with participation in school, work, or leisure activities? z Are there significant technical problems with computer-based ATDs that are limiting performance for users of these products? z Are there any themes or variables that are associated with successful or unsuccessful computer -based ATD use?

Indicators of AT “Outcome” y. Psychosocial impact on the individual y. User satisfaction with device y. User satisfaction with services y. Achievement of user goals y. Prevalence of long-term AT use y. Tasks and roles in which AT is used y. Environmental influences that contribute to functional AT use/abandonment y. Unmet technology needs y. Employment / educational success

AT Outcome Measurement Tools z PIADS: Psychosocial Impact of Assistive Device Scale z QUEST: Quebec User Evaluation of Satisfaction with Assistive Technology z COPM: Canadian Occupational Performance Measure z MPT: Matching Person & Technology

PIADS z Attribute: Quality of Life y 3 sub-scales: Competence, adaptability, self-esteem z 26 item, 7 -point, self-report z Strengths y. Not device-specific y. Identifies potentially useful points of departure for exploration y. Quick, easy to administer y. Useful retrospectively or pre/post

PIADS z. Additional considerations y. Items are abstract, some users will struggle - especially kids/adolescents y. Better to administer in person y. Open-ended probing questions can provide insight about specific scores

COPM z. Semi-structured interview format, approx. 30 -40 minutes z. Process y. Identify problem areas y. Prioritize 5 areas of greatest importance y. User rates each of the 5 areas on x. Current performance level x. Satisfaction with that performance level

COPM z Strengths y. User centered x. Philosophically sound x. Effective as intervention approach; increased consumer ‘stake’ in the outcome (? ) y. Sensitive to clinically significant changes in performance and satisfaction with performance y. Useful for variety of devices and contexts of use y. Outcome indicators are built-in to the assessment process

COPM z. Additional considerations y. Goals are likely to be interviewer-dependent, which will affect inter-rater reliability y. The specificity of goals to the end user's circumstances hinders generalizability of results to other users y. Goals may not suit all stakeholder perspectives

QUEST z 12 items, 5 -point scale y. Device satisfaction y. Services satisfaction z Quick, straightforward to administer (10 -15 mins. ) z Suitable for phone or distance surveys z Suitable for wide range of AT products and user demographics

QUEST z. Additional considerations y. Expectations often color the perceptions of satisfaction, therefore expectations are a potential confounding factor y. Satisfaction judgments are best supplemented with information that explains underlying predictors y. User performance, or function, with use of AT device is not captured

MPT z. Several instruments y. Survey of Technology Use y. ATD Predisposition Assessment (PA) y. Educational Technology PA y. Workplace Technology PA y. Health Care Technology PA

ATD Predisposition Assessment z Two perspectives are captured y. Professional assesses. . . x. Match between user abilities and ATD requirements x. Psychosocial incentives/disincentives to AT use y. End user self-rates. . . x. Sensory & motor capabilities x. Life satisfaction x. Perceived value of device

MPT z. Additional considerations y. AT ‘benefits’ to user are not explicitly captured y. Professional’s ratings of psychosocial incentives and disincentives to AT use require knowing the consumer really well -- beyond the scope of a typical AT evaluation period y. Professional’s ratings of psychosocial factors is subjective, subject to bias y. Overall, does a fine job of identifying factors that could predispose user to frequent or infrequent use

Conclusion z. Acknowledge the futility of identifying a single “best” AT outcomes tool that is appropriate for all contexts and research questions. z. Accept the challenge of matching research methods and measurement tools to your research area