Expanding the Options Alternative Ways to Measure HFE

Expanding the Options: Alternative Ways to Measure HF/E Effectiveness Cristina G. Banks, Ph. D Interdisciplinary Center for Healthy Workplaces University of California, Berkeley May 31, 2018

Executive Summary • Created a baseline of HF/E criterion measures typically used for purposes of comparison with alternative measures. • Pursued an extensive search for alternative criterion measures appropriate for use in HF/E studies in order to broaden the spectrum of evidence of impact. • Identified multiple measures in 11 content areas which met psychometric standards of reliability and validity. • Narrowed the set of recommended measures to the 10 most promising criterion measures for the purpose of showing broader value. • Provided a framework for shifting the value proposition from “less disability” to “more capability. ”

Project Objectives • Create an inventory of scientifically-sound criterion measures for use in HF/E research studies. • Provide recommendations for the use of alternative criterion measures to HF/E researchers and practitioners.

Project Steps • Discuss with OERC leadership to determine the types of criterion measures of greatest interest to the membership and research partners. • Review current criterion measures used in HF/E research to serve as a baseline for comparison with alternative measures. • Review criterion measures used in research outside of HF/E to identify qualifying alternative measures. • Create an annotated spreadsheet of alternative criterion measures. • Assemble an annotated inventory of criterion measures combing both sets of criterion measures. • Write project report summarizing project steps and outcomes.

Background & Considerations • What is meant by “knowledge work” and how can it be evaluated qualitatively and quantitatively? • Performance does not mean productivity. • Performance = “Actions people engage in at work that are directed at achieving the organization’s goals and can be scaled in terms of how much they contribute to said goals. ” • Productivity = Quantity and quality of output given a unit of time or cost • Latent structure of individual performance • Assessment strategies

Latent Structure of Individual Performance 1. 2. 3. 4. 5. 6. 7. 8. Technical performance (not task performance) Communication (verbal and written) Initiative, persistence and effort (observable actions) Counterproductive work behavior (actions that have negative impact on accomplishment) Supervisor, manager, and executive leadership (consideration, initiating structure, goal emphasis, empowerment, training, role model) Management performance (decision-making, goal setting, coordination, monitoring, representation, staffing, administration, commitment) Peer-team member leadership (leadership factors above) Team member-peer management (management factors above) Campbell, J. P. (2013) Assessment in Industrial and Organizational Psychology: An overview. APA Handbook of Testing and Assessment in Psychology, Vol 1, 355 -395.

Levels of Performance Assessment Individual Level: • • Performance ratings Performance in a simulator Performance in a job sample Goal achievement when performance is under a person’s total control Team Level: • • Degree of accomplishment of major substantive task goals • Degree to which team improves its resources, skills, and coordination over time Degree to which team members feel rewarded by, satisfied with their role and committed to team’s goals, commit effort to goal accomplishment Organization Level: • Unit/organizational effectiveness (set of outcomes that the organization wants to maximize, optimize or maintain) • Productivity: ratio of the value of output (effectiveness) to the costs of achieving that level of output • Turnover (voluntary)

Individual Level DV Assessment • Job satisfaction (complex content) • Justice (relative fairness of organization’s procedure for managing and dispensing rewards) • Overall well-being (physical health, mental and psychological health, work-family conflict, work-related stress)

Background & Considerations (cont. ) Path forward: 1. Understand how HF/E interventions and programs could impact individual, team and organizational performance and productivity. 2. Determine how to capture the value of HF/E interventions and programs in all the ways knowledge worker capability might be improved. 3. Identify measures of constructs that reflect how knowledge worker capability might be improved. 4. Identify those measures that have acceptable psychometric properties and could be acquired for use by HF/E professionals.

Background & Considerations (cont. ) Limitations: 1. We cannot pretend to understand HF/E thoroughly; this study is a first pass at identifying what can work for HF/E professionals. 2. Many measures that looked promising were used infrequently; information about many measures is sparse. 3. Some measures are proprietary; they will need to be acquired from the publisher but they were not excluded for this reason. 4. Some measures are highly contextual and will need to be adapted to the organization to which it is applied. 5. The specific measures considered to be useful are likely to be determined by the individual professional.

Presentation Goals ● Increase awareness of alternative measures that arguably apply to HF/E research and applications ● Provide guidance for utilization of alternative measures ● Present a framework for making the case for business value

Sampling of HF/E Criterion Measures Performance/Productivity Keystrokes/hour Keystroke errors/hour Task completion time: tasks/day, Accuracy of tasks/day Mouse-clicks Total computer use Self-reported Job Satisfaction Physical Status Upper body stress caused by typing Gravitational demand Muscle movements in relation to work related activities Acceleration on treadmill workstation Heart Rate Sedentary time Injuries Overall Health & Fitness Health survey Ergonomics climate survey

Sampling of HF/E Criterion Measures Musculoskeletal Disorders Head posture improvement Eye strain reduction Lower Back Pain reduction Neck pain reduction Upper extremity Pain Reduction Torso and upper body posture improvement Reduction of sedentary time Comfort/Discomfort Self-report comfort, discomfort, and fatigue Corrected chairs (angle, height, firmness, materials, etc. ) Education Knowledge about risks Reduction in injuries Reduction in sick leave

Alternative Criterion Measures: Content Areas • • • Task Performance and Productivity Engagement, Motivation, and Commitment Satisfaction and Well-Being Creativity and Innovation Stress and Burnout (Workplace) Social Support Absenteeism and Presenteeism Cognitive Performance Organizational Effectiveness Workplace Health Return on Investment and Utility

Task Performance and Productivity Questionnaires Individual-level ● Health and Work Questionnaire (HWQ; Shikiar, Rentz, Halpern, & Khan, 2001) ● Task performance (unnamed - Williams & Anderson, 1991) ● Work role performance (unnamed - Griffin, Neal, & Parker, 2007)

Task Performance and Productivity Questionnaires Team-level ● Team effectiveness 1 (unnamed – Gibson, Zellmer-Bruhn, & Schwab, 2003) ● Team effectiveness 2 (unnamed – Pearce & Sims, 2002) ● Team productivity (unnamed – Hannah, Walumbwa, & Fry, 2011; Nyhan, 2000

Task Performance and Productivity Processes and Techniques ● Nominal Group Technique (NGT; Dunnette, Campbell, & Jaastad, 1963; Delbecq & Vande. Ven, 1971) ● Behaviorally Anchored Rating Scales (BARS; Smith & Kendall, 1963; Schwab, Heneman, & De. Cotiis, 1975) ● Behavior Observation Scales (BOS; Latham & Wexley, 1977)

Task Performance and Productivity Objective Measures ● Operation Function Analysis (OFA; Bumbarger, 1984) ● Multi-Minute Measurement (MMM; Overby, 1983) ● Achievement method: Completion of goals (Ray & Sahu, 1989) ● Percentage of time spent in value-added activities (Agarwall, 1980)

Task Performance and Productivity Objective Measures (2) ● Professional time utilization (Ray & Sahu, 1989) ● Outcome as a measure (Merrifield, 1994) ● Outcome-input ratio (Thomas & Baron, 1994) ● Cross-Functional Analysis (CFA; Harris & Vining, 1987) ● Efficiency standard times and operating efficiency (Kassen, Russell, & Chrisman, 1998)

Engagement, Motivation, and Commitment ● Utrecht Work Engagement Scale - Abbreviated (UWES-9; Schaufeli, Bakker, & Salanova, 2006) ● Team-level version (Costa, Passos & Bakker, 2014) ● Affective Commitment Scale (ACS; Allen & Meyer, 1996) ● Job involvement (unnamed – Shantz, Alfes & Bailey, 2016) ● Job Involvement Questionnaire (JIQ; Kanungo, 1982) ● Multidimensional Work Motivation Scale (MWMS; Gagné et al. , 2015) ● Intellectual, Social, Affective (ISA) Engagement Scale (Soane et al. , 2012) ● Cognitive and Affective Regulation Scales (CRS & ARS; Yeo & Frederiks, 2011)

Satisfaction and Well-Being Job Satisfaction ● Minnesota Satisfaction Questionnaire (MSQ; Weiss, Dawis, England, & Lofquist, 1967) ● Job Descriptive Index – Abridged (JDI; Stanton et al. , 2002) ● Job in General Scale – Abridged (Ironson, Smith, Brannick, Gibson, & Paul, 1989) ● Overall Job Satisfaction Scale – Short Form (OJS; Judge, Bono, & Locke, 2000) ● Faces Scale (Kunin, 1955)

Satisfaction and Well-Being Work-Related Affect ● Positive and Negative Affect Schedule (PANAS; Watson, Clark, & Tellegen, 1988) ● Job-Related Affective Well-Being Scale (JAWS; Van Katwyk, Fox, Spector, & Kelloway, 2000) ● Psychological Well-Being Scales (Ryff, 1989)

Creativity and Innovation ● Innovative work behavior (unnamed – De Jong & Den Hartog, 2010) ● Innovative work behavior 2 (unnamed – Messmann & Mulder, 2012) ● Innovative behavior (unnamed – Scott & Bruce, 1994) ● Work-Related Curiosity Scale (Mussel, Spengler, Litman, & Schuler, 2012)

Stress and Burnout ● Stress in General (SIG) Scale (Yankelevich, Broadfoot, Gillespie, & Guidroz, 2012) ● Maslach Burnout Inventory (MBI; Schaufeli, Leiter, Maslach, & Jackson, 1996) ● Oldenburg Burnout Inventory (Demerouti, 1999)

(Workplace) Social Support ● Survey of Perceived Organizational Support (SPOS; Eisenberger, Huntington, Hutchison, & Sowa, 1986) ● Modified version assessing subordinate perceived supervisor support (Eisenberger, Stinglhamber, Vandenberghe, Sucharski, & Rhoades, 2002) ● Quality of interactions within teams (unnamed – Hogl & Gemuenden, 2001)

Absenteeism and Presenteeism ● Self-reported retrospective recall ● Sickness absenteeism (Caverly, Cunningham, & Mac. Gregor, 2007 ● General absenteeism (Johns, 2011) ● Presenteeism (Aronsson, Gustafsson, & Dallner, 2000) ● Organizational records ● “Absolute” presenteeism ● Stanford Presenteeism Scale

Cognitive Performance • • N-Back Task (Miller, Price, Okun, Montijo, & Bowers, 2009) Necker Cube Pattern Control (Cimprinch & Ronis, 2003) Posners Cueing Test (Hayward & Ristic, 2013) Operation Span Task (Redick, Broadway, Meier, Kuriakose, Unsworth, & Kane, 2012) • NASA TLX (Hart & Staveland, 1988)

Organizational Effectiveness ● Key Performance Indicators (KPIs) (O’Neill, 2007) ● Workplace Balanced Scorecard (O’Neill, 2007) ● Survey of Organizations Questionnaire (Taylor & Bowers, 1972) ● Productivity and Measurement Enhancement System (PROMES) (Pritchard, Jones, Roth, Stuebing, & Ekeberg, 1988)

Workplace Health • NIOSH (proposed) Framework for Worker Well-Being (TBA; Chang, Sauter, Sayers, Cerully, Schulte, & Uscher-Pines, 2018) • Workplace Environment Survey (Robertson, Huang, O’Neill, & Schleifer, 2008) • Ergonomics Climate Assessment (Hoffmeister, Gibbons, Schwatka, & Rosencranz, 2015)

Return on Investment and Utility ● ● ● Cost/Benefit Analysis (Rouse & Boff, 2012) ROI (Ip, Gober & Rostykus, 2016) Expectancy Charts Utility Models Logic Maps

Conclusion • Multiple alternative criterion measures are available for HF/E researchers and professionals to use for their work. • Choice of criterion measure is critically tied to an understanding of the actual connections between a HF/E intervention or program and a change in individual performance capability. • Area of greatest opportunity is establishing the linkage between human factors/ergonomic strategies and performance improvement. • The proposition is to switch perspectives from one focused on preventing illness, injury or disability to one focused on enhanced capability and motivation to work. • When this switch occurs, collaboration across organizational partners can occur—creating a critical mass of influence.

Conclusion • HF/E impact will be greater when it “floats all the boats” vs. minimizes the harm to the few. • The positive effect on the organization will be more evident when the benefit is reaped by all employees. • Business value can be calculated when improvement is measured in performance gains.

Thank You www. healthyworkplaces. berkeley. edu cbanks@berkeley. edu