Why do we need good content validity an

Why do we need good content validity: an introduction to discriminant content validity. Marie Johnston m. johnston@abdn. ac. uk EHPS 2019 Dubrovnik

Health Psychology Measures: what do we measure? • Objective • Step count, weight, BP, clinic attendance, prescribing • Subjective • External referent, e. g. portion size, social support • Intrapsychic theoretical constructs, e. g. intention, self‐efficacy, stress, pain • Self‐report e. g. • Ratings • Questionnaires How do we know we are measuring what we intend to measure?

Psychometrics in health psychology measurement • Typically • Internal consistency • e. g. Cronbach’s alpha – psychometric problems • Indicates how well – but not what is being measured • Validation • Face validity • Relationship with other measures (convergent/divergent construct validity) • Factor analysis • We can assess ‘how well’ a measure is measuring without knowing ‘what’ is being measured

Content Validity • Content validity definition: ‘the degree to which elements of an assessment instrument are relevant and representative of the targeted construct for a particular purpose’ 1 • i. e. what is being measured • Construct validity – does the measure function as theoretical construct does? • Possible to achieve this without content validity • e. g. a measure of self‐efficacy may predict behaviour but the content may be more aligned with outcome expectancy than self efficacy 2 • Factor analysis • Labels given to factors does not establish the content • May be contaminated by determinants and consequences of the construct 1. Haynes et al. 1995 Psychol Assess 7: 238 -47 2. Williams, D. M. (2010). Outcome expectancy and self‐efficacy: Theoretical implications of an unresolved contradiction. Personality and Social Psychology Review, 14(4), 417‐ 425.

Importance of Content Validity • Decisions • Clinical decisions • Choosing intervention components • Selecting ’good’ items for a questionnaire • What use is a reliable measure if we don’t know what it is measuring? • Eg in testing theory • Confusion • Same label given to different questionnaire content • Different labels given to similar content • • A C B E. g. impairment and disability E. g. perceived control and self‐efficacy • Construct validity • Meaningless without content validity • Contamination – makes theory testing impossible • If construct A predicts construct B but each measure contains a little of the other, then observed relationships may simply be due to contamination C • E. g impairment predicts disability , but most measures of impairment contain some disability and vice versa Dixon, D. , & Johnston, M. (2019). Content validity of measures of theoretical constructs in health psychology: Discriminant content validity is needed. British journal of health psychology.

Why we needed to assess content validity • Testing relationship between impairment (of body structure and function) and disability (inability to do things) • Existing measure were a mixture, e. g. impairment ? disability • I cannot walk due to stiffness • My pain makes dressing difficult • During the past 4 weeks, how much did pain interfere with your normal work? • Required a method for finding ‘pure’ measures of each construct, uncontaminated by the other construct Johnston, M. , & Pollard, B. (2001). Consequences of disease: testing the WHO International Classification of Impairments, Disabilities and Handicaps (ICIDH) model. Social Science & Medicine, 53(10), 1261 -1273.

Achieving and Assessing content validity using Discriminant Content Validation (DCV) Achieving content validity • Generating items x • Selecting items Assessing content validity of items and scales • Relevance • Representativeness Johnston, M. , Dixon, D. , Hart, J. , Glidewell, L. , Schröder, C. , & Pollard, B. (2014). Discriminant content validity: A quantitative methodology for assessing content of theory‐based measures, with illustrative applications. British journal of health psychology, 19(2), 240 -257.

Discriminant Content Validity: Steps 1: definitions of all constructs 2: items either from existing items or generated by experts 3: judges and the number needed to evaluate items against definitions. experts in the domain being assessed or target population Number 4: scale on which each item is judged and rated. i. does it measure the construct? + or ‐ Ii confidence : 0‐ 100 Gives scores from minus 100 to plus 100 WHEN • During scale development • identify pure items • Before using an existing measure • select pure items of relevant constructs

Illustration of a completed DCV judgement item designed to measure self-efficacy is judged against the definition of selfefficacy and perceived behavioural control . ITEM: How confident are you that you will be able to walk more? Definition 1 Perceived control is the perception of one’s capabilities to organise and execute courses of action required to produce given attainments Definition 2 Perceived control is the perception of the ease or difficulty of performing the behaviour of interest Theoretical Definition Question measures definition? Definition 1 Yes No 0 10 20 30 40 50 60 70 80 90 100 Definition 2 Yes No 0 10 20 30 40 50 60 70 80 90 100 How confident are you in your judgement?

Discriminant Content Validity: Steps 1: definitions of all constructs 2: items either from existing items or generated by experts 3: judges and the number needed to evaluate items against definitions. experts in the domain being assessed or target population Number 4: scale on which each item is judged and rated. i. does it measure the construct? + or ‐ Ii confidence : 0‐ 100 Gives scores from minus 100 to plus 100 5: Content Validity • For each item for each theoretical construct • usingle‐sample tests (t‐tests or, more commonly, nonparametric one‐sample test (Wilcoxon) 7. Discriminant Content Validity: does the item have content validity for • one construct (pure item) or • more then one item – lacks DCV – contaminated item test whether stronger for one construct

Example of Results: Perceived Behavioural control (PBC) and Self-efficacy(SE) Judged to be PBC SE Neither PBC items 2 1 2 SE items 0 5 0 Johnston, M. , Dixon, D. , Hart, J. , Glidewell, L. , Schröder, C. , & Pollard, B. (2014). Discriminant content validity: A quantitative methodology for assessing content of theory‐based measures, with illustrative applications. British journal of health psychology, 19(2), 240 -257.

Selecting (single) items with strong content validity Demand Items Demand Control Effort Reward 1. My job requires working very hard 1√ ‐. 90 . 97√ ‐. 97 No – CV for both demand Effort 2. My job requires working very fast 1*√ ‐. 62 . 76 ‐. 95 Yes 3. I have enough time to get the job done . 97*√ ‐. 69 . 76 ‐. 85 Yes . 31 . 37 ‐. 92 2 and 3 stronger ‐. 59 . 92√ ‐. 63 No – only CV for Effort 4. I am free from conflicting demands that. 88*√ others make 5. I am not asked to do an excessive. 73 amount of work Select as measure of Demand? Bell, C. , Johnston, D. , Allan, J. , Pollard, B. , & Johnston, M. (2017). What do Demand‐Control and Effort‐Reward work stress questionnaires really measure? A discriminant content validity study of relevance and representativeness of measures. British Journal of Health Psychology, 22, 295‐ 329. https: //doi. org/10. 1111/bjhp. 12232

Representativeness • Measure can have relevance but not represent the full theoretical construct • Eg many items measuring same aspect • Method of assessing representativeness • Identifies gaps Bell, C. , Johnston, D. , Allan, J. , Pollard, B. , & Johnston, M. (2017). What do Demand‐Control and Effort‐Reward work stress questionnaires really measure? A discriminant content validity study of relevance and representativeness of measures. British Journal of Health Psychology, 22, 295– 329. https: //doi. org/10. 1111/bjhp. 12232

References Bell, C. , Johnston, D. , Allan, J. , Pollard, B. , & Johnston, M. (2017). What do Demand‐Control and Effort‐Reward work stress questionnaires really measure? A discriminant content validity study of relevance and representativeness of measures. British Journal of Health Psychology, 22, 295‐ 329. https: //doi. org/10. 1111/bjhp. 12232 Burrell, A. M. G. , Allan, J. L. , Williams, D. M. , & Johnston, M. (2018). What do self‐efficacy items measure? Examining the discriminant content validity of self‐efficacy items. British Journal of Health Psychology, 23, 597– 611. https: //doi. org/10. 1111/bjhp. 12306 Dixon, D. , & Johnston, M. (2019). Content validity of measures of theoretical constructs in health psychology: Discriminant content validity is needed. British journal of health psychology. https: //doi. org/10. 1111/bjhp. 12373 Gardner, B. , Abraham, C. , Lally, P. , & de Bruijn, G. J. (2012). Towards parsimony in habit measurement: Testing the convergent and predictive validity of an automaticity subscale of the Self‐Report Habit Index. International Journal of Behavioral Nutrition and Physical Activity, 9(1), 102. https: //doi. org/10. 1186/1479‐ 5868‐ 9‐ 102 Huijg, J. M. , Gebhardt, W. A. , Crone, M. R. , Dusseldorp, E. , & Presseau, J. (2014). Discriminant content validity of a theoretical domains framework questionnaire for use in implementation research. Implementation Science, 9(1), 11. doi: 10. 1186/1748‐ 5908‐ 9‐ 11 Why do we need good content validity: an introduction to discriminant content validity. Marie Johnston m. johnston@abdn. ac. uk EHPS 2019 Dubrovnik Thank you Johnston, M. , Dixon, D. , Hart, J. , Glidewell, L. , Schröder, C. , & Pollard, B. (2014). Discriminant content validity: A quantitative methodology for assessing content of theory‐based measures, with illustrative applications. British journal of health psychology, 19(2), 240 -257. https: //doi. org/10. 1111/bjhp. 12095