Psychometrics of the CCHPA 7 th National Conference

Psychometrics of the CCHPA 7 th National Conference on Quality Health Care for Culturally Diverse Populations (Diversity. RX) Baltimore, MD October 20, 2010 Presenter: Kristyn J. Smith, BA Research Assistant CASE Family Medicine Research Division Authors: Sonja Harris-Haywood, MD, MS Assistant Professor, Research Division, Dept. Family Medicine Case Western Reserve School of Medicine Tawara Goode, MS Assistant Professor and Director of the National Center for Cultural Competence (NCCC) Georgetown University Medical Center Suzanne Bronheim, Ph. D Associate Research Professor of Pediatrics Georgetown University Center for Child and Human Development National Center for Cultural Competence Presentation Created By: Kristyn Smith, BA

Psychometrics of the CCHPA • Background ▫ How was the CCHPA developed? • Methods ▫ Who participated in the study ? • Analyses ▫ What analyses was performed? • Results ▫ Is the CCHPA valid and reliable? ▫ What items are on the CCHPA? ▫ What are the respondents scores on the CCHPA? • Conclusion ▫ Implications ▫ Future studies ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Background Cultural and linguistic competency is hypothesized to be a mechanism by which health care providers can reduce health disparities and provide quality and equitable care to racially, ethnically, culturally, and linguistically diverse populations. Effective measures of CLC are needed to test this hypothesis. ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Gap in Knowledge Few valid, reliable tools exist to accurately assess the cultural and linguistic competence (CLC) of health care providers Most tools that exist have not had any validation and reliability analyses performed, meaning they have little data grounding their tools as an actual assessment of CLC ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Objective To examine the validity and reliability of the Cultural Competence Health Practitioner Assessment (CCHPA) ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

The Cultural Competence Health Practitioner Assessment (CCHPA) 129 item self-assessment of health care provider cultural and linguistic competence developed by the National Center of Cultural Competence(NCCC) , Georgetown University Medical Center Contained the following 6 content area subscales: ▫ ▫ ▫ Values and Belief Systems Cultural Aspects of Epidemiology Clinical Decision-Making Life Cycle Events Cross-cultural Communication Empowerment and Health Management ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Cultural and Linguistic Competency The CCHPA is based on 2 definitions: Cultural Competency is “a set of values and principles, that are demonstrated through behaviors, attitudes, policies, and structures that enable effective operations across cultures”. (Cross 1989) Linguistic competency is “the capacity to communicate effectively and convey information that is easily understood by diverse audiences, including persons of limited English proficiency and low literacy skills” (NCCC 2000) ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Development of the CCHPA (2000 -2002) 1) Items were developed by NCCC faculty and staff 2) Item content was validated by a 21 -member expert panel of nationally recognized experts in cultural and linguistic competence and health care delivery 3) Feedback on item content, clarity and format was provided by a focus group composed of community health practitioners 4) Initial version was pilot tested at 5 community health centers ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Psychometrics of the CCHPA Outline • Background ▫ How was the CCHPA developed? • Methods ▫ Who participated in this study? • Analyses ▫ What analysis was performed? • Results ▫ Is the CCHPA reliable and valid? ▫ What are some items on the CCHPA? ▫ What are the healthcare providers’ scores on the CCHPA? • Conclusion ▫ Implications ▫ Future studies ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Methods CCHPA was posted online from January 2005 to May 2008 8, 600 respondents voluntarily completed the CCHPA ▫ 2, 504 self identified as MD/DO (physicians), PA/NP (clinicians), or RN/LPN/BSN (nurses) ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Study Sample Characteristics of Respondents Gender Female Male Age <30 31 -40 41 -50 51 -60 >60 Race White Black Asian Missing+ Profession Nurse (RN, LPN, BSN) Clinician (P. A. /N. P. ) Physician (M. D. /D. O. ) n* % 2176 325 86. 9 13. 0 726 631 665 413 58 29. 1 25. 3 26. 7 16. 6 2. 3 1816 213 145 330 72. 5 8. 5 5. 8 13. 2 1864 341 299 74. 4 13. 7 11. 9 *Due to missing data not all responses sum to the total of N=2504 +Due to the small sample size of n = 40 respondents who identified as Pacific Islander, Native American/Alaskan Native, Latino, and Other were categorized as missing ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Psychometrics of the CCHPA Outline • Background ▫ How was the CCHPA developed? • Methods ▫ Who participated in the study ? • Analyses ▫ What analysis was performed? • Results ▫ Is the CCHPA reliable and valid? ▫ What are some items on the CCHPA? ▫ What are the respondents scores on the CCHPA? • Conclusion ▫ Implications ▫ Future studies ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Psychometric Analysis of the CCHPA • Conducted using a 3 -step process: 1)Factor Analysis 2)Item Response Theory (IRT) 3)Differential Item Function (DIF) ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Psychometric Analysis of the CCHPA 1)Factor Analysis • determines how items group together to measure smaller ideas within a larger construct. Example: Math Test Addition Multiplication ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Psychometric Analysis of the CCHPA 2)Item Response Theory(IRT) • uses mathematical models (eg. Rasch) to provide a framework for evaluating the reliability and validity of a questionnaire. • IRT is based on the idea that the probability of answering an item correctly is a mathematical function of the person’s ability and the item difficulty. Example: Math Test Easy question: 1+1 = 2 More difficult question: (53 + 28) -18 = 135 ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Psychometric Analysis of the CCHPA 3)Differential Item Function(DIF) • determines whether item difficulty differs by groups(eg. gender, race, and profession). • the presence of DIF indicates an item in a measure may be biased toward a particular group. Example: Math Test 1. If the number of yards on a football field were reduced by half, how long would the new field be? ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Psychometrics of the CCHPA Outline • Background ▫ How was the CCHPA developed? • Methods ▫ Who participated in the study ? • Analyses ▫ What analysis was performed? • Results ▫ Is the CCHPA reliable and valid? ▫ What are some items on the CCHPA? ▫ What are the respondents scores on the CCHPA? • Conclusion ▫ Implications ▫ Future studies ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Psychometric Analysis of the CCHPA 1)Factor Analysis • determines how items group together to measure smaller domains with in the larger construct. 9 2)Item Response Theory (IRT) • uses mathematical models(e. g. Rasch) to provide a framework for evaluating the reliability and validity of a questionnaire. • IRT is based on the idea that the probability of a correct response to an item, is a mathematical function of the person’s ability and the item difficulty. 25 3)Differential Item Functioning (DIF) • determines whether item difficulty differs by groups. (eg. gender, race, and profession). The presence of DIF indicates an item in a measure may be biased toward a particular group. 28 62 of the original 129 items were removed 67 valid and reliable items underwent further analysis ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Results • 3 factors responsible for 46% of the variance+. Factor I : Knowledge of cultural and linguistic competence Factor II : Implementation of culturally and linguistically competent care Factor III: Community Engagement Factors Items Separation Index Cronbach’s α Factor I 24 13. 82 0. 92 Factor II* 21 11. 59 0. 88 Factor III* 23 22. 64 0. 92 * Includes 2 questions that loaded on factors 2 & 3 (so 2 questions counted twice) +Factors contributing to at least 10% of the variance are desirable ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Factor 1: Knowledge of Cultural and Linguistic Competence Item Difficulty Calibration Logit Easiest Item 6. I know the impact of family dynamics on health care decisions (e. g. , high value placed on decisions of elders, differing gender roles, role of extended family etc. ). Most Difficult Item -1. 01 1. I know the following values and belief systems for culturally diverse groups in my service area related to: g. traditional healing practices 1. 11 Scored on a 4 point Likert scale of knowledge with 1 being Not at all, 2 barely, 3 fairly well, and 4 very well ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Factor 2: Implementation of Culturally and Linguistically Competent Care Item Difficulty Calibration Logit Easiest Item 30. I modify treatment and discharge plans when serving individuals with disabilities and their families. Most Difficult Item -0. 7 17. I integrate knowledge of diverse values and belief systems related to healing and wellness in treatment/discharge planning. 0. 52 Scored on a 4 point Likert scale of frequency with 1 being Never, 2 Seldom, 3 Sometimes, and 4 Regularly ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Factor 3: Community Engagement Item Difficulty Calibration Logit Easiest Item 21. I keep abreast of current literature on research findings and evidence-based practices that report the complex array of biologic, economic, environmental, social and cultural factors that are known to contribute to health disparities among racial and ethnic groups. Most Difficult Item -1. 05 22. I have developed affiliations with traditional healers (e. g. , curanderas, santeras, medicine men or women, espiritistas) to enhance health care access and utilization within the communities I serve. 1. 38 Scored on a 4 -point Likert scale of frequency and knowledge with 1 meaning “Never or Not at all” to 4 “Regularly or Very Well”, respectively ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Scoring of the CCHPA Each factor is scored individually Scores from LEAST competent to MOST competent Factor 1 (Knowledge) score range: 24 - 96 Factor 2 (Implementation) score range: 21 – 84 Factor 3 (Community Engagement) score range: 23 - 92 ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Factor 1 and 3 : Knowledge & Community Engagement Wright Item Person Map Items Respondents ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

F 2: Implementation Wright Item Person Map Majority of People Majority of Items ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Psychometric Analysis of the CCHPA Wright Item Person maps show that: Factor 1 & 3 need additional items to better distinguish people with high and low ability Factor 2 needs additional items in the difficult range ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Relationship between Factors Scores: Pearson Correlations** FACTORS Factor 1 Knowledge Factor 2 Implementation Factor 3 Community Engagement Factor 1 1 . 568* . 584* Factor 2 . 5 68 * 1 . 665* Factor 3 . 584 * . 665* 1 *p=. 001 **Pearson correlation >0. 5 is a moderate association ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Factor 1: Knowledge Scores by Respondent Demographics Group Race/Ethnicity* White Black Asian N Mean F 1 Score+ (24 -96) Std. Deviation Effect Size 1816 213 145 63. 8 69. 1 65. 0 12. 5 13. 8 . 42. 31 *Effect size ≥ 0. 5 indicates moderate clinical effect +Mean scores may be slightly inflated due to the homogeneity of the sample ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Factor 2: Implementation Scores by Respondent Demographics N Mean F 2 Score (21 -84)+ Std. Deviation Effect Size* Gender* Male 262 64. 7 14. 2 . 30 Female 1614 68. 1 12. 4 Group *Effect size ≥ 0. 5 indicates moderate clinical effect +Mean scores may be slightly inflated due to the homogeneity of the sample ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Factor 3: Community Engagement Scores by Respondent Demographics Group Age Under 30 years 31 – 40 years 41 – 50 years 51 – 60 years Over 60 years Profession Physicians (MD/DO) Clinicians (PA/NP) Nurses (RN/LPN) N Factor 3 Mean Std. Score Deviation (23 -92)+ Effect Size* 413 58 726 631 665 57. 3 58. 6 53. 3 51. 7 53. 6 15. 7 17. 3 15. 0 14. 7 15. 7 small 299 341 1864 58. 5 59. 1 52. 1 15. 3 14. 3 15. 2 small . 31. 43. 30 . 47 *Effect size ≥ 0. 5 indicates moderate clinical effect +Mean scores may be slightly inflated due to the homogeneity of the sample ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Factor Scores & Employer Sponsored Training – Variable Relationship Validity Employer Sponsored Training+ N F 1 Score** Effect Size# F 2 Score* Effect Size# F 3 Score* Effect Size# No Training 307 63 small 64 . 32 47 . 48 Training 2152 65 68 55 *Statistically significant at the. 001 level significant at the. 002 level +45 respondents did not answer the training question #Effect size ≥ 0. 5 indicates moderate clinical effect **Statistically ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Psychometrics of the CCHPA Outline • Background ▫ How was the CCHPA developed? • Methods ▫ Who participated in the study ? • Analyses ▫ What analysis was performed? • Results ▫ Is the CCHPA reliable and valid? ▫ What items are on the CCHPA? ▫ What are the respondents scores on the CCHPA? • Conclusion ▫ Implications ▫ Future studies ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Implications: Cultural & Linguistic Competence as a Continuum CCHPA contains 3 factors representing 3 domains ▫ Knowledge ▫ Implementation ▫ Community Engagement Builds on Cross’ theory of cultural competency as a continuum ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Implications: Cultural & Linguistic Competency 3 Level Continuum ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Conclusion The Cultural Competence Health Practitioner Assessment (CCHPA) is a criteria-related valid and reliable self-assessment of healthcare provider cultural & linguistic competence ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Current & Future Research CCHPA is currently being used to: assess health care provider CLC in multiple settings and disciplines (including cancer and emergency care settings), this data will be used to validate the new items being added to the tool test its association with the newly developed Patient Assessment of Cultural and Linguistic Competence (PACLC) In the future the CCHPA will be used to: test the association of cultural and linguistic competency to patient outcomes to reduce disparities in medication adherence and patient trust ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Funding Information National Institutes of Health KL 2 Multidisciplinary Clinical Research Training Program National Cancer Institute RO 1 Appraisal and Diagnostic Delay of Colon Cancer University Hospitals of Cleveland’s Case Medical Center Minority Faculty Development Fund Award ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Acknowledgements Stephen Zyzanski, Ph. D Psychometrician , Classical Theory Professor Research Division Dept Family Medicine Case Western Reserve School of Medicine Yong Gao, Ph. D Psychometrician, Item Response Theory & Differential Item Functioning Assistant Professor Dept. Kinesiology Boise State University ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Acknowledgements Panel of Content Experts Joseph Betancourt, MD, MPH Alexander Green, MD Denice Cora-Bramble, MD Melanie Tervalon, MD, MPH Robert Like, MD, MS Elena Rios, MD ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

Acknowledgements Senior Investigators: Sonja Harris-Haywood, MD, MS Assistant Professor Dept. Family Medicine, Research Division Case Western Reserve University, School of Medicine Tawara Goode, MA Assistant Professor Director, National Center of Cultural Competence Georgetown University Medical Center Suzanne Bronheim, Ph. D Associate Professor Dept. Pediatrics, Georgetown University Center of Child & Human Development National Center of Cultural Competence, Georgetown University Medical Center ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development

THANK YOU FOR ATTENDING! To reach us email kjs 24@case. edu ©Case Western Reserve University – Family Medicine Research Division & National Center of Cultural Competence – Georgetown University Center for Child and Human Development