The Use of Confirmatory Factor Analysis to Support

® The Use of Confirmatory Factor Analysis to Support the Use of Subscores for a Large-Scale Student Outcomes Assessment Rochelle S. Michel, Ph. D. Evaluation 2009 Annual Conference of the American Evaluation Association November 11 -14, 2009

® Introduction • Institutions consistently request assessment information beyond the total score – – Subscores Item analysis Diagnostic information Comparative data 2 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.

® Introduction • Standard A 4 – Program Evaluation Standards (1994) – Calls for the use of valid information in the evaluation of programs, establishing the validity of inferences drawn from information obtained in relation to the evaluation questions (p. 147) 3 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.

® Subscores • “Easiest” information that can be provided • Often used as the data by which to measure and document program effectiveness • Research on issues related to subscore reporting – Haberman, 2005; Haberman, Sinharay, Puhan, 2007; • High correlation between subscores and total score • Additional information of subscores beyond total score 4 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.

® General Education Measure • 108 multiple-choice items • 4 skills (each measured by 27 items) – – Reading Written communication Critical thinking Mathematics • 3 contexts (each measured by 18 items) – Humanities – Social Science – Natural Science 5 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.

® Research Questions • How well does a four-factor model fit the data from the general education student outcomes measure? • How well does a one-factor model fit the data from the general education student outcomes measure? • What do the results of the CFA indicate about subscore usage for the measure under analysis? 6 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.

® Demographics • Test takers (n=15, 504) – Gender • Females – 59% • Males – 41% – Ethnicity • White – 75% • African American – 12% • Hispanic – 4% • Institutions (n=54) 7 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.

® Psychometric Properties • Test reliability, 0. 94 • Skills – – Reading, . 85 Critical Thinking, . 80 Writing, . 77 Math, . 85 • Contexts – Natural Sciences, . 80 – Social Sciences, . 77 – Humanities, . 71 8 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.

® Correlations among skills Skills Total RD CT W Total 1. 00 RD . 92 1. 00 CT . 90 . 83 1. 00 W . 86 . 75 . 69 1. 00 M . 84 . 66 . 64 . 62 M 1. 00 Note. RD=Reading, CT=Critical Thinking, W=Writing, M=Mathematics 9 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.

® Correlations among contexts Contexts Total Humanities Natural Sciences Total 1. 00 Humanities . 84 1. 00 Natural Sciences . 89 . 74 1. 00 Social Sciences . 87 . 73 . 78 10 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. Social Sciences 1. 00

® Confirmatory Factor Analysis • Analyses performed using LISREL 8. 8 (Joreskog & Sorbom, 2006) • Used the tetrachoric correlation matrix to create the covariance matrix to be analyzed • Chi-square statistic – Traditional measure for evaluating overall model fit – Hypothesizes that the model fits the population data perfectly – Both one-factor and four-factor models had statistically significant chi-square values 11 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.

® Confirmatory Factor Analysis Summary of Fit Indices for the One-Factor and Four-Factor Models of a General Education Measure of Student Outcomes Sample RMSEA ECVI SRMR GFI CFI NNFI M 1 0. 0528 16. 198 0. 0386 0. 770 0. 976 0. 975 M 4 0. 0389 8. 964 0. 0323 0. 858 0. 983 Note. M 1=one-factor model; M 4=four-factor model; RMSEA=root mean square error of approximation; ECVI=expected cross-validation index; SRMR=standardized root mean square residual; GFI=goodness-of-fit index; CFI=comparative fit index; NNFI=non-normed fit index. 12 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.

® Summary • Results are consistent with previous exploratory factor analysis conducted – Conflicting results between various fit indices – Neither one-factor nor four-factor model provide an appropriate fit to the data 13 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.

® Implications • Examination of alternative approaches to subscore reporting – Use of scale anchoring methodology/proficiency scaling • Responsible use of current subscores – Total score may provide more information about test takers’ skills than the subscores – Care should be taken making decisions based solely on skill subscores 14 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.

® Thank you! Thank you to Michael Chajewski and Veleka Allen who performed some of the preliminary analyses on the measure. Contact: rmichel@ets. org 15 Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved.