Fit of Idealpoint and Dominance IRT Models to

Fit of Ideal-point and Dominance IRT Models to Simulated Data Chenwei Liao and Alan D Mead Illinois Institute of Technology

Outline Background and Objective Hypotheses and Methods Results Discussions

Background Personality Used in personnel selection - Incremental validity to predict job performance beyond cognitive ability (Barrick & Mount, 1991; Ones et al, 1993) - Less adverse impact (Feingold, 1994; Hough, 1996; Ones et al, 1993). Model-data-fit - Need to calibrate personality traits - Use IRT models - Degree of fit depends on data structure

Background (cont. ) Item response processes – thinking of data structure IRT models and item response processes: 1) Traditional dominance IRT models: - high trait - high probability of endorsing 2) Ideal-point IRT models - similar item & trait – high probability of endorsing

Background (cont. ) Dominance Model IRF: - x: Theta (trait level) - y: Probability of endorsing Ideal-point Model IRF: - x: distance between person trait and item extremity - y: Probability of endorsing

Background (cont. ) Chernyshenko et al, (2001) - Traditional dominance IRT models have failed. Suggest to look at item response processes and Ideal-point IRT models Stark et al. (2006) - Ideal-point IRT models: as good or better fit to personality items than do dominance IRT models Chernyshenko et al. (2007) - Ideal-point IRT method: more advantageous than dominance IRT and CTT in scale development in terms of model-data-fit

Limitation of previous studies and objective of current study Limitation of previous studies - Unknown item response processes! Objective of current study 1) Investigate model-data-fit by utilizing simulation with known item response processes 2) Test the assumption that the best fit model represents data underlying structure of response processes

Current Study

Models Dominance: - Samejima’s Graded Response Model (SGRM); Ideal Point: - General Graded Unfolding Model (GGUM). Larger sample and longer test were said to be related to a better fit (Hulin et al, 1982; De la Torre et al, 2006).

Hypotheses Generating models H 1: Data generated by an ideal point model will be best fit by an ideal-point model and data generated by a dominance model will be best fit by a dominance model. H 2: The ideal point model will fit the dominance data better than the dominance model will fit the ideal-point data. H 3: The ideal-point model will fit the mixture data better than the dominance model.

Hypotheses (cont. ) Sample Sizes H 4: All models will fit better in larger samples. H 5: The GGUM model will fit relatively worse in smaller samples, as compared to simpler, dominance models. Test Lengths H 6: The GGUM model will fit relatively worse for very short tests, as compared to longer tests.

Datasets Self-Control Scale from the 16 PF Procedure: 1) Calibrate 16 PF data to get item parameters - SGRM: PARSCALE 4. 1; GGUM: GGUM 2004. 2) Generate simulated data: - models: ideal point/dominance/mixed; - sample size: 300, 2000; - test length: 10, 37; - 50 replications;

Model-Data-Fit Cross validation ratio: each item in each condition Only singles – simulation study assures unidimensionality assumption Smaller value – better fit Frequencies of ratios were tallied into 6 groups: very small (<1), small (1 -<2), medium (2<3), moderately large (3 -<4), large (4 -<5), very large (>=5).

Results overview Condition Best fitting model Dominance data generation GGUM Ideal point data generation GGUM Mixed data generation GGUM Small Sample (N=300) GGUM Large Sample (N=2000) GGUM Short Test (n=10) GGUM Long Test (n=37) GGUM

Results

Discussion (1) “GGUM fits better” - Confirm previous findings. - However, because regardless of the underlying response process, GGUM fits better than SGRM, it does not demonstrate that the response process or IRF/ORF is non-monotone. The previous assumption does not hold true. - Possible reason: Software (PARSCALE & GGUM) manifest models differently Better fit in small samples, especially for SGRM - Explanation: chi-square is sensitive to sample size

Discussion (2) Examine similarities of theta metrics - Negative correlation between theta estimates from GGUM and those from SGRM TRUE 1. 000 SGRM 0. 928 1. 000 GGUM -0. 923 -0. 995 GGUM 1. 000

Discussion (3) Scaling issue GGUM: - Reverse the estimate - Add a constant in scaling

Thanks!