ITC Conference Conference Winchester 2002 Computerbased Testing Usability

ITC Conference, Conference Winchester, 2002 Computer-based Testing Usability of Psychometric Admeasurements Dr. J. M. Müller University of Tübingen, Germany http: //www. joergmmueller. de/default. htm

Overview 1. Introduction: Formal test descriptions in practice 2. Definition of usability in the context of test description 3. Illustrating problems: Reliability 4. Criteria of usability: foundation, scaling, general attributes 5. Two examples of enhanced usability: NDR and PDR 6. Summary

Introduction: Psychometric admeasurements in practice today and tomorrow 1. Test users often use poor quality tests (e. g. Piotrowski et al. ; Wade & Baker, 1977) Psychometric knowledge (Moreland et al. 1995)/Competence approach (Bartram, 1995, 1996) 2. What should be described? CBT: Criteria for software usability (ISO 9241/10, 1991; Willumeit, Gediga & Hamborg, 1995) and further criteria: platformindependence, possibility of making own norm banking, protection) 3. How should it be described? 4. “Good practice” guidelines and standards are based on quality criteria (e. g. Standards for educational and psychological Testing, PA, 1999; International Guidelines for testing, ITC, 2000) Quality Supply Quality Demand

Definition of Usability Scope of usability: Usability in the context of psychological testing concerns all important kinds of information for test users to describe a test for various purposes and the ways to communicate them. This includes test manuals as well as a formal test descriptions with the help of psychometric admeasurements. Aim of usability: The product or effect of good usability is that any test user finds all necessary information quickly and in a proper standardized form, ready to use for answering the questions of the test users to enable them to decide whether a test is an appropriate help for the diagnostic question. Frame of usability: Quality assurance in the context of psychological testing refers to test construction, test translation, test description and the use of tests in practice. Methods to enhance quality control can contain guidelines for test use, standards for test description, etc. Usability is a strategy to enhance quality on the level of formal description. Consequences of usability concern the reengineering of formal test description,

CTT Generalizability Theory Dimensional construct IRT Measurement of error nonspecific misclassification Categorical construct Sensitivity TP/(TP+FN) Specificy TN/(TN+FP) Kappa Interrater % or SMC Kappa Reclassification Phi-Coefficient Yule‘s Y Standard error score Information-function Model-Fit Likelihoods intraclass-correlation Cronbach‘s Alpha Spearman-Brown prophecy formula Kuder-Richardson‘s Formula 20 Spearman Correlation Retest Pearson correlation Standard error of a score Indices of measurement of error specific misclassification

Reliability Y/ Kappa/ Phi Korrelation Phi Standard error score Kappa Sensitivity TP/(TP+FN) Specificy TN/(TN+FP) Kappa Interrater % or SMC Kappa Reclassification Phi-Coefficient Yule‘s Y Information-criteria Information-function Model-Fit Likelihoods intraclass-correlation Cronbach‘s Alpha Spearman-Brown prophecy formula Kuder-Richardson‘s Formula 20 Spearman Correlation Retest Pearson correlation Standard error of a score Relationships between indices of error of measurement

Top-down vs. bottom-up strategy to develop a coefficient Practitioner‘s point of view Interpretation of the score (operational meaning) Defining the operational meaning scale (correction) Scale definition Algorithm Index: Defining the influencing factors Index: Generic formula test theory/statistic Specification of within a test theory Scientist‘s point of view

Rescaling reliability: Number of distinctive results (NDR) (Wright & Master, 1982; Lehrl & Kinzel, 1973; Müller, 2001) Formula Test score distribution Rang R x 1 x 2 critical critical difference difference R = test score range k = critical difference

Criteria of usability formal quality criteria (modified from Müller, 2001, 2002 a, b; Goodmann & Kruskal, 1954) Foundation 1. 2. 3. 4. 5. Unambiguous operational meaning Unambiguous formal definition Broad application area Relevant dependencies Independent of irrelevant factors Scale Definition 1. 2. 3. Meaningful scale unit, that implies: • Interval scale • Positive values • Defined range of values Comparable to the reference scale Significant scale unit that implies a minimum of observations (Nmin) Global attributes in using 1. Relevance 2. Informative (not redundant) 3. Predictable for the test user (nominal/actual value comparison) 4. Easy to learn 5. Easy to utilise 6. Fisher(1925) criteria of estimating

NDR at work. . . NDR = 2 NDR = 5 NDR = 10 r =. 50 r =. 92 r =. 98 Distribution of reliability coefficient Distribution of NDR coefficient Conclusion: many precise tests Conclusion: some precise tests

Probability of distinctive results (PDR) Complete score comparison of pairs Formula Gaussian distribution Rectangular distribution shows a 60 % probability shows an 80 % probability to distinguish two test scores

PDR: Simulation study Performance to separate test scores with respect to reliability and score distribution PDR Reliability

PDR: Example Subscale ‚Resignation‘; Stress-Coping-Questionnaire SVF-KJ; Hampel, Petermann & Dickow, 1999; N=1123 Subscale ‚Unsicherheit‘ Symptom Check List (Derogatis, 1977; German Version Franke, 1995; N=875 r = 0. 81 PDR = 41. 6 % PDR = 30. 6 %

Reviewing NDR and PDR 1. NDR and PDR can be derived in any test theoretical model – there is progress in the application area. 2. NDR and PDR have an easy to understand operational meaning 3. NDR and PDR are predictable for the test user for the nominal/actual value comparison NDR and PDR serve as examples of how to develop more usable formal test descriptions

Summary 1. 2. Usability is a possible strategy with explicit and observable criteria, for improving formal test descriptions – and strengthening indirectly the role of guidelines and standards. With NDR and PDR two easy to understood coefficients have been proposed, the application of which in is progress in several test theoretical models.

Thank you for your attention!

Medicine: Effect-size measures Scientific coefficient (Cohen, 1988) Practitioners coefficient NNTs [Number-Needed-to-Treat] the number of patients who need to be treated to prevent 1 adverse outcome. Taken from EBM Glossary Evidence Based Medicine Volume 125 Number 1

Measuring in technical fields: Solutions from engineering The is a German Norm DIN 2257 on how to measure the physical length of an object and how to report the result. The norm allows as output only values with statistical evidence.

Criteria of usability formal quality criteria for NNT Foundation 1. 2. 3. 4. 5. Unambiguous operational meaning Unambiguous formal definition Broad application area Relevant dependencies Independent of irrelevant factors Scale Definition 1. 2. 3. Meaningful scale unit, that implies: • Interval scale • Positive values • Defined range of values Comparable to the reference scale Significant scale unit, that implies a minimum of observations (Nmin) Global attributes in using 1. 2. 3. 4. 5. 6. Relevance Informative (not redundant) Predictable for the test user (nominal/actual value comparison) Easy to learn Easy to utilise Fisher‘s (1925) criteria of estimating

Criteria of software usability (from Willumeit, Gediga & Hamborg, 1995) Questionnaire on the basis of ISO 9241/10 (Iso. Metrics) to evaluate the following dimensions: 1. Suitability for the task 2. Self-descriptiveness 3. Controllability 4. Conformity with user expectations 5. Error tolerance 6. Suitability for individualization 7. Suitability for learning

KR 20 and Cronbach Kuder-Richardson-Formel KR 20 i item pi relative Anzahl von 1 qi relative Anzahl von 0 (aus Cronbach, 1951) Cronbachs Alpha c Anzahl der Variablen si 2 Varianz der Variablen i stot 2 Varianz der Summe

Formula to the error of measurement in categorial constructs Cohen‘s Kappa Weiter 16 Maße zur Konkordanz zweier Messungen für binaäre Daten verglichen Conger & Ward (1984) Yule n Vierfelderinterdependenzmaß n Q-Koeffizient Phi-Koeffizient n n Abhängigkeit von Randsummenverteilung Abhängigkeit des Signifkanztests von N (Yates -Kontinuitätskorrektur, 1934) A 1 A 2 B 1 a c B 2 b d

Formula to the error of measurement in categorial constructs Frickes Übereinstimmungs-koeffizient SS: Quadratsumme innerhalb einer Person; max SS: maximal mögliche Quadratsumme innerhalb der Personen Punkt-biseriale Korrelation X=arithmetisches Mittel aller Testrohwerte XR=arithmetishes Mittel der Pbn mit richtigen Antworten sx=Standardabweichung der Testrohwerte aller Pbn N = Anzahl aller Pbn NR=Anzahl der Pbn, mit richtigen Antworten Tetrachrorische Korrelation A B C I 1 4 3 II 0 4 2 III 0 5 2

Formula to the error of measurement in CTT, IRT + prophecy-formula Spearman-Brown-Formel k= Faktor der Testverlängerung Rasch model CTT

Some Formula for the error of measurement in metric constructs reliability (Kelley, 1921) Pearson(1907) -Correlation Bravais (1846) Spearman‘s rho (1904) Kendalls Tau , 1942 (S=difference of pro- und inversionsnumber) 1 2 3 4 5 3 2 3 5 4

Non-linear Relationsship between reliability, NDR and the standard error score reliability NDR 1 NDR Standard error score

Item-Response-Theory (Fischer & Molenaar, 1994) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. Dichotomous raschmodel Linear logistic test model Linear logistic model for change Dynamic generalization of the raschmodel One parametric logistic model Linear logistic latent class analysis Mixture distribution rasch models Polytomous rasch Models Extended rating scale and partial credit models Polytomous mixed rasch models. . .

. . . more IRT (van der Linden & Hambleton, 1997) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. Nominal categories model Response model for multiple choice Graded response model Partial credit model Generalized partial credit model Logistic model for time-limit tests Hyperbolic cosine IRT model for unfolding direct responses Single-item response model Response model with manifest predictors A linear multidimensional model. . .

Formula of some IRT binomial model rasch model Birnbaum model Unfolding-model Latent-Class-model

Criteria of software usability (from Willumeit, Gediga & Hamborg, 1995) Questionnaire on the basis of ISO 9241/10 (Iso. Metrics) to evaluate the following dimensions: 1. Suitable for the task 2. Self-descriptiveness 3. Controllability 4. Conformity with user expectations 5. Error Tolerance 6. Suitable for individualization 7. Suitability for learning

Norm scales n

SCL-90 -R test score distribution

Simulation study about the relationsship between measures of association Normal distribution- equal marginals dichotome Measure of association A 1 A 2 B 1 a c B 2 b d Skewed distribution - unequal marginals Y/ Kappa/ Phi Linear relationship? correlation Q Measure of association Y/ Kappa/ Phi Phi correlation Y/Kappa/Phi correlation Q SMC Phi SMC Kappa

Efficiency in measuring Content: Concept: efficiency The less effort you need for the same amount of information, the more efficiency the test is efficiency = f(Information; effort) Indice: E = Amount of Information/Time Estimates: Information Theory (Shannon & Weaver, 1949)

Amount of Information of a signal: Chess example The scale unit ‚bit‘ can be understand as the minimal or optimal number of question‘s, to identify a signal out of quantity of alternatives. 1. Frage: linksrechts? 2. Frage: oben 3. Frage unten? 4. 5. 6. In the chess example you need at least (binary, 50 -50 chance) 6 question‘s that are 6 bit.

Rasch variances are a measure of the variability of person‘s within a dimension Schachspieler B 1: 2 A 1: 2 C 1: 2 Als 1: 2 Maßeinheit der 1: 2 Unterschiedlichkeit dient die Differenz der Gewinnwahrscheinlichkeiten.

Interpretable Rasch Variances 2. Gewinnwahrscheinlichkeiten Difference to solve a -> Lösungswahrscheinlichkeiten question or task Gegner -> Testaufgabe (Itemparameter) 3. Spielstärke -> Personenparameter 4. Differenz der Gewinnwahrscheinlichkeit definiert über den Logit des Raschmodells 1.

Empirical Evidence of the range of person parameters in rasch units AID Kubingen & Wurst Standardform Parallelform Alltagswissen 21, 1 21, 3 Realitätssicherheit 13, 3 13, 1 Angewandtes Rechnen 21, 7 20, 5 Eigenschaft Autor Ausdehnung Verbaler Intelligenztest Averbale Intelligenz Einstellung zur Sexualmoral Einstellung zur Strafrechtsreform Beschwerdeliste Räumliches Vorstellungsvermögen Umgang mit Zahlen bei Kindern Metzler & Schmidt Forman & Pieswanger Wakenhut Fahrenberg Gittler Rasch 11, 4 8, 2 8, 1 7, 2 6, 4 5, 9 3, 5

Usability criteria explanations • Relevant dependencies: Example: Reliability and test length, stability, . . . • Irrelevant dependencies: Example: Reliability and test score distribution • Displaying numbers: Integer, positive, predictable range • Meaningful scale unit • Familiarness: each new coefficient should distinctly more usable than the traditional

7. Linearität zur Unit-in-Change Erläuterung: ‚Linearität zur Unit-in-Change‘ - Im Falle der Messgenauigkeit betrifft dies die Beziehung der Reliabilität zum Messfehler. - Im Falle der Übereinstimmung betrifft dies die Beziehung von Yules Y zur Veränderung der Zellhäufigkeit a bzw. d. Korrelation/Reliabilität Standardmessfehler Yules Y Freq (Zelle a)

Evaluation the progress trough enhancing usability 1. 2. Formal test criteria are used more frequently for test selection Tests in practice are of higher quality

Ergonomics in psychological test selection Ergonomics Configuration of Environment Software conception Psychological diagnostic Designing a tool to fit in hand. Developing a program to be used intuitively Restrict a test description, that relevant information are ready to use

Integrating ergonomics in the formal test description Analysis of usage 1. 2. Usability criteria evaluation Human interface techniques test user Psychometric admeasurements test Formal test criteria are used more frequently for test selection Tests in practice are of higher quality

Ergonomics and the development of criteria of usability Requirement Analysis (Mayhew, 1999) User-Profile Task. Analysis Platform Capabilities/ Constrains Testuser Test selection Test theory

Top-down vs. bottom-up strategy to develop a coefficient Practitioner‘s point of view association P-R-E Interpretation of the score (operational meaning) Defining the operational meaning NDR none scale (correction) Scale definition SEDTTS CTT Algorithm Index: Defining the influencing factors Index: Generic formula test theory/statistic Specification of within a test theory Scientist‘s point of view f(me, score range, probability)