FRAMEWORKS FOR CONSIDERING ITEM RESPONSE DEMANDS AND ITEM

FRAMEWORKS FOR CONSIDERING ITEM RESPONSE DEMANDS AND ITEM DIFFICULTY KRISTEN HUFF COLLEGE BOARD STEVE FERRARA CTB/MCGRAW-HILL NCSA SESSION: THEORY AND RESEARCH ON ITEM RESPONSE DEMANDS: WHAT MAKES ITEMS DIFFICULT? CONSTRUCT-RELEVANT? JUNE 20, 2010 DETROIT

THESIS A coherent and comprehensive understanding of the interaction between items and examinees, and the controllable features of items that elicit predictable interactions is needed to: Design items that do a better job of measuring what we’re interested in knowing Design tests that are better suited to facilitate valid inferences about student performance More generally, bridge the gap between large-scale assessment and teaching & learning

OBJECTIVE To conduct research (& review past research) that informs the development of a framework, or conceptual structure, of item response demands. Research questions: How much do existing item response demand schemas cover current achievement constructs? What are the features of items that influence item difficulty? What student responses are triggered by different item features, and how do these responses influence item difficulty?

Item Design feature Item statistic

Item Design Reading comprehension # words Item reading level overlap between key and text statistic feature Math number of variables graphics fractions vs whole numbers

Item Design Item statistic

Item Design Item statistic

Item Design FRAMEWORK Item statistic

Item Design FRAMEWORK Item statistic

Intended cognitive complexity Simple Complex Easy “Good” item (Good design and OTL) Item design flaw or Exceptionally effective instruction? Or flawed assumptions? Or restriction in range? Hard Item design flaw or No OTL “Good” item or No OTL Observed item difficulty

Opportunity to Learn Easy Observed item difficulty Hard Taught Not Taught Effective Instruction Item design flaw or just tough stuff? No OTL

CONCLUSIONS & POINTS OF DISCUSSION Mapping the landscape of response demands, item features and the interaction between the two is messy, difficult work Achieving “coherent and comprehensive” frameworks needs to be higher priority in research Start now using draft in operational testing programs Need more effective and easier ways to gauge opportunity to learn for many reasons, but also to inform the work we are recommending