Scenariobased scales Integrating Guttman facet theory and Rasch

Scenario-based scales: Integrating Guttman facet theory and Rasch measurement Larry H. Ludlow, Ph. D. Boston College Lynch School of Educational Research, Measurement, and Evaluation ludlow@bc. edu http: //www. bc. edu/content/bc/schools/lsoe/academics/departments/erme. html Prepared for: Institute of Education Higher School of Economics Moscow, Russia October, 2014

Variable Map for Af. L Did this work roughly enough as intended? Levels of Practice Peer-and Self-Assessment Learning Intentions and Success Criteria Feedback Questioning and Classroom Discussion BUT what is the “meaning” for a given teacher? 2

Average Item Response |P=50% 1 | |P=50% at Average|P=50% 5 | on each Item | . . ## + | | | + | | | | + | | | | 5 . ######## + | | + | PSA 5 | | | PSA 1 PSA 2 + | | 4 . ########## + | | | | + PSA 4 LISC 5 PSA 3 FB 5 FB 4 QCD 3 LISC 4 + LISC 3 | | QCD 4 | | + . ##. #. . . | | | | + | | | | FB 2 LISC 1 QCD 5 FB 3 | QCD 4 | FB 1 LISC 2 | QCD 1 + | LISC 3 | | QCD 4 | + | | PSA 1 PSA 2 | | PSA 4 + LISC 5 PSA 3 | FB 5 FB 4 | QCD 3 | LISC 4 | FB 2 LISC 1 QCD 5 + FB 3 | QCD 4 | FB 1 LISC 2 | QCD 1 | + LISC 3 | | QCD 4 | | + | | | | + | | 6 Now we can show “average” response levels showing how teachers responded. . . 3 2 + | | | | + | | 1 | PSA 5 Embedded PSA 5 This is the level to which we would like to see them advance— 50% prob of a 6. PSA 1 PSA 2 PSA 4 LISC 5 PSA 3 FB 5 FB 4 QCD 3 LISC 4 FB 2 LISC 1 QCD 5 FB 3 QCD 4 FB 1 LISC 2 QCD 1 Established Emerging Sporadic Most of the teachers are at these two levels. A few teachers are at this level. Never Fortunately, no one is at this level— 50% prob of a “ 1”. 3 Don’t understand

Teachers’ Af. L Benchmark Descriptions Very useful description of how to interpret average scores. Why not make a “scenario description” exactly like we have tried to do here? 9/29/2020 Ludlow-ERME 4

Personal Engagement According to Kahn (1990): • Personal engagement represents “the simultaneous employment and expression of a person’s ‘preferred self’ in ways that promote deep connections to one’s work (whatever form that might take, paid or unpaid) and to other people. • Such engagement leads to active, full, and satisfying involvements rather than obligatory, passive, or emotionally anemic ones. • We become personally engaged when we find that it is meaningful and safe to express our full selves and when we are psychologically ready to do so. (p. 700). 5

Our Broader Definition of Engagement • A positive, enthusiastic, and affective connection with a role that both motivates individuals to invest their valuable resources and simultaneously energizes them. Ranges across lower levels through higher levels. • Characterized by persistence, energy, focus, and interest experienced while enacting a role. Highly correlated and range from lower to higher levels. • A unidimensional construct that cuts across different adult roles we may assume. 6

What is psychological engagement? • What does the concept of “full” engagement or “less • • than full” engagement look and feel like? Is it possible to be “disengaged”? Is this construct on a continuum? And, if so what is on each end of that continuum? How do you measure engagement? Can it be measured across roles or is this a role-specific concept?

Conceptual Model 8

Productive Engagement in Later Life Paid Employment Caregiving Currently working for pay Currently involved in helping a friend or relative age 18 or older who has trouble taking care of themselves because of a physical or mental illness, disability, or for some other reason--includes caring for them directly or arranging for their care by others Volunteering Informal Helping Did volunteer work in the last 4 weeks for any of the following: • a church, synagogue or other religious organization • a school or educational organization • a political group or labor union • a senior citizen group or related organization • any other national or local organization, including United Fund, hospitals, and the like Provided unpaid help in the last 4 weeks to someone who does not reside with them (excluding ill/disabled), including: • providing transportation, shop, run errands • helping with housework, car maintenance, etc. • providing childcare 9

Sample • Conducted in September 2012 through Survey Sampling International (SSI). • Stratification for activity involvement resulted in equal numbers of respondents per activity category: • • 120 paid workers 120 volunteers 120 caregivers 120 informal helpers • 480 individuals age 50 to 89 • • • Mean age 63. 18 (SD: 8. 32) 60. 6% Female 38. 8% Bachelors degree or higher 90. 9% Caucasian 57. 9% Married/cohabiting • 92. 1% Live independently 10

Classical Test Theory (CTT) vs Item Response Theory (IRT) Classical Test Theory Item Response Theory • “Traditional” approach since 1904 • “Modern” approach since 1980’s • Based on notion of “true score” X=T+E • Based on probability notion of “expected response” for a person to an item • Emphasis is on use and application of “total score” • Emphasis on meaning and interpretation of individual items • Typical procedures are Cronbach alpha, test-retest reliability, factor analysis • Many different statistical models for determining “expected responses” • Simple items, often using Likert scoring— SA to SD • A particularly powerful model for item/scale development is the Rasch measurement model. 11

Rasch Model Measurement Principles When measuring a construct, items should define a construct with these characteristics: • Variation: wide range of beliefs, attitudes, opinions, abilities • Unidimensional: address a single, common attribute of the construct • Hierarchical: progress from “easier-to-difficult” to endorse or accomplish items/tasks • Continuum: form a uniform continuum across this hierarchical, unidimensional variation 12

Structure of a “variable” High Engagement Every variable stretches across a continuum: levels of knowledge, cognitive abilities, affective characteristics (e. g. engagement) We construct a deliberate hierarchical sequence of items, like steps on a ladder, that show where a person is located and what can then be expected to advance along that continuum (or up the ladder). Low Ludlow-ERME 13 Engagement

How to turn Rasch principles into scenarios? Facet Theory (L Guttman) • Methodological approach to reveal how we form beliefs, attitudes, perceptions such as “engagement”. • Each such perception is based on many influences, circumstances, factors, dimensions, considerations. o Each of these influences is called a “facet”. o Each facet is made up of different levels/elements. • In essence, FT or facet design, reveals the facets and their elements. 14

Sentence mapping (Guttman) • A visual, grammatical device to link the elements of the facets. • All possible combinations of one element from each facet are formed. • We have 4 facets, each with 3 levels for a total of 3 x 3 x 3 = 81 combinations o o These combinations are used to form sentences. These sentences are paired to become our scenarios. Our facets and elements are: X’s Role Engagement Level Captured by His/Her Interest Focus Energy Perseverance (fascinated) (difficult to tear away) (gets energized) (gives all) (moderate) (does not care) (completely checked out) (does not invest) (almost always gives up) 15

Choosing Scenarios All 81 combinations are impractical, how do you choose? • Randomly select from the 81 combinations of Ii, Fi, Ei, Pi. • Extreme groups contrast procedure (B Bloom). We chose the following combinations 3 from the extreme positive (I 1, F 1, E 1, P 1) group, 3 from the extreme negative (I 3, F 3, E 3, P 3), and; 3 from the middle (I 2, F 2, E 2, P 2). Level Interest Focus Energy Perseverance 3 (fascinated) (difficult to tear away) (gets energized) (gives all) 2 (moderate) 1 (does not care) (completely checked out) (does not invest) (almost always gives up) 16

How did we start thinking about the enagagement construct?

Level Interest Level Focus 3 Identifies strongly 3 3 Fascinated 3 2 Level Energy Level Perseverance Person total Difficult tear self 3 away Gets Energy 3 Gives All Anan (9) 12 3 Intensely focused 3 Invests much energy 2 Persists when difficulties arise Melissa (7) 11 Enthusiastic 2 Pretty focused Goes above and 2 beyond Deals with challenges Tina (1) 10 Interested 2 Mind wanders 2 occasionally but pays attention Does what it takes Handles difficulties Clair (6) 8 3 2 How did we think about the “level” of the scenarios? 2 Somewhat interested 2 Generally focuses 2 Does what is required 2 Keeps at it when Stan (2) difficulties arise 8 2 -1 Indifferent 1 Often thinks other things 2 Little desire do more than required 2 Forces self keep Elyssa (4) going when difficult 7 1 Tired of work 1 Thinks about other things 2 Does not invest much energy 1 Does not go out Jackie (8) of way when difficulties arise 5 1 Unenthusiastic 1 Usually thinking 1 about other things Does not go out 1 of way Gives up when any effort required Larry (5) 4 1 Does not care 1 Completely checked out Does not invest any energy Almost always gives up when any effort required Jaime (3) 4 1 1

Scenarios (using Work as the “Role”) On a typical day , are you… o Much more engaged than X o More engaged than X o About as engaged as X o Less engaged than X o Much less engaged than X 9) Anan identifies strongly with his work (in a positive way) and sometimes gets so wrapped up in what he is doing that it is difficult to tear himself away. He gives all of himself to his work and finds that he gets energized from doing so. 7) Melissa is fascinated by her work and is usually intensely focused on whatever she’s working on. She is willing to invest much energy in order to do a good job and she persists when difficulties arise. 1) Tina feels enthusiastic about her work and is pretty focused on the task at hand. She goes above and beyond what is required and when challenges arise, she deals with them. 6) Clair is interested in her work and pays attention to whatever she’s working on. She does what it takes and handles difficulties when they arise, though her mind wanders occasionally. 2) Stan is somewhat interested in his work and generally focuses on whatever he’s working on. He does what is required and keeps at it when difficulties arise. 4) Elyssa feels indifferent about her work and often thinks about other things. She has little desire to do more than is required and has to force herself to keep going when things get difficult. 8) Jackie is tired of her work and usually thinks about how much she would rather be doing something else. She does not invest much energy in what she does and doesn’t go out of her way when difficulties arise. 5) Larry feels unenthusiastic about his work and is usually thinking about other things. He does not go out of his way to get tasks done and tends to give up when any effort is required. 3) Jamie does not care about his work and is completely checked out. He does not invest any energy at all in what he does and almost always gives up when effort is required. 19

“Variable Map” Work Higher level of engagement High scoring people Lower level of engagement MEASURE Person - MAP - Item <more>|<rare> 8 # + | | 7 + | | 6 + | | 5 + | | 4 +. # | |. ## | 3 + ## | | ##### | SC 9 #### | 2 +. ##### | # M| SC 1 SC 7 ##### | 1. ## + | | ### | SC 2. ## | SC 6 0. # +M # | |. |. | -1. + | SC 4 SC 8 |. # | | SC 5 -2. + SC 3 | | -3 + <less>|<frequent> EACH "#" IS 2. EACH ". " IS 1. Harder scenarios 9) Identifies strongly with “X” (in a positive way)…gets so wrapped up it is difficult to tear self away…gives all of self to and gets energized from doing so. 7) Fascinated and intensely focused…willing to invest much energy…persists when difficulties arise. 1) Enthusiastic and pretty focused…goes above and beyond and deals with challenges 2) Somewhat interested and generally focused…does what is required and keeps at it when difficulties arise. 6) Interested and pays attention…does what it takes and handles difficulties, though mind wanders occasionally. 4) Feels indifferent and thinks about other things…little desire to do more than is required…forces self to keep going when things difficult. 8) Tired and usually thinks about how much rather be doing something else…does not invest much energy and doesn’t go out of way when difficulties arise. 5) Unenthusiastic and usually thinking about other things…does not go out of way…gives up when effort is required. 3) Does not care…completely checked out…does not invest any energy…gives up Easier scenarios 20

MEASURE Work But what does a score actually mean? For example, what is the difference between a person who scored 38 and one who scored 22? Where are those scores and what is their “engagement difference? Person - MAP - Item <more>|<rare> 8 # + | | 7 + | | 6 + | | 5 + | | 4=41 +. # | |. ## | 3=39 + ## | | ##### | SC 9 #### | 2=35 +. ##### | # M| SC 1 SC 7 ##### | 1=31. ## + | | ### | SC 2. ## | SC 6 0=27. # +M # | |. |. | -1=22. + | SC 4 SC 8 |. # | | SC 5 -2=19. + SC 3 | | -3 + <less>|<frequent> EACH "#" IS 2. EACH ". " IS 1. 9) Identifies strongly with “X” (in a positive way)…gets so wrapped up it is difficult to tear self away…gives all of self to and gets energized from doing so. 4) Feels indifferent and thinks about other things…little desire to do more than is required…forces self to keep going when things difficult. 8) Tired and usually thinks about how much rather be doing something else…does not invest much energy and doesn’t go out of way when difficulties arise. 21

MEASURE Person - MAP - Item WORK 8 7 # 5 4. #. ## # 3 ## ######## 2 0 -1 . ##### # #####. ## ###. # #. . # -2 -3 Person - MAP - Item INFORMAL HELPING + | | | | + | | | | + | | M+ | | | | +M | | | | + VOLUNTEER. ## + | | | | + | | + | M| | | + | | | | +M | | | | + How well did our predicted scenario order come out? 6 1 + | | | | + | | | | + | M| | | +M | | | | + Person - MAP - Item CAREGIVING . . . +. . . # ## SC 9 SC 1 SC 7 G A P SC 2 SC 6 SC 4 SC 8 SC 5 SC 3 EACH "#" IS 2. EACH ". " IS 1. G A P . ##### #####. . # # #. . | | | | + | | | M| + | | +M | | | | + . # ####### SC 9 . ####### SC 7 SC 1 G A P SC 6 SC 2 G A P SC 4 SC 8 SC 3 SC 5 EACH "#" IS 2. EACH ". " IS 1. . ######. ## ####. . ###. ## ##### SC 9 ###### SC 1 SC 7 G A P . ######. # SC 2. ##. SC 6 # G A P . . SC 8 SC 4. SC 3 SC 5 EACH "#" IS 2. EACH ". " IS 1. SC 9 SC 1 SC 7 G A P SC 2 SC 6 G A P SC 8 SC 4 SC 3 SC 5 EACH "#" IS 2. EACH ". " IS 1. 22

MEASURE Person - MAP - Item WORK 8 # 7 + | | | | + | | | | + | M| | | +M | | | | + Person - MAP - Item CAREGIVING + | | | | + | | Person - MAP - Item INFORMAL HELPING . + | | | | + | | | | M+ | | | | +M | | | | + Person - MAP - Item VOLUNTEER. ## + | | | | + | | + | M| | | + | | | | +M | | | | + Where is the average level of engagement? 6 5 4. #. ## # 3 ## ######## 2 1 0 -1 . ##### # #####. ## ###. # #. . # -2 -3 . . +. . . # ## SC 9 SC 1 SC 7 SC 2 SC 6 SC 4 . ##### #####. . # #. SC 8 SC 5 SC 3 EACH "#" IS 2. EACH ". " IS 1. . . # # #. . | | | | + | | | M| + | | +M | | | | + . # ####### SC 9 . ####### SC 7 SC 1 #####. ## ### SC 6 SC 2 . # ####. . #. SC 4 SC 8 SC 3 SC 5 EACH "#" IS 2. EACH ". " IS 1. . . ###. ## ##### SC 9 ###### SC 1 SC 7. ######. # SC 2. ##. SC 6 #. . SC 8 SC 4. SC 3 SC 5 EACH "#" IS 2. EACH ". " IS 1. SC 9 SC 1 SC 7 SC 2 SC 6 SC 8 SC 4 SC 3 SC 5 EACH "#" IS 2. EACH ". " IS 1. 23

MEASURE Person - MAP - Item WORK 8 # 7 6 5 4. #. ## # 3 ## ######## 2 1 0 -1 . ##### # #####. ## ###. # #. . # -2 -3 . + | | | | + | | | | + | M| | | +M | | | | + Person - MAP - Item CAREGIVING + | | | | + | | Person - MAP - Item INFORMAL HELPING . + | | | | + | | | | M+ | | | | +M | | | | + Person - MAP - Item VOLUNTEER. ## + | | | | + | | + | M| | | + | | | | +M | | | | + Where is the same score across the 4 roles? . +. . . # ## SC 9 38 SC 1 SC 7 SC 2 SC 6 SC 4 . ##### #####. . # #. SC 8 SC 5 SC 3 EACH "#" IS 2. EACH ". " IS 1. . . # # #. . | | | | + | | | M| + | | +M | | | | + . # ####### SC 9 . ####### 38 SC 7 SC 1 . ######. ## ### SC 6 SC 2 . # ####. . #. SC 4 SC 8 SC 3 SC 5 EACH "#" IS 2. EACH ". " IS 1. . . ###. ## ##### SC 9 38 SC 1 ###### SC 7. ######. # SC 2. ##. SC 6 #. . SC 8 SC 4. SC 3 SC 5 EACH "#" IS 2. EACH ". " IS 1. SC 9 38 SC 1 SC 7 SC 2 SC 6 SC 8 SC 4 SC 3 SC 5 EACH "#" IS 2. EACH ". " IS 1. 24

Conclusions & Implications • Was it possible to offer a new definition of engagement? Yes! • Was it possible to construct meaningful and useful scenario items that captured one’s level of engagement? Yes! • Was it possible to define engagement and construct scenario-based scales that were invariant across the adult roles of work, caregiving, informal helping and volunteering? Yes! 25

Patterns of Practice for Equity (POPE) The literature clearly identified the importance of teachers: • Selecting worthwhile content and designing learning opportunities aligned to valued outcomes • Connecting to students’ lives and experiences • Creating learning-focused, respectful and supportive learning environments • Taking an inquiry stance through using evidence to scaffold learning, and improve teaching, • Taking responsibility for further professional engagement and learning, and • Challenging inequities.

Teaching for Equity Enactment Scenarios We potentially have six interconnected components (aspects/facets) of patterns of practice for equity but this number may be modified. We will initially construct scenario-type items for each selected pattern of practice for equity facet, defining enactment of that dimension from excellent, through moderate, to limited (high, medium, low). We will use the following steps to develop and test the Rasch tool: • Select the patterns of practice for equity facets to be included in the tool and identify the elements of each facet. Typically the identification of facets and their elements requires considerable iterations and revisions. • Select an exemplar term, plus synonyms for this term, for each pattern of enactment (excellent, moderate, limited) of the selected facets. • Construct mapping sentences. These contain the formal elements of the facets and the informal components of natural language in order to contextualize the facets. • Determine which patterns of enactment will form part of the tool. We will combine the exemplar terms from one scale with those of the other facets. If we select 6 facets, each with three levels, there are potentially 36 = 729 possible combinations. We will, instead, use an “extreme groups contrast” procedure with three scenarios from the higher facet level, three from the moderate level and three from the lower level. We intend to follow this approach as it maximizes the possibility of working with scenarios that capture the boundaries of patterns of practice for equity enactment levels.