Making measurements Ben Whalley Online code XXXXXX What
Making measurements Ben Whalley Online code: XX-XX-XX
What we’re doing 1. Develop a causal model 2. Create a self-report measurement for one construct 3. Learn to process and visualize the data we create 4. Does the construct predict academic achievement? 5. Check: Do our predictions vary depending on the context? 6. Interpret and report these analyses appropriately
Assessment XXXTODO • Last week: • Part 1: Q’s 1, 2, 3 • After this week: • Part 1: Q 4, and start on 5 and 6 • Next week: • Part 2: Q’s 1, 2, 3, and more….
Hidden causes
Boring, 1923 / Formative measurement model
How tall am I? “True scores” and reliability X = T + e. X My height = T + e 1 + e 2 + e 3 ~= 0 https: //socialresearchmethods. net/kb/reliablt. php
Face validity
Diversity
Correlated errors
Common cause of shared error Other examples include scaling effects or “method error”
True score
Repetition can increase bias too!
Task 1: How relaxed are you? • Make a list: Identify at least 10 observations you could make which would tell you if someone is relaxed. • Discuss: Could some of the observations also relate to other constructs, or be caused by something else? Could our observations be subject to any biases? • Individually: Rank-order the list of observations. Put the most relevant observations first, and the least-relevant last. • As a group: Compare your rankings. Do you sometimes disagree?
Task 2: Measurement model • Pick a construct from your model • It should have a direct link to attainment • It should also act like a ‘hidden cause’, as discussed • Draw a box for this construct in the middle of the page • Link it to all the observations you think you could make for it • Make links to other constructs in your model, if you think they might be caused by more than one thing.
Creating questionnaire items (De Coster 2005) • Simple • Do you think that the technical service department is prompt and helpful? • Only people in the military should be allowed to personally own assault rifles. • Clear • Avoid jargon, use plain English • How do you feel about your parents? • Unbiased/not leading • Do you really think Boris Johnson should be the UK prime minister? • Common structure • Repeat scale format and labels where possible • Questions randomizable?
Task 3: Create a questionnaire • Brainstorm: what could you ask people which would reveal information about your construct/observations? • Write several questions for each aspect/facet of the phenomena you identify. • Questions should be either: • Yes/No • Likert 1 -7 (identify the scale labels) • Evaluate your items against the criteria in De. Coster (2005) • Aim for no more than 10 questions in total (for now)
Piloting questionnaires “Think aloud” protocol/cognitive interview • • • Read questions aloud (or allow participant to) As them to verbalize their thinking in real time Explain what they think the items mean, what it is “getting at” Explain what factors influence how they will answer the question Identify confusing or misleading questions…
“Think Aloud” example from Willis, 2015. • Interviewer (reading target question): In the past 12 months, have you been treated unfairly when out in public because you are [self-reported race: e. g. , White, Black, Vietnamese, Asian. . . ]? • Participant (45 -year old, male, Caucasian): Let’s see—the past 12 months? Well, not really, because —you know—I’m White, and I don’t tend to be treated badly because of my race—it’s not like I’m Black or Hispanic or have to worry about being thought of as a terrorist when I’m really a Sikh but they think I’m Muslim or something. There was one time I was on the bus and. . . I was getting on. . . and bumped into a Black guy who had a broken arm or. . . was injured or something because he said “watch out, ” and then something I couldn’t really hear
Task 4: Think aloud As practice (from Chapter 3, Willis 2015): Try to visualize the place you live and think about how many windows are in that place. As you count up the windows, tell me what you are seeing and thinking about. (mute your mic first!)
Homework • Find your unique participant ID (see github worksheet) • Do the MCQ
This is the MCQ we will use as our primary outcome (a proxy for academic achievement). https: //forms. office. com/Pa ges/Response. Page. aspx? id= 6 c 3 VPu. DGk 2_07 skfg. Yb 8 WEZK 5 mp. Nnx. Dgl. U_t 85 t. VHt. UNVR KVDU 1 S 1 I 1 M 1 A 2 T 1 BPOFUy. T DIy. TE 9 IWC 4 u
Thursday/Friday Sessions • Pilot your questionnaire using Think Aloud (swap groups) • Finalise items, with reference to De. Coster and other resources • Put them online • Post the links to Psych: EL (read the instructions carefully) We will pick 3 questionnaires for everyone to complete for the final assessment analyses.
TARA LEAD SESSION SLIDES
Measurement Session 2 Online code: XX-XX-XX
Task 5: (preparation) • Type your groups’ questions into a shared document (make sure every student has their own copy if sharing this work) • Don’t worry if the questions are not perfect yet – you can still work on them! 15 minutes max
Task 5: Think aloud (see github pages) https: //benwhalley. github. io/rmip/measurement. html#tara-led-workshop-tasks • Divide into random pairs of “experimenters” and “participants” (i. e. don’t pair with someone in your current group) • Please listen to the TARAs and respect social distancing if you need to move places when doing this. • Participants follow the ‘talk aloud’ protocol for each question • Experimenters record/take notes 40 minutes max
Task 6: Put your questionnaire online • Make any last-minute tweaks to your questions/phrasing based on your piloting • Use Microsoft Forms 365 • Recommend doing at least a dry-run in the Workshop • See notes on Github for guidance • COMPLETE BY END OF PLAY NEXT TUESDAY AT THE VERY LATEST • You MUST complete the Psych: EL task too to send us the urls
- Slides: 29