Understanding what makes a good questionnaire Reliability and

Understanding what makes a “good” questionnaire: Reliability and validity: COURSE QUESTIONS Dan Coletti, Ph. D. Pediatric Subspecialty Fellows Research Course April 18 th, 2017

Scenario: A fellow in Developmental and Behavioral Pediatrics is interested in learning more about how parents understand the risks and benefits of childhood immunizations. An exhaustive literature review reveals that no such measure currently exists. Therefore, a decision is made to construct a new 10 -item paper-and pencil questionnaire--the “Risks And Benefits of Infant Disease prevention (RABID) --to learn more about this construct.

The fellow includes the following item in the measure: Circle the number that describes how much you agree with this statement: “Parents who allow their innocent children to be subject to the ticking time-bomb of having a measles-mumpsrubella shot are: ” FOOLISH 0 1 2 3 1. What’s the best way to describe the measurement scale this item uses? A. B. C. D. Nominal Ordinal Temporal Interval 4 5 6 7 8 9 10 NOT So FOOLISH 2. The wording of this question… A. Might indicate a respondent is “faking good. ” B. Is “loaded” in content and tone C. Is likely to have poor testretest reliability. D. Will show floor effects.

3. The fellow administers the RABID to a sample of 100 parents on a Friday morning. The same parents take the exact same test again later that day. The association between scores in the morning and in the afternoon is 0. 89. This would be an example of acceptable: A. B. C. D. Interrater reliability Test-retest reliability Internal Consistency Face Validity

4. Two groups of parents are administered the RABID before and after two different workshops. The first group attends the workshop entitled “Autism and Childhood Illness: Dispelling Myths. ” The second group’s workshop is entitled “Decorating Ideas for your Young Child’s Bedroom. ” Here is the data: Mean Score BEFORE “Autism and Childhood Illness: Dispelling Myths”…. . Score AFTER the workshop: [higher scores show more positive attitudes] ……. 54. 23 86. 59 Mean Score BEFORE “Decorating Ideas for your Young Child’s Bedroom. ”……. Score AFTER the workshop: [no change in scores]………………. . . . 53. 87 53. 96 This set of findings is an example of A. B. C. D. Response sets. Demand characteristics and social desirability. Documenting convergent and divergent validity. Alternate-form reliability.

CORRECT ANSWERS: 1. D- Likert scales with numerical values provide interval -level data. 2. B- This is a loaded question written with an apparent bias against childhood immunizations. Even the scale anchors are loaded. 3. B- Scores on the morning administration are highly correlated with the scores on the afternoon administration—thus supporting acceptable test-retest reliability. 4. C- Convergent/divergent validity. Measure is sensitive to change after learning more about the construct of interest. Conversely, mean scores don’t change after participating in an un-related presentation.