1 B 6 Potential Internal Validity Threats in

1 B. 6 Potential Internal Validity Threats in One-Group, Pretest/Posttest Designs Maturation History Regression to the mean Attrition Testing Instrumentation

Attrition (aka Mortality) Threat A differential loss of participants from the various experimental groups � Textbook examples Campers – rowdiest campers leave Therapy clients – most depressed dropped out of therapy

Attrition Threat A cognitive psychologist is examining the effects of age on eyewitness memory. The participants watch a video of a theft. The participants are asked to perform an identification task immediately after viewing the video and are asked to perform the identification task a week later after receiving some misinformation. While 90% of the young adults return a week later only 50% of the older adults return a week later. What threat to internal validity do you need to worry about?

Attrition Threat To prevent threat – remove dropouts’ data from the pretest group average (A) If two people (noted by blue dots) drop out of a study, both of whom scored at the high end of the distribution on the pretest, the group mean changes substantially when their scores are omitted, even if all other scores stay the same. (B) If the dropouts’ scores on the pretest are close to the group mean, removing their scores does not change the group mean as

Testing Threat (aka Repeated Testing) A kind of order effect in which scores change over time just because participants have taken the test more than once � Textbook examples Practice effects To prevent threats Do not use a pretest � If use a pretest, use alternative forms of the test for pretest and posttest � Comparison group – testing effect can be ruled out if the treatment group shows a larger change than the comparison group �

Testing Threat (A) If there is no comparison group, it’s hard to know whether the improvement from pretest to posttest is caused by the treatment or simply by practice. (B) The results from a comparison group can help rule out testing threats. Both groups might improve, but the treatment group improves even more, suggesting that both practice and a true effect of the treatment are causing the improvement.

Testing Threat Professor Beach wanted to demonstrate to his cognitive psychology class how using retrieval cues can improve memory. First, he read a list of 16 words in random order to his class, and immediately afterwards he had them write down all the words they could remember. On average, they recalled about seven words. Then, he told them that the words could be sorted into four categories: automobiles, cutting instruments, sports, and fruits. Professor Beach encouraged his students to do this in their minds while listening to the list again, after which they would be given another chance to recall the words. On the second test of recall, students remembered, on average, 14 words. Can Professor Beach tell his students that this proves the effectiveness of using retrieval cues as a mnemonic strategy? Why or why not? Taken directly from: http: //psych. athabascau. ca/html/Validity/index. shtml

Instrumentation Threat (aka Instrument Decay) Any changes that occur when a measuring instrument changes over time from having been used before (e. g. , coders, interviewers) � Textbook examples Counselors became more tolerant of the campers’ rowdy behavior The therapist used different measures of depression at pretest and posttest and the pretest measure may have overestimated depression � Examples (not in your textbook) Interviewers’ interviewing skills may become better over time With practice, research assistants become better at attaching electrodes to participants’ scalps for an EEG study

Instrumentation Threat To prevent threats � Train interviewers/coders � Retrain interviewers/coders throughout the study � Train research assistants � Consider using a posttest only design � Make sure alternative tests are equivalent � Counterbalance tests used at pretest and posttest

Instrumentation Threat In an effort to help prevent childhood obesity, some researchers have developed behavioral interventions. In on school program, nutrition researchers wanted to teach children to make healthier food choices in the cafeteria. They developed a short training program that involved a 20 minute video along with a behavioral rehearsal in which students practiced choosing foods in a mockup cafeteria. To test the effectiveness of this video training program, the researchers designed a study with an Experimental Condition, in which school children in Grade 1 were exposed to the new program, and a Control condition, in which children watched a 20 -minute cartoon having nothing to do with nutrition. A Food Choice test measured the children’s behavior in their real school cafeteria. Each food line presented 4 choice points, in which a healthier option (e. g. , fruit) competed against a less healthy option (e. g. , vanilla pudding). By closely observing a child as they moved through the cafeteria line, a researcher (graduate student) marked down the number of healthy options a child selected. A child could get a score from 0 (child made no healthy choices) to 4 (child made the healthy choice at all choice points).

Instrumentation Threat Each child’s name was put into a computer, and using an online randomizer, student were assigned to the Experimental Group and to the Control Group. One day at school, research assistants showed each group their respective videos in separate classrooms. Two days later, the Food Choice test was conducted. For ease of record keeping, all Control Group children were tested at lunch first, then all the Experimental Group children were tested. The graduate student observed all of the children’s food choices. In the beginning they hid behind a cashier’s booth and strained to see the food line clearly. Later on, they decided they could see better by standing nonchalantly near the food line. The mean score for the children in the Control Group was 1. 0 and the mean score for the children in the Experimental Group was 3. 5. Can the researchers conclude that the 20 minute interactive video improved children’s food choices? Why or why not? Taken from King & Morling (2012)

Combined Threats Occasionally, in a pretest/posttest design, selection effects can be combined with � History threats (selection-history threat) � Attrition threats (selection-attrition threat) � Maturation threats (selection-maturation threat)