Lesson 4 Threats to Internal and External Validity

Lesson 4 Threats to Internal and External Validity Spring 2017 GED 500 Lesson 4

Lesson Objectives l l Given a description of a quantitative study learners will be able to correctly identify threats to validity (internal and external). Given a research article, learners will be able to judge the validity/believability of the results. GED 500 Lesson 4

What is validity? l The validity of an experimental research study is its degree of accuracy. Another way of saying it, is: The extent to which we (the researchers) measured what we said we were measuring. – The question is whether the results of the study came from the treatment or from other sources. And if the same results should be expected in different settings. GED 500 Lesson 4

What is validity? l We speak of validity terms of strong or weak. GED 500 Lesson 4

What is validity? l We can say we have strong validity if we are actually measuring what we say we are (that means there are no threats to validity in our study). GED 500 Lesson 4

What is validity? l We say we have no validity if we have even one threat. GED 500 Lesson 4

What is validity? l There are two types of validity: – – Internal validity is the degree to which confounding variables are controlled. External validity is the degree to which the results can be generalized. It is important to remember that we automatically have threats to external validity if we have any threats to internal validity. GED 500 Lesson 4

Internal Validity § § Confounding variables (factors not controlled in the study) are variables that influence the results. Suppose you are comparing two teaching methods. § § If you find a difference, did it come from the methods or from something else? Think about some confounding variables that might interfere in a study of two different teaching methods. Then go to the next slide. GED 500 Lesson 4

Internal Validity § Possible confounding variables in a study comparing teaching methods: §Teachers’ skills (if each method is taught by a different person) §Teachers’ motivation (if each method is taught by a different person) GED 500 Lesson 4

Internal Validity Another possible confounding variable: §Students’ abilities (Were the students in the two classrooms inherently different to begin with? ) GED 500 Lesson 4

Internal Validity Another possible confounding variable: § Enthusiasm (if the researcher is the teacher of one class, his/her enthusiasm may influence the outcome) GED 500 Lesson 4

Internal Validity And there are many more-- you probably thought of a bunch. (-: GED 500 Lesson 4

Internal Validity § Another example: A researcher is comparing two types of after-school programs with the goal of increasing learning (measured by test scores) § What might be some confounding variables that would interfere with internal validity? GED 500 Lesson 4

Internal Validity § Possible confounding variables when comparing after-school programs: § § § Qualifications/experience of the staff Activities provided for students (as long as “activities” isn’t the independent variable) Absenteeism Demographics of student participants How students are chosen to participate And the list goes on and on. GED 500 Lesson 4

External Validity § External validity is based on the accuracy when results are generalized to a different setting/group of people. (we use the word permissible) § When we say that results are generalizable, we mean that when research is conducted, other researchers will assume that their results would be the same if they use the same methods. GED 500 Lesson 4

External Validity § § For example, suppose a study shows that using word walls increases the writing skills of middle school students. Other middle school teachers and administrators would assume that they would have the same results if their students used word walls. GED 500 Lesson 4

External Validity § Problems with external validity often occur when a study uses a very specific setting or group of subjects. § In the middle school word wall example, suppose the research was based on students in a private school whose classrooms all had multiple writing word walls (e. g. , literature based, seasonal, parts of speech, etc. ). § What problems with external validity might that cause? GED 500 Lesson 4

External Validity § If you teach in an urban public middle school where only writing word walls were used, your results might not be the same. § § Your students may have different overall academic skills than those in a private school. Your students may not have as much access to supportive materials as students who own laptops. Your teachers may not have the same creative skills or the motivation to use word walls. And so on. GED 500 Lesson 4

Threats to Internal Validity § § § There are specific names for certain threats to validity. The next few slides will explain the ones you are responsible for knowing for this class. If you have a copy of the “custom” edition of the textbook I suggested, then turn to page 152 to see a list of Threats to Internal Validity. I am not requiring that you learn all of the threats in the textbook—just those in this Power. Point. GED 500 Lesson 4

Threats to Internal Validity – Selection (in the text the full name is Differential Selection of Participants, which is quite a mouth full! So, we’ll us Selection as the name. § § There is a threat to validity called selection if there is a systematic difference (between the groups) that caused the results. In other words, if the groups were different to start with, you would expect them to be different at the end, no matter what the treatment. GED 500 Lesson 4

Threats to Internal Validity Selection § § Example: You are trying out two different methods of teaching reading. One of the classes has a large number of English Language Learners, and the other class is all English-only Learners. You would expect the students in the two classes to have different reading abilities in English before the treatment, so they are likely to be different after treatment as well. GED 500 Lesson 4

Threats to Internal Validity Selection § The selection threat is always expected when intact groups are used (reminder: that would be a quasi-experimental group research design). GED 500 Lesson 4

Threats to Internal Validity Selection § So, what do you do to try and control that threat? § § § If at all possible, use random assignment to “equalize” the groups, use both a pretest and posttest to assess initial and final differences, and there are specific statistical analyses that can be used that take initial differences into account. (more about that in another lesson) GED 500 Lesson 4

Threats to Internal Validity - Testing § § A testing threat occurs when subjects learn something from the pretest that changes the effect of the treatment. Example: You are using the one-group pretest posttest design when studying the effectiveness of a geography tutorial. When taking the pretest, students notice that certain areas of the world are being emphasized. Why might this be a problem? GED 500 Lesson 4

Threats to Internal Validity - Testing § § If the pretest and posttest are exactly the same, students may pay attention to the emphasized information rather than using all features of the tutorial. When they take the posttest, they may do better just because they knew what to look for and not because the tutorial was effective. GED 500 Lesson 4

Threats to Internal Validity - Testing § How can you prevent the Testing threat? § § Giving a pretest is usually a good idea, so not giving one probably isn’t an option. It is a good idea to make the pretest and posttest equivalent, but not exactly the same. § § In that way, even if students pay particular attention to the pretest content, it will not help them on the posttest. One simple way to do this is to change the order of the questions on the test, or reword them slightly. Be careful though, if you change what the test is measuring or the format of the test very much, you will have the next threat. GED 500 Lesson 4

Threats to Internal Validity Instrumentation § § In experimental research, there is always some sort of assessment. The instrumentation threat occurs when the assessment is not valid or is inappropriate for the subjects or the situation or is changed from pretest to posttest to the extent that they are no longer measuring the same thing. § The validity and reliability of tests will be covered in a later lesson, but there are some non-technical factors to watch for. GED 500 Lesson 4

Threats to Internal Validity Instrumentation § Example: Suppose a teacher has designed a new video game to teach students about subtraction. If she makes up the subtraction test herself, there is no proof that the test measures what it is supposed to measure (the effect of the video game). GED 500 Lesson 4

Threats to Internal Validity Instrumentation § Example: A psychologist is testing a counseling method. He uses the MOTIV personality test for the pretest (it is an online test), but he conducts the posttest as an interview. GED 500 Lesson 4

Threats to Internal Validity Instrumentation § How can you prevent the instrumentation threat? § Use standardized tests, observations checklists, etc. whenever possible, making sure they have proven reliability and validity. Also, make sure the posttest is comparable to the pretest. § Note: Some standardized assessments require training before a person can administer them. They are also usually copyrighted, so they cannot simply be copied—they must be purchased. GED 500 Lesson 4

Threats to Internal Validity – Diffusion of Treatment § § Diffusion of treatment occurs when the treatment (teaching method, etc. ) being given to the experimental group is shared with the control group or the comparison group. This situation would likely result in a “no difference” finding, when otherwise there might have been a difference. GED 500 Lesson 4

Threats to Internal Validity – Diffusion of Treatment § § Example: A teacher is studying the effects of a method of teaching multiplication (rapping the times tables). Students in the experimental group teach the rap to other students during recess. If these other students are in a control group (who should not receive the treatment), the results of the study will be contaminated. GED 500 Lesson 4

Threats to Internal Validity – Diffusion of Treatment § How can you prevent diffusion of treatment? § § One way is to have comparison or control groups at different sites than the experimental group. If this is not possible, caution teachers and others who are administering the treatment not to share the methods being used until after the study is complete. GED 500 Lesson 4

Threats to Internal Validity – Maturation § § A maturation threat to validity occurs when subjects gain skills through naturally occurring biological changes that take place over time, instead of from a specific treatment. Example: A children’s soccer coach tries a new physical training method over three seasons. Children are much more coordinated in the third season. GED 500 Lesson 4

Threats to Internal Validity – Maturation § § Most maturation threats can be prevented by gathering data after shorter periods of time of treatment. The researcher can also use a control group of the same age subjects to compare maturation levels. GED 500 Lesson 4

Threats to Internal Validity – Mortality (often called subject attrition) § § The mortality threat to validity occurs when subjects do not participate for the entire length of the study. NOTE: It is natural to lose some subjects during an educational study. This threat only occurs if there is a systematic loss of subjects that affects the outcome. GED 500 Lesson 4

Threats to Internal Validity – Mortality § Example: You are using different teaching methods in two classrooms. You have given a pretest and found that the two rooms are approximately equal in ability. During the study, a magnet school opens and many high achieving students from one of the classrooms transfer to the new school. § This situation would unbalance the two classrooms and cloud the results. GED 500 Lesson 4

Threats to Internal Validity – Mortality § § § It is impossible to guarantee that you will not lose subjects and losing some subjects generally does not invalidate a study. It depends on how many subjects are lost—and why they left the study. Data from students who drop out can be eliminated, but that causes other issues and isn’t the strongest way to strengthen the validity of your study. There are statistical tests that can take into account initial differences. (more about those in another lesson) GED 500 Lesson 4

Threats to Internal Validity – Experimenter Effects § The behavior and attitudes of the experimenter can affect the outcome of a study. § § If a researcher has developed a new teaching method and is the one who will deliver it, it is difficult for him/her not to be extra enthusiastic! This is also true in the reverse. If someone does not believe that a method will work, but is required to use it, the results may be contaminated. GED 500 Lesson 4

Threats to Internal Validity – Experimenter Effects § § Example: You have developed a great way to teach reading through dance. Your students make significant progress. Is it your method or your attitude? Example: A veteran teacher is asked to use a new teaching method. She does not believe it will work. The students do not make much progress. Is it the method or the teacher’s attitude? GED 500 Lesson 4

Threats to Internal Validity – Experimenter Effects § How can you prevent this threat to validity? § § If the researcher developed the treatment, someone else can deliver it. The researcher (or others) can observe the treatment being administered several times during the study to assure that it is done appropriately. GED 500 Lesson 4

Threats to Internal Validity – Subject Effects § § This is such a common problem that there are subtypes; but for this class, you only need to know the broader name: Subject Effects. If you are interested in the subtypes, they are described on pages 157 -158 in the custom edition of the text. NOTE: questions I ask on the will be answered by Subject Effects, NOT the subtypes. GED 500 Lesson 4

Threats to Internal Validity – Subject Effects § This threat arises when subjects react to the study because they “think” one way or another about what is going on, instead of being impacted by the treatment. For example, if the researcher is the subjects’ beloved teacher, they may try harder to learn something to “help” or please the teacher, than they usually do for other teachers. GED 500 Lesson 4

Threats to Internal Validity – Subject Effects § Or, the control group may try harder to achieve the goal simply out of a spirit of competition. § It may look like there is no difference between the groups, when there might otherwise have been. GED 500 Lesson 4

Threats to Internal Validity – Subject Effects § On the other hand, some subjects might think that: “Those people (the experimental group) are getting something we (the control group) aren’t. We’ll just give up. ” § § The control group may feel bad/neglected and not try when the assessment is given. It may look like there is a difference between the groups, when there might otherwise not have been. GED 500 Lesson 4

Threats to Internal Validity – Subject Effects § The students are trying something new, and that fact alone makes a difference in how they act or work. § This novelty wears off eventually, but in the mean time a significant effect may result. If a study is relatively short there isn’t time for the novelty to wear off before assessments are given, making it look like the treatment was effective, when in reality it really wasn’t. GED 500 Lesson 4

Threats to Internal Validity – Subject Effects § Preventing subject effects can be difficult. § § § Due to the rules of human subject protection, it is unlikely that subjects will not be aware that they are in a study. Having control or comparison groups at a different site is one solution. The researcher may also want to observe behavior as part of the assessment in the study. GED 500 Lesson 4

Threats to Internal Validity - History § There is a history threat to validity if an extraneous event affects the outcome. Be careful that you don’t rely on what you think history means because in research methods, the event takes place during the study, not before (as in how we generally define history). GED 500 Lesson 4

Threats to Internal Validity - History § Example: A researcher is studying two methods of teaching reading over a period of four months. During this time, the school is locked down four times because of gang violence in the neighborhood. GED 500 Lesson 4

Threats to Internal Validity - History § Example: During a study of a new science curriculum using on-line resources, the school gets a new computer lab and all students get additional computeruse instruction for 6 weeks. This additional instruction is given by the new computer lab teachers. How is the teacher-researcher going to know which computer use (in-class science instruction: the independent variable) or the “extraneous” computer lab instruction) impacted the subjects’ assessment results? GED 500 Lesson 4

Threats to Internal Validity - History § How can you prevent history threats? § It’s pretty hard, unless you are psychic. GED 500 Lesson 4

Threats to Internal Validity - History § Sometimes researchers just have to start over if there is a great deal of interference with the study. That’s not something we want to do. So, it’s best to “control” this threat before it gets started and check to make sure nothing is planned at your school that might impact your study. GED 500 Lesson 4

Threats to Internal Validity – Statistical Regression § A statistical regression threat occurs when the researcher uses the results from a pretest with students scoring very high or very low. (this is a probability issue: the very highest have no where to go, but down and the very lowest have no where to go, but up) § Don’t get too concerned about this, it only becomes an issue when you use only the highest (or the lowest) scoring subjects—and it can easily be controlled by the research design you use or the statistical analyses your perform. GED 500 Lesson 4

Threats to Internal Validity – Statistical Regression § It is likely that the researcher hopes that the posttest scores will be higher because of the treatment. § § But, if students score very high on the pretest, there is no room for growth. And, if they score very low, they are likely to increase, just by chance. (it’s probability) GED 500 Lesson 4

Threats to Internal Validity – Statistical Regression I always ask my students at this point in the face-to-face class: “How many of you have ever gotten 100% correct on a test? ” (There always quite a few who raise their hands, indicating that they have. ) I next ask them how they think they would have done if they had taken the test again. (Assuming that not too much time had passed. ) No one ever thinks he/she would get the same 100%--because we believe there is no place to go, but down. And, probability supports that belief. The same is true for those close-to-zero test scores: no where to go but up. GED 500 Lesson 4

Threats to Internal Validity – Statistical Regression § Example: A graduate student is evaluating some geometry software he developed with a pretest and a posttest. He begins the study after students have already been taught about the geometry content that is the same as his software content. Students score high on the pretest and about the same on the posttest. Showing no growth between the pre- and posttests makes his software look ineffective. GED 500 Lesson 4

External Validity l l External validity is the degree to which the results can be generalized. That is, can you (in your situation) apply the results of a study you read about? There are two general types of threats to external validity: – – Population Ecological GED 500 Lesson 4

External Validity: population threat l Please be careful as I will only be asking you to learn and correctly use the terms Population and Ecological threats to external validity. GED 500 Lesson 4

External Validity: population threat l A population threat occurs when the study’s population is very different from yours. – For example, the method you want to use in your classroom (because you read it was very successful in a 5 th grade classroom and you teach 5 th grade) is peer tutoring. But, the study your read about took place in an urban charter school and you teach in a semi-rural public school. GED 500 Lesson 4

External Validity: ecological threat l An ecological threat occurs when there were any threats to internal validity. – Although there are specific subtypes of external validity listed in many research methods texts, (the custom version the suggested text has a list on page 158), you are not responsible for knowing them, just for knowing the difference between the population and ecological threats to internal validity and being able to indicate which one of the two is present in a scenario when it is presented. For example, in the practice problems that will follow. GED 500 Lesson 4

Self-Check: Validity l Which type of validity refers to the influence of confounding variables? – – Internal Validity External Validity GED 500 Lesson 4

Self-Check: Validity l Which type of validity refers to the influence of confounding variables? – – Internal Validity External Validity GED 500 Lesson 4

Self-Check: Internal Validity l Which possible threat to internal validity? A researcher is super enthusiastic about his new method that he is using to teach his students. – – Subject Effects Experimenter Effects Selection History GED 500 Lesson 4

Self-Check: Internal Validity l Which possible threat to internal validity? A researcher is super enthusiastic about his new method that he is using to teach his students. Subject Effects – – – Experimenter Effects Selection History GED 500 Lesson 4

Self-Check: Internal Validity l Which possible threat to internal validity? A researcher uses a math test that she created as a pretest and the book publisher’s test as a post-assessment for an after school tutoring program. – Subject Effects – Experimenter Effects – Instrumentation – Testing GED 500 Lesson 4

Self-Check: Internal Validity l Which possible threat to internal validity? A researcher uses a math test that she created as a pretest and the book publisher’s test as a post-assessment for an after school tutoring program. – Subject Effects – Experimenter Effects – Instrumentation – Testing GED 500 Lesson 4

Practice Problems Next Ten practice problems follow. Just keep track of the ones you do not get correct with your first guess. Write down your original answer, the correct answer and why you think you missed it. Use the link that will be in the Week 6 folder to post that information on Blackboard. If you get all 10 correct with your first guesses, let me know with a message 10/10 correct. GED 500 Lesson 4

Practice Problems 1. What threat to validity is suggested if the 300 lowest scoring students, out of an initial 4000 who were pretested, were the only ones used in a study? GED 500 Lesson 4

ANSWER Statistical regression GED 500 Lesson 4

Practice Problems 2. What threat to validity would be suggested if 143 students out of the 568 who were participating in science instruction research transferred from the district during the school year? GED 500 Lesson 4

ANSWER Mortality GED 500 Lesson 4

Practice Problems 3. What threat to validity is suggested if the same form of a test is given for both pretest and posttest, and the intervention does not last for more than a few weeks? GED 500 Lesson 4

ANSWER Testing GED 500 Lesson 4

Practice Problems 4. What threat would be present if a researcher gives the CTBS as a pretest and a district competency test as a posttest? GED 500 Lesson 4

ANSWER Instrumentation GED 500 Lesson 4

Practice Problems 5. What threat is most likely present in a one day study that requires students to remain in a classroom beginning at 7: 00 am and ending at 4: 00 pm? GED 500 Lesson 4

ANSWER Maturation (They would be very, very tired! GED 500 Lesson 4

Practice Problems 6. What threat to validity is suggested if two weeks after the study began 25% of the students enroll in private math tutoring on weekends, as part of a new Community Based Organization (CBO) program about which you were unaware? GED 500 Lesson 4

ANSWER History GED 500 Lesson 4

Practice Problems 7. What threat would be suggested if the teachers participating in a study of a new instructional method were more interested in the stipend that they received than doing a good job? GED 500 Lesson 4

ANSWER Subject Effects GED 500 Lesson 4

Practice Problems 8. If two 2 nd grade classes participated in a study comparing the effects of two different reading methods and one class of 28 had 23 boys and the other class of 27 had 12 boys, what threat to validity is most likely? GED 500 Lesson 4

ANSWER Selection (way too many boys in one class for a fair comparison) GED 500 Lesson 4

Practice Problems 9. A 11 th grade history teacher getting his MA in education is conducting a study on the effect of using movement (specifically stationary dance movements) on students’ memory for history facts. He teaches two classes (1 st period is the experimental class and 4 th period is the control class). Unfortunately for his study, a pretty large group of students who are in Model United Nations are split between the two classes and the ones in the experimental class teach the movements to their friends in the control class. GED 500 Lesson 4

ANSWER Diffusion of treatment GED 500 Lesson 4

Practice Problems 10. Twenty middle school teachers are “volunteered” by their principals to be trained in a “new” teaching method at the beginning of a new school year— this is right after they spent the entire summer learning a different “new” method that was supposed to be implemented. The students of these teachers will be the subjects in a study on the effectiveness of the method. The good news is that the teachers will receive one period release for the training—the bad news is that they will not receive a stipend for the extra work they will be expected to do as researchers in the study. They are especially irritated at that piece of news since they had received stipends for the summer work and they are pretty disgusted by the new change in plans for the school year. In fact, some of them don’t fully implement the new method at all. GED 500 Lesson 4

ANSWER Experimenter Effects GED 500 Lesson 4

End of Lesson 6 Finally! GED 500 Lesson 4