On experimental data vs observational data Prof Keunkwan

On experimental data vs. observational data Prof. Keunkwan Ryu STATISTICS

(1) Real-life examples Effect of “stop-smoking” on health Effect of foreign investment on Chinese provincial growth Effect of apartment age on apartment price in Seoul (see ppt file on Apartment in Seoul) Effect of fertilizer level on yield (see ppt file on Farmer’s example) Taken from “E. Leamer (1983), “Let’s Take the Con Out of Econometrics, ” American Economic Review, 73, 31 -43. STATISTICS 2

(2) What are the problems with observational studies? simultaneity reverse causality endogeneity STATISTICS 3

(3) Need to control for “systematic” differences a. Experiments=Universal control (c. f. “double blind, randomized controlled experiments) b. Randomized controlled experiments vs. Natural experiments c. Natural experiments Quasi-experiments Pseudo-experiments These “experiments” offer instrumental variables. STATISTICS 4

(4) What to control? How to control? a. To be able to identify the effect of color on taste, it is better to compare “red apples and yellow apples” than to compare “apples and oranges” b. Observed difference can be controlled by multiple regressions (see apartment example in Seoul: ppt file downloadable from http: //ezstat. co. kr) STATISTICS 5

(4) What to control? How to control? c. Unobserved difference causes the so-called “sample selection bias”, if not suitably accounted for. For example, in evaluating a medical treatment, random assignment into a treatment group and a control group does not solve the problem entirely. treatment group Some members drop out Selfselected treatment group We could just look at the outcome for the people who “choose” to stay in the treatment group (endogenous compliance). The self-selected” treatment group may be more health conscious and following a healthier diet. Even if the treatment itself has no effect, this self -selected group may have a better average outcome than the control group. STATISTICS 6

(4) What to control? How to control? What to do with “self-selection”? Try to control for the unmeasured as well as the measured differences using the so-called “sample-selection models. ” Controlling for Measured difference Unmeasured difference A matter of adding a bunch of “x” variables as covariates Still possible! We can “guess” individual “types” (unmeasured characteristics) based on their observed choices given observed characteristics. STATISTICS 7

(4) What to control? How to control? Controlling for unmeasured differences Example 1 A woman who has chosen to work even with lots of young kids, should be of a type of woman who are very eager to work. Then, see whether these types of women make more money than other types of women to identify the effect of the (unmeasured) type on wages. STATISTICS 8

(4) What to control? How to control? Controlling for unmeasured differences Example 2 A Ph. D student in his or her 9 th year with very high GPA (observed characteristics), should be a very lazy type of student. From Examples 1 & 2 As we have seen from the above examples, because we can “indirectly measure” the unmeasured types of women or students, we can control for their unmeasured types as well as identify their effects on quantities of interest. STATISTICS 9

(4) What to control? How to control? d. Look for “natural experiments” : For this purpose, you may better know institutional changes and/or historical events. “Knowledge of history matters!” In “event studies, ” an event should be rather “exogenous. ” STATISTICS 10

(4) What to control? How to control? Natural experiments Example 1 Univ. of Tokyo did not admit students in the entering class of April 1969, which has nothing to do with the quality of the entering students. Study how the career paths of those students, who would have gone to the Univ. of Tokyo but ended up with other “good” substitute universities, differ from the career paths of Univ. of Tokyo graduates. Carry out the analysis separately for government officials and for the private sector employees. This empirical research will identify whethere exists “Univ. of Tokyo premium” in terms of preferential promotion through “U-Tokyo mafia” networking, and if so, whether the premium is bigger in the public sector than in the private sector. STATISTICS 11

(4) What to control? How to control? Natural experiments Example 2 c. f. Jeffrey Milyo, and Joel Waldfogel(1999), “The Effect of Price Advertising on Prices: Evidence in the Wake of 44 Liquormart, ” American Economic Review, 89, 1081 -1096. The US supreme court decision (44 Liquormart decision), eliminating Rhode Island’s ban on liquor price advertising, made Rhode Island the subject of a natural experiment for measuring the effect of advertising on prices. Using Massachusetts prices as controls, they find that advertising stores substantially cut only prices of the products that they advertise. Prices of other products do not change. STATISTICS 12

(4) What to control? How to control? Natural experiments Example 3 c. f. Bruce Meyer, W. Viscusi, and D. Durbin (1995), “Workers’ Compensation and injury Duration: Evidence from a Natural Experiment, ” American Economic Review, 85, 322 -340 This paper examines the effect of workers’ compensation on time out of work. It introduces a “natural experiment” approach of comparing individuals injured before and after increases in the maximum weekly benefit amount. The increases examined in Kentucky and Michigan raised the benefit amount for high-earnings individuals by approximately 50%, while low-earnings individuals, who were unaffected by the benefit maximum, did not experience a change in their incentives (motivating “difference in differences” approach). Time out of work increased for those eligible for the higher benefits and remained unchanged for those whose benefits were constant. The estimated duration elasticities are clustered around 0. 3 -0. 4. (50% benefit increase causes 15%-20% increase in time out of work. ) STATISTICS 13

(4) What to control? How to control? Natural experiments Example 4 c. f. R. Barro and Jong-Hwa Lee, “IMF Programs: Who is Chosen and What are the Benefits? , ” NBER working paper, W 8951, 2002. They try to measure the effect of IMF lending on economic growth of the recipient countries. The recipient countries grow at a significantly lower rate than the countries not receiving the lending. What is wrong with the IMF lending? This is simply an endogeneity problem, or reverse causality or simultaneity. They use “voting alliance with the US in the past” and “IMF staff representation from each country” as instrumental variables to single out the growth effect net of the endogeneity bias and find no significant negative effect on growth of the IMF funding. (They use “quasi experimental” variables; “lagged” variables. ) STATISTICS 14

(4) What to control? How to control? From the abstract of their paper “IMF lending practices respond to economic conditions (which makes IMF lending an endogeneous variable in growth regression equations) but are also sensitive to political-economy variables (if unrelated with economic conditions, qualify for “instrumental variables”) Specifically, the sizes and frequencies of loans (endog) are influenced by a country’s presence at the Fund, as measured by the country’s “share of quotas” and “professional staffs”. IMF lending is also sensitive to a country’s political proximity by “voting patterns in the United Nations” and economic proximity to some major shareholding countries of the IMF, the US, France, Germany, and the UK. We measured political proximity by “voting patterns of in the United Nations” and economic proximity by “bilateral trading volumes. ” These provide instrumental variables for estimating the effects of IMF lending on economic performance. Instrumental estimates indicate that the size of IMF lending is insignificantly related to economic growth in the contemporaneous five -year period but has a significantly negative effect in the subsequent five years. ” STATISTICS 15

(4) What to control? How to control? Natural experiments Example 5 Twin study Twin brothers or sisters are pretty much the same in many aspects. Well-controlled. c. f. concept of “matching” STATISTICS 16