Valuefree Research in the Social Sciences Principles and

Value-free Research in the Social Sciences: Principles and Controversies 2017 -2018 Semester 1, block 1 Week 3 of course “Quality and Ethics in Social Science Research” Matthijs Kalmijn University of Amsterdam, Department of Sociology

Content • Principles • Merton’s classic norms • Objectivity • Bias (Mac. Coun) • Subgroup work & presentations • Controversies • Same-sex parenting: bad for kids? • Racial diversity: good for companies? • Remedies

My own approach • Sociology (family, life course, inequality) • Quantitative • Survey based • Statistically analyze data to • Describe reality • Test theories about reality • Primacy of science

The norms of science Robert K. Merton • Universalism • Communalism • Disinterestedness • Organized skepticism

Examples of violations • Bourdieu said it …. • Epidemiological surveys in the Netherlands • Undocumented research data • Theoretical paradigms

Objectivity in research • Objective / subjective is a continuum • Don’t throw out the baby with the bathwater • Aim for more rather than less objectivity • Distinguish • Subjectivity in the research questions we choose • Subjectivity in theories we wish to test • Subjectivity in the use of research findings • Distinguish • Subjectivity of the researcher • Subjectivity of the things/people we study

Value free science • Political and moral values may motivate you to do research • Political and moral values may distort your interpretation of research results • The problem of value free science is part of a more general problem of biased evidence interpretation

Biased evidence interpretation: Mac. Coun’s typology No motivation Motivation Not intentional Cold bias Hot bias Intentional – Fraud Advocacy

Example 1 • Mental contamination

Example 2

Disorder & prejudice FR WAR AU N DU ING LE NT ! Stapel, D. A. and S. Lindenberg. 2011. "Coping with Chaos: How Disordered Contexts Promote Stereotyping and Discrimination. " Science 332(6026): 251 -53. 11

Diederik Stapel’s method • PI works out theory and hypotheses in detail with collaborator • Collaborator develops full questionnaires for schoolbased survey experiment (makes several revisions) • Printing of questionnaires, purchase of incentives and other research material • School-based field stage (PI is only contact) • Questionnaires put in Excel by school pupils • Data back via PI to collaborator, starts analyzing • Hypotheses confirmed (“you have gold there”) • Elaborate co-writing stage and submission (e. g. , JPSP)

Diederik Stapel’s method • PI works out theory and hypotheses in detail with collaborator • Collaborator develops full questionnaires for schoolbased survey experiment (makes several revisions) • Printing of questionnaires, purchase of incentives and other research material • School-based field stage (PI is only contact) • Questionnaires put in Excel by school pupils • Data back via PI to collaborator, starts analyzing • Hypotheses confirmed (“you have gold there”) • Elaborate co-writing stage and submission (e. g. , JPSP)

Other examples of fraud • Biological science (Hwang Woo-Suk, stem cell research) • Medical research (Potti, cancer research) • Anthropology (Mart Bax, wars in Bosnia) • Political science (La. Cour & Green, on how contact changes homophobia)

Example 3 • Statistical significance • Is b significant? àb has a t-value which has a p-value, e. g. 4% • Meaning: Suppose the real b (effect) is 0, the chance of finding that value of b (or higher) is 4% • Since this chance is low (below 5%), we reject the null hypothesis (0 effect unlikely) • We call the observed b significant

Disconfirmation bias Masicampo, E. J. and D. R. Lalande. 2012. "A Peculiar Prevalence of P Values Just Below. 05. " Quarterly Journal of Experimental Psychology 65(11): 2271 -79.

Summary Not intentional Intentional No motivation Motivation The clock P-values Diederik Stapel

Motivation for bias • Self-interest • Interests of (external) organization • Political and moral views • Societal effects/consequences • Overcommitted to a theory

Subgroup work Not intentional Intentional No motivation Motivation Example 1 Example 2 Example 3

Subgroup work • Subgroups formed based on discipline and/or substantive research interests • Work out examples of biases that can arise in your own research, at least one for each cell • These examples can be fictitious but must be ‘realistic’ (how could it go wrong…. ) • Make a Powerpoint and present back to class (one presenter) • Send to matthijskalmijn@gmail. com

Controversies 1: Regnerus (2012) Criticized by Amato, Eggebeen and Osborne and others

Regenerus ‘motivated’ or not? "As Christians, our lives should reflect our relationship with God and our desire to glorify Him, " Regnerus says. "I've noticed that some Christian professors see a disconnect between their faith and their profession. I believe that if your faith matters, it should inform what you teach and what you research. ” (Old quote in Trinity Christian Vollege Alumni Magazine)

Replication by Cheng and Powell 2015

Controversies 2: Herring Criticized by Stojmenovska, Bol & Leopold

Herring ‘motivated’ or not? “Yes, diversity is still a good thing, but not just because it is related to business outcomes. Diversity is also a good thing because it reinforces the belief that everyone—no matter their race, ethnicity, gender, sexuality, or religion—deserves an equal opportunity. This message is even more important now than it was two presidential elections ago. ” (in Herring’s response to Stojmenovska, Bol & Leopold)

Replication by Stojmenovska et al.

RACIAL DIVERSITY CONFOUNDING VARIABLE COMPANY SIZE SPURIOUS! COMPANY SALES

Some remedies – institutional • Socialize students • Take an oath (? ) • Let social scientists compete • Organize peer review • Share data • Conduct meta analyses

Some remedies – individual • Engage alternative hypotheses • Strive for accuracy and precision • Look (also) for falsification options • Allow (and do) replication studies • Have respect for the annoying facts

Moral and political values are sometimes good motivators but always poor guides