RCR Forum Humility and Hubris in Science The

RCR Forum Humility and Hubris in Science: The Rewards and Risks of Emerging Technologies Daniel A. Vallero, Ph. D. Duke University, Pratt School of Engineering March 2, 2010 1

Much is expected…. The Brain—is wider than the Sky— For—put them side by side— The one the other will contain With ease—and You—beside— The Brain is deeper than the sea— For—hold them—Blue to Blue— The one the other will absorb— As Sponges—Buckets—do— The Brain is just the weight of God— For—Heft them—Pound for Pound— And they will differ—if they do— As Syllable from Sound— Emily Dickinson (1862) “Wider than the Sky. ” 2

3 Parables 1. Trolleys and Medicine 3

3 Parables 1. Trolleys and Medicine 4

3 Parables 1. Trolleys and Medicine 5

The Double Effect • Equation with 2 constraints. 1. Socrates & Hippocrates: “First do no harm. ” 2. Saint Thomas Aquinas: “Good is to be done and promoted and evil is to be avoided. ” • Act leading to good but with negative side-effects permitted, but deliberate harm (even for good causes) is wrong. • Humans are not “objects. ” • Good effect must be at least as directly an effect of the action as is the negative bad effect. – Bad must not cause the good effect. • Benefit of the bad must not outweigh the benefit of the good. • Other ethically acceptable approach without the side effects must not be available. 6

3 Parables 1. Trolleys and Medicine 2. Rolheiser’s “Good” Town 7

3 Parables 1. Trolleys and Medicine 2. Rolheiser’s “Good” Town 3. Hardin’s “Tragedy of the Commons” 8

Exercise • • • Assess expertise and interests Read summary Discuss Appoint spokesperson Answer questions Reach consensus, if possible, but log minority views as well • Summarize for all of us 9

An 6 -step approach to ethical decision making (from Resnik) 1. 2. 3. 4. State or define the problem/issue Gather information Delineate options. Apply different values, rules, principles, regulations to the different options. 5. Resolve conflicts among values, rules, etc. 6. Make a decision and act 10

“Settled Science” • • • Climategate Lessons? Is science a social enterprise? Is science’s value instrumental? Is it inherent? 11

Chaos Chain of events Initial event Present Mitigating measures Subsequent event series 1…n Subsequent event series 1…p Subsequent event series 1…q Subsequent outcome series 1…r Actual outcome Probability of outcome at outset Desired environmental outcome 0. 970 Fortuitous, positive environmental impact 0. 003 Neutral environmental impact 0. 026 Unplanned negative environmental impact 0. 001 Future 12

Improve desired outcome, but at what cost? Chain of events Subsequent event series 1…n Initial event Subsequent event series 1…p Subsequent event series 1…q Subsequent outcome series 1…r Present Actual outcome Probability of outcome at outset Desired outcome 0. 975 Fortuitous, positive impact 0. 002 Neutral impact Unplanned negative impact 0. 020 0. 003 Future 13

Decision Tree (hypothetical) 14

Precaution versus Evidence-Based Views • What’s the difference? • Which is better for…. – Environmental protection? 15

The environment can be seen as series of reactors…. 16

Precaution versus Evidence-Based Views • What’s the difference? • Which is better for…. – Environmental protection? – Biotechnologies? 17

European classes of risks posed genetically modified microorganisms Hazard Level Description of Microbial Hazard Least Never identified as causative agents of disease in humans and that offer any threat to the environment. Hazardous when contained, low human risk May cause disease in human and which might, therefore, offer a hazard to laboratory workers. They are unlikely to spread in the environment. Prophylactics are available and treatment is effective. Severe when contained, moderate human risk Severe threat to the health of laboratory workers, but a comparatively small risk to the population at large. Prophylactics are available and treatment is effective. High human population risk Severe illness in humans and serious hazard to laboratory workers and to people at large. In general, effective prophylactics are not available and no effective treatment is known. Greatest ecological and human population risk Most severe threat to the environment, beyond humans. May lead to heavy economic losses. Includes several classes, Epl, Ep 2, Ep 3 (see Table 1. 2 for descriptions) to accommodate plant pathogens. 18

European classes of microbes causing diseases in plants. Biotechnology Class Description of Microbes in Class Ep 1. May cause diseases in plants but have only local significance. They may be mentioned in a list of pathogens for the individual countries concerned. Very often they are endemic plant pathogens and do not require any special physical containment. However, it may be advisable to employ good microbiological techniques Ep 2. Known to cause outbreaks of disease in crops as well as in ornamental plants. These pathogens are subject to regulations for species listed by authorities in the country concerned Ep 3. Mentioned in quarantine lists. Importation and handling are generally forbidden. The regulatory authorities must be consulted by prospective users 19

Precaution versus Evidence-Based Views • What’s the difference? • Which is better for…. – Environmental protection? – Biotechnologies? – Drug safety? – Devices? 20

Precaution versus Evidence-Based Views • What’s the difference? • Which is better for…. – Environmental protection? – Biotechnologies? – Drug safety? – Devices? – Sustainability? 21

Disaster Project Hmmmm…… 22

Precaution versus Evidence-Based Views • What’s the difference? • Which is better for…. – Environmental protection? – Biotechnologies? – Drug safety? – Devices? – Sustainability? – Greatest good for the greatest number? 23

Precaution versus Evidence-Based Views • What’s the difference? • Which is better for…. – – – – – Environmental protection? Biotechnologies? Drug safety? Devices? Sustainability? Greatest good for the greatest number? Professional duty? Research integrity? Advancement of knowledge? 24

So, then what is risk, really? • Definition: Probability of harm or loss • Part of our everyday lives – Different for each of us – Basis for decision-making • But is it quantifiable? 25

… depends on how you ask but, . . . • Yes it is quantifiable: üRisk = f(Hazard x Exposure) • A probability, a fraction • Part of our everyday lives – Different for each of us – Basis for decision-making 26

Risk assessment is informed by science (you)… 27

Different Processes at Work: Not everyone thinks like you do…. * Analytical Phase Identifying risk Risk Assessment Processes Risk Perception Processes Physical, chemical, and biological monitoring and measuring of the event Personal awareness Deductive reasoning Intuition Statistical inference Estimating risk Magnitude, frequency and duration calculations Personal experience Cost estimation and damage assessment Intangible losses and nonmonetized valuation Economic costs Evaluating risk Cost/benefit analysis Personality factors Community policy analysis Individual action *Adapted from K. Smith, 1992 28

Perception is crucial • Which line is longer? The Müller-Lyer Illusion. 29

Perception is complex • Which line is longer? The Müller-Lyer Illusion. 30

Perception is variable • Which line is longer? The Müller-Lyer Illusion. 31

Perception is a benchmark • Which line is longer? The Müller-Lyer Illusion. 32

Perception is personal • Which line is longer? The Müller-Lyer Illusion. 33

Perception just is…. • Which line is longer? The Müller-Lyer Illusion. 34

And sometimes, perception is pretty accurate…. Source: Pardon, ca. 1970. 35

Factors increasing perceived risk: (after Covello, 1992) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Possible Severely Negative or Catastrophic Outcome Unfamiliarity with Situation and Potential Risks Inability to Explain Processes and Mechanism Little Certainty in the Science and Engineering Perception of Personal Control Involuntary Exposures to Risks Risk to Children and Sensitive Groups Long-term Exposures, Latency Periods, Chronic Risk Possible Transgenerational Exposures and Risks Uncertainty about Potential Victims 36

… and the risk is perceived to increase even more when … • Greater “dread” – Major problem for nuclear power industry • Mistrust of corporate or governmental partners – Guilt by association • Negative media attention • A history of accidents and failures at this site or in similar situations • Benefits are not clear • Mistakes are irreversible – Global climate change, for example 37

Discussion Questions • What can you do to ensure your work meets ethical standards? • How can you make sure that it is not misused by others? 38

A few final words… • • Do you agree with the risk paradigm? Should it be evidence based? How about the precautionary principle? Hubris is thinking you know when you don’t…. 39

Trust depends on humility – knowing that you don’t know…. • Ethike aretai – Engineering needs character – Engineering needs skill • Credat emptor! • Resolve today to keep building competence and character. 40

If you have questions, contact me…. • dav 1@duke. edu • 919 -541 -3306 41