1985 CPSRMIT Debate Michael Dertouzos moderator David Parnas

1985 CPSR-MIT Debate Michael Dertouzos, moderator David Parnas, against SDI (Joseph Weizenbaum, against) Charles Seitz, for SDI (Danny Cohen, for) 1

Charles Seitz, arguing for 2

Pause for Analysis Sketch Seitz’ argument in premise-conclusion style: Since Premise, and Premise, … Therefore Conclusion. (Hint: identify conclusion first. ) 3

Seitz’ Conclusion It is possible to create reliable SDI software. 4

Seitz’ Premises Since • A hierarchical architecture seems best, (because more natural, used in nature, understood by military, allows abstraction up levels …) 5

Seitz’ Premises Since • A hierarchical architecture seems best, • Physical organization should follow logical organization, (simplest choice, natural) 6

Seitz’ Premises Since • A hierarchical architecture seems best, • Physical organization also hierarchical, • Tradeoffs to make software problem tractable are in the choice of system architecture (not in new / radical methods) 7

Seitz’ Premises Since • A hierarchical architecture seems best, • Physical organization also hierarchical, • This makes software problem tractable, • Loose coordination allows us to infer system performance (assume stat. independence, …) 8

Seitz’ Argument Since • A hierarchical architecture seems best, • Physical organization also hierarchical, • This makes software problem tractable, • And allows system reliability estimate, Therefore – It is possible to create reliable SDI battle management software. 9

Pause for Analysis Whose argument is better? Why? Do they start with the same problem definition? 10

David Parnas, Rebuttal 11

Charles Seitz, Rebuttal 12

Pause for Analysis Relevant analogies to SDI? Why / why not? • • Space shuttle software Telephone system software Nuclear plant software others? 13

Pause for Analysis Outline the most realistic SDI software testing that you can. 14

Pause for Analysis How did you account for … • • • real-world sensor inputs variable weather conditions target / decoy appearance variable attack structure attacked components failing 15

Fault Tolerant Software? James Ionson, in “Reliability and Risk, ” a CPSR video. 16

Fault Tolerant Software? “It is not error-free code, it is fault-tolerant code. And if another million lines has to be written to ensure faulttolerance, so be it. ” - James Ionson 17

Fault Tolerant Software? Diagram in premise-conclusion form the argument being made by James Ionson. Does the argument make sense? Why / why not? 18

“Star Wars” Today Current SDI-like programs are called “National Missile Defense. ” There are some potentially important differences. 19

“Star Wars” Today “One of the remarkable aspects of the evolution of missile defenses is that few policy makers question the fundamental ability … to be effective. Instead they focus on timing, cost, …. ” (Mosher, page 39, IEEE Spectrum, 1997) 20

“Star Wars” Today “This is a sharp change from the Reagan years, perhaps because the technology used is closer at hand the threats are smaller. ” (Mosher, page 39, IEEE Spectrum, 1997) 21

“Star Wars” Today Smaller anticipated mission: “protect the U. S. … against an attack by a rogue state using a handful of warheads outfitted with … simple countermeasures. ” (Mosher, page 36, IEEE Spectrum, 1997) 22

“Star Wars” Today Smaller anticipated mission: “also provide protection against an accidental launch of a few warheads by Russia or China. ” (Mosher, page 36, IEEE Spectrum, 1997) 23

“Star Wars” Today One talked-about version does not use space-based weapons: “… no more than 100 hit-to-kill interceptors based at old ABM site near Grand Forks, ND. ” (Mosher, page 37, IEEE Spectrum, 1997) 24

Pause for Analysis How fundamentally does it change Parnas’ argument if the anticipated attack uses fewer and simpler missiles? 25

Parnas’ Argument How are the premises changed? • • Specifications not known in advance, Realistic testing is not possible, No chance to fix software during use, No foreseeable technology changes this, None are changed “in principle” but overall it seems somehow less impossible. 26

“Star Wars” Testing “In the last 15 years, the U. S. has conducted 20 hit-to-kill intercepts, …. Six intercepts were successful; 13 of those test were done in the last five years, and among them three succeeded. ” (Mosher, page 39, IEEE Spectrum, 1997) 27

“Star Wars” Testing “No real attempts have been made to intercept uncooperative targets – those that make use of clutter, decoys, maneuver, anti-simulation, and other countermeasures. ” (Mosher, page 39, IEEE Spectrum, 1997) 28

“Star Wars” Testing “Test … of a powerful laser has been blocked by … bad weather and software problems. … a software problem caused the laser to recycle, or unexpectedly lose power …. ” (R. Smith, Washington Post, Oct 8, 1997) 29

Schwartz versus TRW In 1996, ex TRW engineer Nira Schwartz filed a “False Claims Act” suit, alleging that results of tests to distinguish warheads and decoys were falsified by TRW. (featured on “ 60 Minutes II” in January 2001) 30

Schwartz versus TRW Schwartz claims that TRW “knowingly made false test plans, test procedures, test reports and presentations to the government … to remain in the program. ” 31

Schwartz versus TRW Schwartz claims – “I say to my boss, “It is wrong, what we are going; it is wrong. ” And the next day, I was fired. ” 32

Schwartz versus TRW says – “TRW scientists and engineers devoted years to this complex project, while Ms. Schwartz, in her six months with the company … Her understanding … is insufficient to lend any credibility to her allegations. ” 33

Schwartz versus TRW DOD criminal investigator says – “absolute, irrefutable, scientific proof that TRW’s discrimination technology does not, cannot, and will not work” … TRW “knowingly covering up. ” 34

Schwartz versus TRW DOD panel then said – “TRW’s software and sensors are “well designed and work properly” provided that the Pentagon does not have any wrong information about what kind of warheads and decoys an enemy is using. ” 35

Schwartz versus TRW Lt. General Kadish – “Right now, from what I see, there is no reason to believe that we can’t make this work. But there’s a lot more testing to be done. ” 36

Schwartz versus TRW Congressman Curt Weldon, R-PA: “If we don’t build a new aircraft carrier, we have older ones. If we don’t build a new fighter plane, we have older ones. If we don’t build missile defense, we have nothing. ” What is the premise-conclusion summary of this argument? 37

Schwartz versus TRW Congressman Curt Weldon, R-PA: On 50 Nobelists’ anti-BMD letter “I don’t know any of them that’s come to Congress or me. I mean … its easy to get anyone to sign a letter. I sign letters all the time. ” What is the premise-conclusion summary of this argument? 38

Schwartz versus TRW Congressman Curt Weldon, R-PA: “There were scientists that who made the case against Kennedy that it was crazy, we’d never land on the moon. And I characterize Postol now as one of those people. ” What is the premise-conclusion summary of this argument? 39

Ethical Issues What are some of the important ethical questions? And what guidance do the codes of ethics give on these questions? 40

Ethical Issues • How to interact with colleagues with whom you disagree? • When to blow the whistle? • Should you accept work on an “impossible” but $$$ project? 41

Dealing with Colleagues AITP Standards of Conduct: “In recognition of my obligation to fellow members and the profession I shall cooperate with others in achieving understanding and in identifying problems. ” 42

Dealing with Colleagues Item 5. 12 of ACM / IEEE-CS Software Engineering Code: “Those managing or leading software engineers shall not punish anyone for expressing ethical concerns about a project. ” 43

Accept Impossible Work? Item 3. 2 of ACM / IEEE-CS Software Engineering Code: “Software engineers shall ensure proper and achievable goals and objectives for any project on which they work or propose. ” 44

Accept Impossible Work? Item 1. 3 of the ACM / IEEE-CS Software Engineering Code: “Software engineers shall accept software only if they have a well founded belief that it is safe, meets specifications, passes appropriate tests, …” 45

Blow the Whistle? AITP Standards of Conduct: In recognition of my obligation to society, I shall never misrepresent or withhold information that is germane to a problem or situation of public concern nor allow any such known information to remain unchallenged. 46

Blow the Whistle? Item 1. 4 of ACM / IEEE-CS Software Engineering Code: “Software engineers shall disclose to appropriate persons or authorities any actual or potential danger to the user, the public … that they reasonably believe …” 47

Summary • Difficult ethical issues arise in creation of safety-critical software. • Trustworthy SDI software is more clearly impossible in retrospect. • Modern, smaller SDI-like programs appear more tractable. 48

Thanks to National Science Foundation grant DUE 97 -52792 for partial support of this work. 49

Thanks to the Computing Professionals for Social Responsibility (www. cpsr. org) for permission to distribute digitized video of the debate. 50

Thanks to David Parnas Chuck Seitz for commenting on a draft of the paper describing this module. 51

Thanks to the The Ronald Reagan Presidential Library (www. reagan. utexas. edu) for help in obtaining the video of Reagan’s 3/23/83 speech. 52

Thanks to Christine Kranenburg Laura Malave Melissa Parsons Joseph Wujek for technical assistance. 53

The End. 54

Overheads from Parnas’ Presentation • The next slides are transcribed versions of (most of) the transparencies in Parnas’ presentation. 55

Why is it important that the software can never be trusted? • “We” will make decisions as if it was not there. • “They” will make decisions as if it might work. 56

A necessary condition for trustworthy engineering products is validation by: • Mathematical analysis, or • Exhaustive case analysis, or • Prolonged, realistic, testing or a combination of the above 57

Why software is always the unreliable glue in engineering systems: • The best mathematical tools require that a system be described by continuous functions • Exhaustive case analysis can only be used when the number of states is small or the design exhibits a repetitive structure 58

Why do we have some usable software? • Sometimes the requirements allow untrustworthy software • There has been extensive use under actual conditions • Operating conditions are controlled or predictable • “Backup” manual system available when needed 59

What makes the SDI software much more difficult than other projects? • Lack of reliable information on target and decoy characteristics • Distributed computing with unreliable nodes and unreliable channels • Distributed computing with hard real-time deadlines • Physical distribution of redundant real-time data • Hardware failures will not be statistically independent 60

What makes the SDI software much more difficult than other projects? • Redundancy is unusually expensive • Information essential for real-time scheduling will not be reliable • Very limited opportunities for realistic testing • No opportunities for repairing software during use • Expected to be the largest real-time system ever attempted, frequent changes are anticipated 61

Software Espionage and Nuclear Blackmail • Fact: Software systems, because of their rigid predetermined behaviors are, easily defeated by people who understand the programs • Fact: Changes in large software systems must be made slowly and carefully with extensive review and testing 62

What about new Soft. Eng. techniques? • Precise requirement documents • Abstraction/information hiding • Formal specifications The use of these techniques requires previous experience with similar systems • Co-operating sequential processes requires detailed information for real-time scheduling • Structured programming reduces but does not eliminate errors 63

What about Artificial Intelligence? • AI-1 - Defined as solving hard problems. – Study the problem, not the problem solver. No magic techniques just good solid program design. • AI-2 - Heuristic or Rule Based Programming/Expert Systems – Study the problem solver, not the problem – Ad hoc, “cut and dry” programming – Little basis for confidence 64

What about new programming languages? • No magic • They help if they are simple and well understood • No breakthroughs The fault lies not in our tools but in ourselves and in the nature of our product. 65

What about automatic programming? • Since 1948 a euphemism for programming in a new language? 66

What about program verification? • The right problem but do we have a solution? • What’s a big program? • Wrong kind of program? How do you verify a model of the earth’s gravitational field? • Implicit assumption of perfect arithmetic • What about language semantics? 67

Is there a meaningful concept of tolerance for software? • The engineering notion of “tolerance” depends on an assumption of continuity. • Statistical measures of program quality are limited in their application to situations where individual failures are not important. 68

Overheads from Seitz’ Presentation • The next slides are transcribed versions of (most of) the transparencies in Seitz’ presentation. 69

From “The Strategic Defense Initiative” White House pamphlet dated Jan, 1985. “SDI’s purpose is to identify ways to exploit recent advances in ballistic missile defense technologies that have potential for strengthening our security and that of our Allies. The program is designed to answer a number of fundamental scientific and engineering questions that much be addressed before the promise of these new technologies can be fully assessed. The SDI program will provide to a future president and a future congress the technical knowledge necessary to support a decision in the early 1990’s on whether to develop and deploy advanced defensive systems. ” 70

From 1985 “Report to the Congress on the Stategic Defense Initiative” (Section III): “The goal of the SDI is to conduct a program of rigorous research focused on advanced defensive technologies. ” “The SDI seeks, therefore, to exploit emerging technologies that may provide options for a broader-based deterrence by turning to a greater reliance on defensive systems” 71

From 1985 “Report to the Congress on the Stategic Defense Initiative” (Section III): “It should be stressed that the SDI is a research program that seeks to provide the technical knowledge required to support a decision on whether to develop and later deploy these systems. All research efforts will be fully compliant with U. S. treaty obligations. ” 72

• Weapons – Incapable of causing damage at Earth’s surface – Range 1000 km. – Partial deployment ineffective in boost phase • Sensors – Some located in high orbits – Can be passive – Useful in early deployments • Battle Management System – Computers and communication 73

Coordination • Lowest Level - stereo and “sensor fusion” • Middle Levels - target discrimination, attack and coordination • High Levels - assignment of priorities of target in midcourse in order to prevent particular areas from being overwhelmed in terminal defense, or to prevent any single area to accept too high a concentration for terminal defense • Top Level - command control decisions 74

Conclusions of the Panel: “The feasibility of the battle management software and our ability to test, simulate, and modify the system are very sensitive to the choice of system architecture. In particular, the feasibility of the BMS software is much more sensitive to the system architecture than it is to the choice of software engineering technique” 75

Conclusions of the Panel: “Software technology is developing against what appears today to be relatively inflexible limits in the complexity of systems. The treadeoffs necessary to make the software tractable are in the system architecture” 76

Conclusions of the Panel: “We must prefer an unconventional system architecture whose programming is within the anticipated limits of software engineering over reliance on radical software development approaches and the risk that we could not develop reliable software at any cost…” 77

Conclusions of the Panel: “One promising class of system architecture for a strategic defense system are those that are less dependent on tight coordination… [because of]… the ability to infer the performance of full -scale deployment by evaluating the performance of small parts of the system. ” 78