Power Point Slides to Accompany A Gift of

Power. Point® Slides to Accompany A Gift of Fire: Social, Legal, and Ethical Issues for Computers and the Internet (2 nd Edition) by Sara Baase San Diego State University Power. Point slides created by Sherry Clark Copyright 2003 Prentice Hall A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 1

A Gift of Fire Can We Trust the Computer? What Can Go Wrong? Case Study: Therac-25 Increasing Reliability and Safety Perspectives on Failures, Dependence, Risk, and Progress Computer Models A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 2

What Can Go Wrong? Facts About Computer Errors • Error-free software is not possible. • Errors are often caused by more than one factor. • Errors can be reduced by following good procedures and professional practices. Q: How do we distinguish between tolerable or unavoidable errors in software versus careless software development? A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 3

What Can Go Wrong? The Roles of People in Computer-related Problems: Computer User • At home or work, users should understand the limitations of computers and the need for proper training and responsible use. Computer Professional • Understanding the source and consequences of computer failures is valuable when buying, developing, or managing a complex system. Educated Member of Society • Personal decisions and political, social, and ethical decisions depend on understanding computer risks. A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 4

What Can Go Wrong? Categories of Computer Errors and Failures Problems for Individuals: • usually in their role as consumers. • who are incorrectly identified by inaccurate law enforcement databases. System Failures: • affecting large numbers of people and/or costing large amounts of money. Safety-Critical Applications: • where people may be injured or killed. A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 5

What Can Go Wrong? Problems for Individuals Billing Errors • Lack of tests for inconsistencies and inappropriate amounts. Database Accuracy Problems • Incorrect information resulting in wrongful treatment or acts. Q: Describe a computer error or failure that has affected you. A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 6

What Can Go Wrong? Problems for Individuals (cont’d) Causes: • • • Large population. Human common sense not part of automated processing. Overconfidence in the accuracy of data from a computer. Errors in data entry. Information not updated or corrected. Lack of accountability for errors. A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 7

What Can Go Wrong? System Failures Communications: • Telephone, online, and broadcast services. Business: • Inventory and management software. Financial: • Stock exchange, brokerages, banks, etc. . Transportation: • Reservations, ticketing, and baggage handling. A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 8

What Can Go Wrong? System Failures (cont’d) Causes: • • Insufficient testing and debugging time. Significant changes in specifications (during and after project begun). Overconfidence in system. Mismanagement of the project. Q: Describe a recent system failure that affected many people or resulted in a great monetary loss. A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 9

What Can Go Wrong? Safety-Critical Applications Military Power Plants Aircraft Trains Automated Factories Medicine …others. A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 10

What Can Go Wrong? Safety-Critical Applications Causes: • • • Overconfidence. Lack of override features. Insufficient testing. Sheer complexity of system. Mismanagement. Q: What activities do you participate in that are controlled by safety-critical applications? A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 11

Case Study: The Therac-25 was a software-controlled radiation-therapy machine used to treat people with cancer. Overdoses of radiation • Normal dosage is 100– 200 rads. • It is estimated that 13, 000 and 25, 000 rads were given to six people. • Three of the six people died. Q: What determines whether the risks associated with a safety-critical application are acceptable? A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 12

Case Study: Therac-25 Radiation Overdose Multiple Causes: • • • Poor safety design. Insufficient testing and debugging. Software errors. Lack of safety interlocks. Overconfidence. Inadequate reporting and investigation of accidents. A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 13

Increasing Reliability and Safety What Goes Wrong? Computer Systems Fail Because: • The job they are doing is inherently difficult, and • The job is done poorly. Compounding the Reliability Issue: • Developers and users exhibit overconfidence in the system. • Reused system software may not work in different environments. Q: Identify the elements needed as an incentive to increase reliability and safety. A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 14

Increasing Reliability and Safety Professional Techniques • • Follow good software practices. Exhibit professional responsibility at all levels of development and use. Construct well-designed user interfaces. Take human factors into account. Include built-in redundancy. Incorporate self-checking where appropriate. Follow good testing principals and techniques. Q: What human interface features should be considered for ordinary business applications? A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 15

Increasing Reliability and Safety Law and Regulation Criminal and Civil Penalties: • to recover loss from faulty or unsafe systems. Liability and Civil Laws: • to provide incentives to produce reliable and safe systems. Warranties: • to guarantee a certain level of quality. Federal or State Regulations: • to protect the public. A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 16

Increasing Reliability and Safety Law and Regulation (cont’d) Database Accuracy Enforcement: • to protect the public from inaccurate information maintained by private companies and government. Mandatory Licensing of Software Developers: • to ensure proper training, competency, and continuing education. Q: How can consumers protect themselves from faulty software? A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 17

Perspectives on Failures, Dependence, Risk, and Progress Failures • What are acceptable rates of failures? • How accurate should software be? Dependence • How dependent on computer systems are our ordinary activities? • How useful are computer systems to our ordinary activities? Risk and Progress • How do new technologies become safer? • Can progress in software safety keep up with the pace of change in computer technology? A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 18

Computer Models Points to Consider: • Models are simplifications of either physical or intangible systems. • Those who design and develop models must be honest and accurate with results. • Computer professionals and the general public must be able to evaluate the claims of the developers. Q: What problems in your community have been or could be studied with computer models? A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 19

Computer Models Evaluating Models Why Models Might Not Be Accurate: • • • Developers have incomplete knowledge of the system being modeled. Data might be incomplete or inaccurate. Power of the computer might be inadequate. Variables are difficult to numerically quantify. Political and economic motivation to distort results. Q: For each item above, give an example of a model or simulation that was inaccurate. A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 20

Computer Models Evaluating Models (cont’d) Regarding the Car-Crash Models Described in the Text: • How well do the modelers understand the system and/or materials being studied? How accurate and complete are the data? • What are the assumptions and simplifications in the model? • Do the results or predictions correspond with the real world? A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 21

Computer Models Evaluating Models (cont’d) Regarding the Climate Models Discussed in the Text: • How well do the modelers understand the system and/or materials being studied? How accurate and complete are the data? • What are the assumptions and simplifications in the model? • Do the results or predictions correspond with the real world? A Gift of Fire, 2 ed Chapter 4: Can We Trust the Computer? 22