Ethics risk and software development Advanced Software Engineering

Ethics, risk and software development Advanced Software Engineering Risk Analysis - Lecture 8 SPS © 2006 Department of Computer Science Durham University 8. 1

Ethics of risk • The study of risk involves the use of judgement and option • When we define a tolerable failure rate, we are defining how much effort should be expended • For safety, this is place a value, monetary or otherwise, on human life or suffering • Such decisions cannot be just a matter of engineering judgement but must reflect the view of society • However society views may be unpredictable and illogical © 2006 Department of Computer Science Durham University 8. 2

Value of human life? • The value placed on human life can vary enormously, depending on the situation • Consider the court ruling to the dependants of people killed in an accident where the victim is: – A baby, a child, a young adult, a breadwinner or a senior citizen • Courts sometimes use a “formula” based on issues such as an estimate of loss of income or the victim’s commitments • Such a formula formalises the value judgement of the formula’s creator © 2006 Department of Computer Science Durham University 8. 3

Preservation of life? • For safety – A decision to spend a certain amount of money in order to protect a given number of people gives a minimum “monetary” value to their lives – A decision not to perform work that would protect a given number of people because of the expense, places a maximum value on those lives • Looking at such decision in real cases allow us to assess perceptions of human worth in a number of areas – Varies by a factor of more than 1000!!! © 2006 Department of Computer Science Durham University 8. 4

Perception of risk • Greatly effected by the nature of that risk • Accidents in which large numbers of people are killed simultaneously are seen as being more serious than those where a similar number of people are killed individually – Train/plane crash vs. car crash statistics • Based on our “ability” to control the situation – The risk of a nuclear accident vs. hit down by a car – Which are we more worried about? – Which is more likely to happen? © 2006 Department of Computer Science Durham University 8. 5

Emotional impact • Society’s view of importance of an event is related to the emotional impact • Local vs. national events • Relevant to projects of national interest • Space shuttle vs. commercial airliner © 2006 Department of Computer Science Durham University 8. 6

Ethical issues and codes of conduct • Any scientific based activity requires a level of responsibility • There also associated moral questions • For safety-critical systems it is critical to consider the ethical questions involved in software development • The motivation for the development of codes of conduct – But before that …. © 2006 Department of Computer Science Durham University 8. 7

Seven deadly sins - Undesirable practises to avoid 1. Used for effect • Determine if a technique/method is actually necessary 2. Exaggeration • Beware overstated claims on certain development methods 3. Too trusting • Trust in a method vs. common sense or reasoning power 4. Rule by a few • Challenge expert promotion of particular solutions 5. Transitory issues • Short-lived vs. long term implications cf. reuse 6. Additional words • “Can’t see the forest for the trees” documentation 7. Meandering • Early use of formal specification to discover errors and reduce development time © 2006 Department of Computer Science Durham University 8. 8

Codes of practice • It is common for professional bodies to have codes of practice that govern the behaviour of their members • They specify a level of competence. For example, the BCS: – “The quality of every safety-related computer system should be the responsibility of a named engineer, who should be a Chartered Engineer, or a named individual whose competence and qualifications for such tasks have been registered with an appropriate professional body” • Codes of practice are a form of self-regulation and are not legally binding © 2006 Department of Computer Science Durham University 8. 9

Use of standards and guidance • They do not absolve the engineer of responsibility but provide valuable guidance. Can be augmented with codes of practice Engineering Council’s code of professional practice on risk issues • 1 Professional responsibility Exercise reasonable professional skill and care 2 Law Know about and comply with the law 3 Conduct Act in accordance with the code of conduct 4 Approach Take a systematic approach to risk issues 5 Judgement Use professional judgement and experience 6 Communication Communicate within your organisation 7 Management Contribute effectively to corporate risk management 8 Evaluation Assess the risk implications of alternatives 9 Professional development Keep up to date by seeking further training 10 Public awareness Encourage public understanding of risk issues © 2006 Department of Computer Science Durham University 8. 10

IEEE International standard for professional software development and ethical responsibility • “Software engineers shall commit themselves to making the analysis, specification, design, development, testing and maintenance of software a beneficial and respected profession” • Eight principles: – Public, client and employer, product, judgement, management, profession, colleagues, self • Relates to the behaviour of and decisions made by professionals software engineers including practitioners, educators, managers, supervisors, policy makers, trainees and students © 2006 Department of Computer Science Durham University 8. 11

IEEE Eight principles 1. Public – Act consistently with the public interest 2. Client and employer – Act in the best interests of their client and employer, consistent with 1. 3. Product – Ensure their products and related modifications meet the highest professional standards possible 4. Judgement – Maintain integrity and independence in their judgement © 2006 Department of Computer Science Durham University 8. 12

IEEE Eight principles 2 5. Management • Managers and leaders subscribe and promote ethical approach of the management of software development and maintenance 6. Profession • Shall advance the integrity and reputation of the profession with the public interest 7. Colleagues • Shall be fair and supportive of their colleagues 8. Self • Participate in lifelong learning regarding the practise and promote the ethics of their profession © 2006 Department of Computer Science Durham University 8. 13

Adequately safe • Absolute safety is not possible • Goal is to produce a system that is adequately safe, given the benefits • In many cases there are standards related to a product that are enforce by regulatory authorities – These are the minimum requirements • There is also a professional and moral responsibility to make systems as safe as is reasonably possible © 2006 Department of Computer Science Durham University 8. 14

Scenario - Software developer • Relying on questionable inputs • A software professional was assigned the task of developing software to control a particular unit in a large system. Preliminary analysis indicated that the work was well within the state of the art, and no difficulties were anticipated with the immediate task. To function properly, or to function at all, however, the software to be developed required inputs from other units in the system. Someone gave the professional an article by an eminent software specialist that convinced him that the inputs could not be trusted. Thus neither the software he was designing nor the unit his company was providing could correctly accomplish the task. The professional showed the article to his supervisor and explained its significance. The supervisor's response was, "That's not our problem; let's just be sure that our part of the system functions properly. " The software professional continued to work on the project. © 2006 Department of Computer Science Durham University 8. 15

Scenario - Software developer 2 Group 1 Is there an ethics issue involved? Was the software professional's action unethical or not unethical? Was the supervisor's attitude unethical or not unethical? What should be done by whom? Group 2 What risks are involved? What are the benefit/consequence tradeoffs? Who should take the responsibility? © 2006 Department of Computer Science Durham University 8. 16

Readings • Section 4. 7 - Safety-critical systems – Neil Storey • IEEE International standard for professional software development and ethical responsibility http: //seeri. etsu. edu/Codes/The. SECode. htm • Set reading - “The Ethics of Safety-Critical Systems” – Jonathan Bowen – April 2000 Vol 43, No 4 - Communications of the ACM © 2006 Department of Computer Science Durham University 8. 17