Black Boxes and Due Process Transparency in Expert
Black Boxes and Due Process: Transparency in Expert Software Systems AAFS 2016 By Nathan Adams – Adams@Bioforensics. com Dan Krane – Dan. Krane@Wright. edu
Financial disclosure Employees of Forensic Bioinformatic Services, Inc.
Laboratory processes What to do How to do it Did it? Done Developmental validation Internal validation Laboratory protocols Casework Review Report
“Black box” “A device which performs intricate functions but whose internal mechanism may not readily be inspected or understood; (hence) any component of a system specified only in terms of the relationship between inputs and outputs. ” "black box, n. " OED Online. Oxford University Press, December 2015. Web. 19 February 2016.
Validating likelihood ratios • Physical measuring can be validated (underlying true value) • Likelihood ratios (LRs) are more abstract, but we can: • Study underlying models • Measure with “performance tests” Steele, Christopher D. , and David J. Balding. "Statistical evaluation of forensic DNA profile evidence. " Annual Review of Statistics and Its Application 1 (2014): 361 -384.
What can we do? • Validation studies • “The system, as a whole, generates reasonable answers. ” • Systemic, holistic, sum of the parts • Examine the underlying models • “These models are independently reliable. ” • Each part works right
What can we do? • Validation study guidelines • FBI QAS, SWGDAM, etc. • Examine the underlying models • Consult your neighborhood statistician and molecular biologist about uncertainty and stochastic effects. Google Scholar citations. Accessed 19 February 2016. Available at https: //scholar. google. com/scholar? cites=16343486730052257994
Laboratory processes What to do How to do it Did it? Done Developmental validation Internal validation Laboratory protocols Casework Review Report
Software engineering “the systematic application of scientific and technological knowledge, methods, and experience to the design, implementation, testing, and documentation of software. ” ISO/IEC/IEEE, “Systems and software engineering -- Vocabulary, ” ISO/IEC/IEEE 24765: 2010(E), vol. 2010.
Software development Requirements Specification Design Implementation Testing Maintenance What it does How it does it Source code Software testing Fixes & updates
Matching game Software Lab Requirements Developmental validation Specification Internal validation Design These Laboratory protocols terms are not Casework Implementation interchangeable Testing Review Maintenance Report
Matching game Software Lab Requirements Specification Design Laboratory protocols Implementation Casework Testing Maintenance
Open source: Panacea? “Open source software is highly desirable in the court environment because openness to scrutiny by any interested party is an invaluable source of bug reports and suggestions for improvement. ” Implementation Source code Steele, Christopher D. , and David J. Balding. "Statistical evaluation of forensic DNA profile evidence. " Annual Review of Statistics and Its Application 1 (2014): 361 -384.
Case studies • Therac-25 • Radiation therapy machine • Killed 6 patients in 1980 s by radiation overdosing (software errors + negligence) • Classic case study in software engineering • Alcotest 7110 Mk III-C • Breath alcohol analyzer software • Subjected to code review/testing
Therac-25 • “A ‘small amount’ of software testing was done…” • Users were not software experts • Oversight and regulation were reactive, not proactive • Proprietary claim to software design Leveson, N. G. , & Turner, C. S. (1993). An investigation of the Therac-25 accidents. Computer, 26(7), 18– 41.
Therac-25 “The materials submitted by the manufacturer have not been in sufficient detail and clarity to ensure an adequate software QA program currently exists. ” Leveson, N. G. , & Turner, C. S. (1993). An investigation of the Therac-25 accidents. Computer, 26(7), 18– 41.
Alcotest 7110 Mk III-C • “Key Findings- Testing the Alcotest 7110 Mk III Source Code uncovered 24 major defects. ” • “Alcotest software would not pass U. S. Industry Standards for Software Development and Testing” • “Proof of incomplete software testing” Wisniewski, John. Report: Alcotest 7110 Mk III C. Base One Technologies. Accessed 15 March 2015. Available at https: //www. dwi. com/files/2012/04/1174 -base-one-findings. pdf
Alcotest 7110 MKIII-C breath alcohol analyzer system • “Catastrophic Error Detection is Disabled” • “Diagnostics Adjust/Substitute Data Readings” • “Error Detection Logic” Wisniewski, John. Report: Alcotest 7110 Mk III C. Base One Technologies. Accessed 15 March 2015. Available at https: //www. dwi. com/files/2012/04/1174 -base-one-findings. pdf
Acknowledgements Special thanks to: David Bentley Jennifer Friedman Pam King Jennifer Mnookin Michael Risinger Bill Thompson
Black Boxes and Due Process: Transparency in Expert Software Systems AAFS 2016 By Nathan Adams – Adams@Bioforensics. com Dan Krane – Dan. Krane@Wright. edu Available at – www. bioforensics. com
- Slides: 20
