Software Cybernetics The Next Frontier Aditya P Mathur

Software Cybernetics: The Next Frontier Aditya P. Mathur Professor Department of Computer Sciences Purdue University, West Lafayette 15 th Anniversary Celebration of Society for Design and Process Science (SDPS) , Dallas, TX, USA Friday June 11, 2010 1

Collaborators Kai-Yuan Cai, Beijing University of Aeronautics and Astronautics Joao Cangussu, Microsoft Ray De. Carlo, Purdue Scott Miller, Purdue 2

Key Question Can we model and control a software development process using theories and techniques available to control engineers? 3

What is Software Cybernetics? A (new) discipline that aims at the development of models and control methodologies for software and the software development processes. 4

Focus of Software Cybernetics • Formalization and quantification of feedback mechanisms in software processes and systems • Adaptation of principles and techniques of control theory to software processes and systems • Application of the principles of software related theories and engineering to control systems and processes • Integration of theories of software engineering and control engineering 5

Software and Control Software • Code • Development • Maintenance • Evolution Control • A finite or infinite sequence of actions applied to the controlled object to achieve a given apriori goal for the controlled object. 6

Software Cybernetics: Domain Modeling and Simulation: Abdel-Hamid� and S. E. Madnick. Software Project Dynamics. Prentice Hall 1991. Role of feedback in software evolution [Basili et al. 1994; Lehman, 1996] 7

Software Cybernetics: Domain Modeling and control for decision support: Software synthesis [Sridharan et al. 2003] Software test process control [Cangussu et al. 2002] Adaptive testing [Cai 2002] Software development process control, incremental lifecycle model [Miller et al. 2008] 8

Software Cybernetics: Domain Modeling and control for reliable operation: Eliminating Concurrency Bugs with Control Engineering [Kelly et al. IEEE Computer 2009] 9

Feedback Control: The basic idea Specifications Effort + Program Additional effort Required Quality Observed Quality - f(e) What is f ? 10

Software Development Process: Definitions A Software Development Process (SDP) is a sequence of well defined activities used in the production of software. An SDP usually consists of several sub-processes that may or may not operate in a sequence. The Design Process, the Software Test Process, and the Configuration Management Process are examples of sub-processes of the SDP. 11

Software Development Process: A Life Cycle Techniques proposed here can be adapted to any life cycle model. Requirements Elicitation Requirements Analysis Design Code/Unit test Integrate/Test System test Not all feedback loops are shown. More test Deploy 12

Study 1: Modeling the system test phase Software Test Process (STP): System test phase; physical system analogy Objective: Control the STP so that the quality of the tested software is as desired. Quantification of quality of software: • Number of remaining errors • Reliability 13

Study 2: Modeling the development process Software Development Process (SDP): All phases, queuing system anaogy [Miller et al. COMPSAC 2004] Objective: Offer change suggestions capable of achieving a specified deviation from the predicted system trajectory. Quantification of quality of software: • Number of remaining errors • Reliability 14

The System Test Phase: Parametric Control Cangussu et al. TSE 2002 15

Problem Scenario r 0 r - number of remaining errors observed Approximation of how r is likely to change schedule set by the manager rf cp 1 cp 2 cp 3 cp 4 cp 5 cp 6 cp 7 cp 8 cp 9 t 0 t- time deadline cpi = check point i 16

Our Approach s c r 0 + Initial Settings (wf, ) wf+ wf + Actual STP ’ Controller Test Manager wf + wf: workforce : quality of the test process robserved(t) rerror(t) w’f STP State Model s c r 0 rexpected(t) 17

Physical and Software Systems: An Analogy Dashpot External force To err is Human Block Spring Xequilibrium Xcurrent Software X: Position Number of remaining errors Rigid surface Spring Force Effective Test Effort Mass of the block Software complexity Viscosity Quality of the test process 18

Physical Systems: Laws of Motion [1] First Law: Every object in a state of uniform motion tends to remain in that state of motion unless an external force is applied to it. Does not (seem to) apply to testing because the number of errors does not change when no external effort is applied to the application. 19

Physical Systems: Laws of Motion [2] Newton’s Second Law: The relationship between an object's mass m, its acceleration a, and the applied force F is F = ma. CDM First Postulate: The relationship between the complexity Sc of an application, its rate of reduction in the number of. . remaining errors, and the applied effort E is E=Sc r. 20

Physical Systems: Laws of Motion [3] Third Law: For every action force, there is an equal and opposite reaction force. When an effort is applied to test software, it leads to (mental) fatigue on the tester. Unable to quantify this relationship. 21

CDM First Postulate The magnitude of the rate of decrease of the remaining errors is directly proportional to the net applied effort and inversely proportional to the complexity of the program under test. This is analogous to Newton’s Second Law of motion. 22

CDM Second Postulate Analogy with the spring: The magnitude of the effective test effort is proportional to the product of the applied work force and the number of remaining errors. for an appropriate . Note: While keeping the effective test effort constant, a reduction in r requires an increase in workforce. 23

CDM Third Postulate Analogy with the dashpot: The error reduction resistance is proportional to the error reduction velocity and inversely proportional to the overall quality of the test phase. for an appropriate . Note: For a given quality of the test phase, a larger error reduction velocity leads to larger resistance. 24

State Model Force (effort) balance equation: : Disturbance. x(t) = Ax(t) + B u(t) 25

Computing the feedback-Question Given: r(T): the number of remaining errors at time T r(T+ T): the desired number of remaining errors at time T+ T Question: What changes to the process parameters will achieve the desired r(T+ T) ? 26

Computing the feedback-Answer From basic matrix theory: The largest eigenvalue of a linear system dominates the rate of convergence. Hence we need to adjust the largest eigenvalue of the system so that the response converges to the desired value within the remaining weeks ( T). This can be achieved by maintaining: Obtain the desired eigenvalue. 27

Computing the feedback-Calculations ( max) Given the constraint: Compute the desired max We know that the eigenvalues of our model are the roots of its characteristic polynomial of the A matrix. 28

Computing the feedback-Calculations ( max) We use the above equation to calculate the space of changes to wf and such that the system maintains its desired eigenvalue. 29

Computing the feedback-Input to the Manager The space of changes in the workforce and the quality of the process is made available to the test manager in the form of suggestions for possible process changes. The test manager may decide to select a combination of these values for implementation or simply ignore them. In each of the two commercial studies we carried out, the manager ignored the suggestions given using the model. 30

Case Study: The Razorfish Project Goal: translate 4 million lines of Cobol code to SAP/R 3 A tool has been developed to achieve the goal of this project. Goal of the test process: (a) Test the generated code, not the tool. (b) Reduce the number of errors by about 85%. 31

Razorfish Project Test Process continue testing output 1 yes = output 2 no run SCobol modify Transformer run SSAP R/3 input Select a Test Profile 32

Razorfish Project: Results (intermediate) Expected behavior Project data Prediction using the model Prediction using feedback If the process parameters are not altered then the goal is reached in about 35 weeks. 85% reduction achieved. 33

Alternatives from Feedback: STP Quality Desired eigenvalue=-0. 152 Improving quality alone will not help in achieving the goal. 34

Alternatives from Feedback: Workforce Desired eigenvalue=-0. 152 Changing the workforce alone can produce the desired results. 35

Alternatives from Feedback: STP quality and workforce Set of valid choices for changing the quality and the workforce 36

Razorfish Project Results (final) The project was completed in 32 weeks. The model predicted 85% error reduction in 35 weeks. 37

The incremental development process: Queuing model and control Miller et al. COMPSAC 2008 38

Interconnected dynamical systems* u C 1 C 2 C 4 y C 3 *De. Carlo and Saeks, 1980 C: Component (sub-system) u: System input y: System output x: System state vector 39

Incremental software development: Components R R: Requirements 40

Component data dependencies 41

System of systems model 42

System of systems model L 11: Linear feedback from sub-system outputs to sub-system inputs L 12: Entry of external inputs into subsystem inputs L 21, L 22: Similar relationships for system outputs 43

Workforce productive capability : : productivity measure workforce size remaining work items process quality : calibrated average productive capability/FTE 44

Productivity state model 45

Productivity state model: behavior Cangussu et al. [2002] (Csikszementmihalyi, 1988) 46

State model of a queue a 1 a 2 b 1 (x 1) b 2 (x 2) b 3 (x 3) 47

Coordination constraints x: features coded c(x): step function; test case execution threshold Cubic splines used for smoothing the threshold function. 48

Additional models Defect introduction: Based on Barry Boehm’s COQUALMO model Defect detection; failure analysis: Lotka-Volterra predator-prey population dynamics model Proportional splitting model: defects in code versus defects in test case code 49

Simulation study: System description • • Incremental development process Two serial increments First increment: 10 features, 70 tests, 70 regression tests Second increment: 20 features, 130 tests, 130 regression tests 29 FTE: 5 assigned to each development activity except feature and test correction that are assigned 5 each. 50

Simulation study: Sample results 51

Simulation study: Sample results 52

Summary Analogy between physical and software systems presented. The notion of feedback control of software processes introduced. Two case studies described. Parameter estimation techniques used for model calibration. Made use of system identification techniques. 53

Challenges Control objects: humans Time constants: large; process changes are slow Parameter estimation: need for data collection; must account for the impact of changes in the development environment Tool: needed to assist the manager evaluate multiple change scenarios 54

Thank you! 55