Improving Traceability Management through Tool Integration an Experience

Improving Traceability Management through Tool Integration: an Experience in the Automotive Domain Domenico Amalfitano Vincenzo De Simone Anna Rita Fasolino Stefano Scala ICSSP 2017 – Paris, France – July 6 th

�Context ◦ Embedded software development process management �Model-in-the-loop (MIL) testing process �Goal ◦ MIL Testing Process Improvement �Contribution ◦ Implementation of a Tool Integration Architecture for automating: �Testing Process execution �Traceability management ICSSP 2017 – Paris, France – July 6 th

2 Embedded Software Development Process in FCA V-Model relationships between each phase of the development life cycle and its associated phase of testing Model-Based Design Development Process Development focuses on modeling. Code is automatically generated. MIL Validation 1 Simulink or Stateflow models validated in a simulated environment Traceability Management Artifacts are managed and traceability links are created ICSSP 2017 – Paris, France – July 6 th

Traceability Software Traceability = creation and the use of links (or connections) between different kinds of software artifacts such as requirements, models, source code, test cases, or test results. It is required by Evaluation Frameworks and Safety Standards. Composed of Test Run To test a Test Case Software Component Implements a Requirement ICSSP 2017 – Paris, France – July 6 th

Issues of Traceability Management Difficulties in implementing and achieving software traceability due to: �Development by distributed teams �Artifacts and links undergo constant change �Involve multiple stakeholders with different background �Trace Links often manually created or not created at all ICSSP 2017 – Paris, France – July 6 th

Application Lifecycle Management �Provides features for supporting all the software development process (artifacts, actors, roles, tasks, etc. ) �Aims at improving collaboration, efficiency and ICSSP 2017 – Paris, France – July 6 th

ALM Main Features �Planning �Resource Management �Artifact Management �Workflow Management �Change & Configuration Management �Build & Release Management �Issue Management �Support for collaboration �Audits, Metrics and Reports ICSSP 2017 – Paris, France – July 6 th

Current MIL Testing Process in FCA – from the POV of the Test Engineer MX-Suite MTP Create Test Run for the SWC Create Traceability Links Select Test Cases and Models Import Test Cases Results Launch Test Cases Execution Wait for the termination of Test Cases Execution ICSSP 2017 – Paris, France – July 6 th

Current MIL Testing Process in FCA – from the POV of the Project Manager TE PM TE TE Monitor Testing Process Execution The PM can obtain the information about the MIL Testing Process Execution by querying the Polarion ALM. This information does not reflect the actual state of the process execution. He need to directly query the Test Engineers involved in the process. TE TE PM Project Manager TE Test Engineer ICSSP 2017 – Paris, France – July 6 th

MIL Testing Process Many testing process phases are still manually executed using heterogeneous and not integrated tools. This negatively affects Issues process quality in terms of: • Visibility Absence of accurate, reliable and timely information. Difficult to follow the process progress and results. • Rapidity High testing process execution time due to long time needed to exchange data between the different tools • Ineffective Traceability Management Manual creation of the needed traceability links, that was time consuming and error prone • Acceptability Test Engineers have to use many different tools at the same time and execute tedious, time consuming and error prone manual tasks Possible Solution Integration of different tools to automate the testing process execution and the creation of the needed traceability links ICSSP 2017 – Paris, France – July 6 th

Proposed Tool Integration Architecture 1/3 A Tool Integration Architecture has been designed to achieve improvements in the Software Testing Process. To this aim we defined: � Connectors to enable communication between different tools � A well defined data-exchange format It has been implemented: � Adapting a Commercial Off-The-Shelf solution to exploit already provided functionality and improve maintainability Jenkins has been chosen as the integration framework to implement the Tool Integration Architecture. It offers: � A distributed architecture with a master/slave paradigm � Extensibility through Java plugins MX-Suite MTP ICSSP 2017 – Paris, France – July 6 th

Proposed Tool Integration Architecture 2/3 ICSSP 2017 – Paris, France – July 6 th

Proposed Tool Integration Architecture 3/3 �Coordinator ◦ orchestrates the Testing process tasks through the features offered by Jenkins automation engine �Data Connectors ◦ integrate the data involved in MIL Testing Process, e. g the Test Results produced by the MTP Matlab Testing Environment with the data to introduce in Polarion ALM �Reverse Engineering of data models �Call Connectors ◦ call the involved tools when needed �Need to wrap the features exposed by legacy testing tool �Reverse engineering of the tool features and definition of callable APIs ICSSP 2017 – Paris, France – July 6 th

Reverse Engineering of Data Models Most of the artifacts involved in the Testing Process are Spreadsheets �Application of Reverse Engineering process to recover the data model from the spreadsheet documents 1 to correctly integrate them 1 Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Vincenzo De Simone, Giancarlo Di Mare, Stefano Scala: A Reverse Engineering Process for Inferring Data Models from Spreadsheet-based Information Systems: An Automotive Industrial Experience. DATA (Revised Selected Papers)2014 ICSSP 2017 – Paris, France – July 6 th

Reverse Engineering of Data Models � The execution of the Reverse process was supported by EXACT 2 It reports the main structure of the spreadsheet By clicking on an element, further details . . . a view on the showing its element … dependencies is reported 2 Domenico Amalfitano, Vincenzo De Simone, Anna Rita Fasolino, Porfirio Tramontana: EXACT: A tool for comprehending VBA-based Excel spreadsheet applications. Journal of Software: Evolution and Process 28 ICSSP 2017 – Paris, France – July 6 th

Proposed MIL Testing Process in FCA – from the POV of the Test Engineer Create & Launch Test Run for the SWC Select Test Cases and Models MX-Suite MTP Test Cases Execution Create Traceability Links Import Test Cases Result ICSSP 2017 – Paris, France – July 6 th

MIL Testing Process Execution – ALM Page ICSSP 2017 – Paris, France – July 6 th

Proposed MIL Testing Process in FCA – from the POV of the Project Manager PM Monitor MIL Testing Process Execution The PM can directly obtain the information about the MIL Testing Process Execution by querying the Polarion ALM. This information reflects the actual state of the process execution. No further action is required. ICSSP 2017 – Paris, France – July 6 th

Case Study in FCA EMEA SW Factory Comparison of the MIL Testing Process using the proposed Tool Integration Architecture against the usual company practice, with the aim to understand: Ø RQ 1 How does the adoption of the proposed solution affect the process rapidity? Ø RQ 2 How does the adoption of the proposed solution influence the effectiveness of the traceability links management related to the process? Ø RQ 3 How does the adoption of the proposed solution impact on the process visibility from the point of view of the Project Manager? Ø RQ 4 How does the adoption of the proposed solution impact the process acceptability from the point of view of the Test Engineers? ICSSP 2017 – Paris, France – July 6 th

Subjects & Objects Subjects: � 1 Project Manager and 3 Test Engineers Objects: 3 already tested Software Components (SWC) SWC Available Test Cases Speedometer 34 Tachometer 75 Trip 150 ICSSP 2017 – Paris, France – July 6 th

Objects Instrument Panel Cluster (IPC): set of instrumentation that displays with a digital readout information to the driver FCA ® Speedometer: controls the gauges that measure and display the instantaneous speed of a vehicle Tachometer: controls the gauges that show the rate of rotation of the engine's crankshaft Trip: provides features for record, calculate, and display the distance travelled by the car. ICSSP 2017 – Paris, France – July 6 th

Evaluation �Rapidity ◦ MIL Testing Process Execution Time �Effectiveness of Traceability Management ◦ #ITL incorrect traceability links = links created between artifacts that are not related ◦ #MTL missing traceability links = no traceability link is established between artifacts that are related �Visibility ◦ Qualitative analysis through semi-structured interview with the Project Manager �Acceptability ◦ Qualitative analysis through semi-structured interviews with the Test Engineers ICSSP 2017 – Paris, France – July 6 th

Comparison Results � SWC MIL CTR STC LTC ITCR Speedometer 7% 0% 73% 0% 79% Tachometer 9% 0. 1% 81% 0% 90% Trip 14% 0. 1% 91% 0. 3% 96% MIL = Entire MIL Testing Process CTR = Create Test Run STC = Select Test Cases LTC = Launch Test Cases ITCR = Import Test Cases ICSSP 2017 – Paris, France – July 6 th Result

Comparison Results �Effectiveness Traceability Management SWC ITLRP MTLRP Speedometer 100% Tachometer 100% Trip 100% ICSSP 2017 – Paris, France – July 6 th

Comparison Results �Visibility ◦ Improved from the point of view of the Project Manager �He report the possibility to monitor the progress and results of the MIL testing process execution when needed �In the past he could obtain the needed information only on defined milestones or by directly querying the involved Test Engineers ICSSP 2017 – Paris, France – July 6 th

Comparison Results �Acceptability ◦ Improved from the point of view of the Test Engineers �No need to switch between different tools since the execution of their tasks is carried out through one tool (the ALM). The need for manual interventions was reduced �In the past they had to switch between different tools executing tedious manual tasks (e. g. storing of the test results) ICSSP 2017 – Paris, France – July 6 th

Conclusions �Tool Integration Architecture aids the execution of Testing processes by automating tedious and time-consuming tasks �Traceability can be automatically established when new artifacts are created in the automatic Testing Process Future Work execution �Wider experimentation in different domains �Other Verification & Validation activities can be integrated and automated through the Tool Integration Architecture ICSSP 2017 – Paris, France – July 6 th

Thanks for your attention Questions? Feedbacks ? Further Information: http: //reverse. dieti. unina. it @REv. ERSE_UNINA vincenzo. desimone 2@unina. it ICSSP 2017 – Paris, France – July 6 th