A Embedded software component quality verification framework Fernando

A Embedded software component quality verification framework Fernando Ferreira de Carvalho Advisor: Silvio Romero de Lemos Meira Informatics Center - Federal University of Pernambuco ffc@cin. ufpe. br Recife, 23 de Dezembro 2008

A Embedded software component quality verification framework 1 - Introduction / Motivation / Problem Embedded system is at the heart of many systems Needs of embedded system industry, • • • Low production cost Short time to market High Quality, and others to be more efficient and competitive (Brown, 2000) The CBD with reuse technique had been a nice direction to reach this objectives… The Robust framework for Software Reuse [Almeilda et al. , 2004] is good way to support CDB approach and component reuse for embedded systems. But, Component reuse without quality assurance give catastrophic results [Jezequel et al. , 1997]. So, components quality assurance is crucial before stored in a Repository System. 2 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework Proposed solution An Embedded Software Component Quality Verification Framework It is composed of four inter-relation module: , based on a set of activities, metrics and guidelines. • Embedded software component Quality Model (EQM) • Maturity Level evaluation Techniques • Metrics Approach • Component Certification based on a set of activities, metrics and guidelines. 3 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework Proposed solution This Framework is based in the standards • ISO/IEC 9126, 2001 - Quality Model for Software Product • ISO/IEC 14598, 1998 - Software Product Evaluation Process This two standards converged to: • ISO/IEC 25010, 2005 - Software product quality requirements and evaluation The Framework adapted the quality model and evaluation to component context and embedded domain. 4 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework Out of scope This Framework is part of broad context, some aspects were expected since initial definition. Nevertheless, other process can be added in the future. • Cost Model • Formal Proof • Prediction of the component assembly 5 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 2 – Embedded System Design Embedded system design comprise: • Ultra-small device x simple functionality • Small system x sophisticated functions • Large systems and distributed systems • Systems produced in large quantities x low production cost • Systems produced in low volume x important features 6 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 2 – Embedded System Design The different requirements of embedded systems have a impact on feasibility, on use of CBD for it. A common characteristic in different area of embedded domain is increasing importance of software [Crnkovic, 2003]. Example, the software cost in embedded systems: • in industrial robots constitute about 75% of total cots • in car industry it is about 30% Fifteen year ago: • 25% of total cots in industrial robots • Negligible for cars 7 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 2 – Embedded System Design Properties that involves embedded software component is divided in: • Functional property (component interface) • Non-functional or Extra-functional property, so called Quality attributes, fox example: • Timing • Performance • Consumption • Resource Behavior, and others. This properties can be classified in run-time and life-cycle. 8 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 2 – Specific Requirements for Embedded System In the most of case, embedded system is real-time with limited resource. So, it has specifics characteristics which depends on domain application, but it have strong implication on requirements. The REQUIREMENTS are related Extra-functional property or Quality attributes, and its priority depends on the domain application. 9 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 2 – Specific Requirements for Embedded System There has been developed a research in order to find the most important characteristics in different areas in embedded domain. • Industrial Automation • Automotive • Medical • Electronic consumer • Other domain 10 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 2 – Specific Requirements for Embedded System – Industrial Automation was classified by research’s Larsson, [Larsson, 2002] The most important characteristics, following the research: At low level: a. Availability b. Timeliness c. Reliability At high level: a. Performance b. Usability c. Integrability 11 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 2 – Specific Requirements for Embedded System - Automotive Akerholm [Akerholm, 2005] executed a research in vehicle industry. The resulting list of characteristics is presented below 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Safety Reliability Predictability Usability Extendibility Maintainability Efficiency Testability Security Flexibility 12 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 2 – Specific Requirements for Embedded System - Medical Wijnstra [Wijnstra, 2001] describe their experience with characteristics in the development of medical imaging family. The resulting list of characteristics is presented below 1. 2. 3. 4. 5. 6. 7. Reliability Safety Functionality Portability Modifiability a. Configurability b. Extensibility and Evolvability Security Serviceability 13 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 2 – Specific Requirements for Embedded System – Others Domain Crnkovic [Crnkovic, 2003] summarized the main characteristics and subcharacteristics in the CBD approach apply to embedded system in his research. The table show the results. Characteristics Real-time properties Sub-characteristics Response time or latency execution time worst case execution time Deadline Dependability Reliability Availability integrity confidentiality safety Resource consumption Power consumption computation (CPU) power memory Consumption Life cycle properties 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br) execution (CPU) time, maintainability 14

A Embedded software component quality verification framework 2 – Embedded System Design – Software component quality So, embedded software component quality verification must be different that general propose component, because the component evaluation is realized focused in specifics requirements We divided the quality verification in two groups: • General propose software component quality process o desktops, servers, x 86 architecture • Specific propose software component quality process o embedded systems 15 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey The relevant research explore theory of component quality and certification in academic scenarios, but not rich in reports in practical experience. The pioneering works focus in mathematical and test model, while recent researchers have focused in techniques and model based on predicting quality requirements. 16 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey Timeline of research in the embedded software component quality and certification area proposed standard X fail → a work was extended by another 17 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey In 1993, Poore [Poore et al. , 1993] develop an approach based on three mathematical model (sampling, component and certification models), using test cases to report the failures to achieve a reliability index Poore estimated the reliability of a complete system, and not of individual software units, although, they did consider how each component affected the system’s reliability. 18 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey Wohlin [Wohlin et al. , 1994] presented the first method of component certification using modeling techniques, making it possible not only to certify components but to certify the system. • It is composed of the usage model and the usage profile. • The failure statistics from the usage test form the input of a certification model. • An interesting point of this approach is that the usage and profile models can be reused in subsequent certifications However, even reusing those models, the considerable amount of effort and time that is needed makes the certification process a hard task. 19 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey In 1994, Merrit (Merrit, 1994) presented an interesting suggestion: the use of components certification levels. These levels depend on the nature, frequency, reuse and importance, as follows: • Level 1: No tests are performed; the degree of completeness is unknown; • Level 2: A source code component must be compiled and metrics are determined; • Level 3: Testing, test data, and test results are added; and • Level 4: A reuse manual is added. These levels represent an initial component maturity model. However, this is just a suggestion of certification levels and no practical work was actually done to evaluate it. 20 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey In 1996, Rohde (Rohde et al. , 1996) provided a reuse and certification of embedded software components at Rome Laboratory of the US Air Force, a Certification Framework (CF), that included: • To define the elements of the reuse context that to certification; • To define the underlying models and methods of certification; and, • To define a decision-support technique to construct a context-sensitive process for selecting and applying the techniques and tools to certify components. 21 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey • A Cost/Benefit plan that describes a systematic approach of evaluating the costs and benefits. Rohde et al. considered only the test techniques to obtain the defects result in order to certify software components. This is only one of the important techniques that should be applied to component certification. 22 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey Voas [Voas, 1998] defined a certification methodology using automated technologies, such as black-box testing and fault injection to determine if a component fits into a specific scenario. This methodology uses three quality assessment techniques: (i) Black-box component testing determine if the component quality is high enough; (ii) System-level fault injection determine how well a system will tolerate a faulty component; (iii) Operational system testing determine how well the system will tolerate a properly functioning component 23 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey According to Voas, this approach is not foolproof and perhaps not well suited to all situations. The methodology does not certify that a component can be used in all systems. This approach certify a component within a specific system and environment. 24 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey Wohlin and Regnell [Wohlin and Regnell, 1998] extended their previous research (Wohlin et al. , 1994), now, focusing on techniques for certifying both components and systems. Thus, the certification process includes : (i) usage specification (consisting of a usage model and profiles), and (ii) certification procedure, using a reliability model. 25 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey The main contribution of that work is the division of components into classes for certification and the identification of three different ways of certifying software systems: i. Certification process, the functional requirements are validated during usage-based testing; ii. Reliability certification of component and systems, the component models that were built are revised and integrated to certify the system that they form; and, iii. Certify or derive system reliability, where the focus is on reusing the models that were built to certify new components or systems. 26 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey However, the proposed methods are theoretical without experimental study. According to Wohlin et al. , “both experiments in a laboratory environment and industrial case studies are needed to facilitate the understanding of component reliability, its relationship to system reliability and to validate the methods that were used only in laboratory case studies” (pp. 09). Until now, no progress in those directions was achieved. 27 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey In 2000, Jahnke, Niere and Wadsack [Jahnke, Niere and Wadsack, 2000] developed a methodology for semi-automatic analysis of embedded software component quality. This approach evaluates data memory (RAM) utilization in Java technology by the component. The work is restricted because: - Verifies the component quality from only one point of view, use of data memory in a specific language, - Java is widely used for the development of desktops systems not useful for embedded development. 28 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey Stafford (Stafford et al. , 2001) developed a model for the component marketplaces that supports prediction of system properties prior to component selection. The model use functional verification and quality-related values associated with a component, called credentials. This work introduced notable changes in this area. It use a specific notation such as <property, value, credibility>. Through credentials, the developer chooses the best components to use in the application development based on the “credibility” level. 29 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey Stafford introduced the notion of active component dossier, its is an abstract component that defines credentials. Stafford et al. finalized their work with some open questions, such as: • • • how to certify measurement techniques? What level of trust is required under different circumstances? Are there other mechanisms that might be used to support trust? 30 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey i. iii. iv. In 2002, Comella-Dorda et al. (Comella-Dorda et al. , 2002) proposed a COTS software product evaluation process. The process contains four activities, as follows: Planning the evaluation -> evaluation team, stakeholders, required resources, basic characteristics of the evaluation Establishing the criteria -> evaluation requirements , evaluation criteria; Collecting the data -> component data are collected, the evaluations plan is done and the evaluation is executed; and Analyzing the data -> the results of the evaluation are analyzed and some recommendations are given. The proposed process is an ongoing work and, no real case study was accomplished, becoming unknown the real efficiency. 31 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey In 2003, Beus-Dukic (Beus-Dukic et al. , 2003) proposed a method to measure quality characteristics of COTS components, based on the international standards for software product quality (ISO/IEC 9126, ISO/IEC 12119 and ISO/IEC 14598). The method is composed of four steps: i. Establish evaluation requirements, specifying the purpose and scope of the ii. iv. evaluation, specifying evaluation requirements; Specify the evaluation, selecting the metrics and the evaluation methods; Design the evaluation, considers the component documentation, development tools, evaluation costs and expertise required in order to make the evaluation plan; Execute the evaluation, the execution of the evaluation methods and the analysis of the results. However, the method proposed was not evaluated in a real case study 32 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey In 2003, Hissam (Hissam et al. , 2003) introduced Prediction. Enabled Component Technology (PECT) as a means of packaging predictable assembly. This work, which is an evolution of Stafford et al. ’s work (Stafford et al. , 2001), attempts to validate the PECT and its components, giving credibility to the model 33 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey During 2003, a CMU/SEI’s report, Wallnau extended Hissam work (Hissam et al. , 2003), in order to achieve Predictable Assembly from Certifiable Components (PACC). This novel model requires a better maturation by the software engineering community in order to achieve trust in it 34 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey Magnus Larsson, in 2004 (Larsson, 2004), define A predictability approach of the quality attributes, where one of the main objectives is to enable integration of components as black boxes. According to composition principles, results types of attributes: • Directly compassable attributes. is a function of only the same attribute. • Architecture-related attributes. is a function of the same attribute and of the software architecture. • • Derived attributes. depends on several different attributes Usage-depended attributes. is determined by its usage profile. This work is very useful, but before the component quality must be known. 35 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey Finally, in 2006 Daniel Karlson (Karlson et al. , 2006) presented the verification of component-based embedded system designs. These techniques is Formal Methods based modeling approach(Petri net), called PRES+. Two problems are addressed: • component verification and • Integration verification. This approach verifies the component from only one perspective: functionality. Formal verification, it is used only in few cases when it is mandatory, because much time and financial effort are employed. 36 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey Failures in Software Component Certification Two failure cases that can be found in the literature. First failure occurred in the US government, when trying to establish criteria for certificating components (NIAP). Thus, from 1993 until 1996, NSA and the NIST used the Trusted Computer Security Evaluation Criteria (TCSEC), “Orange Book”. It had defined no means of features across classes of components, but only for a restricted set of behavioral assembly properties (Hissam et al. , 2003). 37 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: A Survey Failures in Software Component Certification The second failure happened with an IEEE committee, in an attempt to obtain a component certification standard. The initiative was suspended, in this same year. The committee came to a consensus that they were still far from getting to the point where the document would be a strong candidate for a standard. (Goulao et al. , 2002 a). 38 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization One of the main objectives of software engineering • • Improve the quality of software products, Establishing methods and technologies to build software products. The quality area could be basically divided into two main topics (Pressman, 2005): • • Software Product Quality: aiming to assure the quality of the generated product; and Software Processes Quality: looking for the definition, evaluation and improvement of software development processes. 39 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization Software Product Quality: • • ISO/IEC 9126 (ISO/IEC 9126, 2001), ISO/IEC 12119 (ISO/IEC 12119, 1994), ISO/IEC 14598 (ISO/IEC 14598, 1998), SQua. RE project (ISO/IEC 25000, 2005) (Mc. Call et al. , 1977), (Boehm et al. , 1978), among others Software Processes Quality: • • • Capability Maturity Model (CMM) (Paulk et al. , 1993), Capability Maturity Model Integrated (CMMI) (CMMI, 2000), Software Process Improvement and Capability d. Etermination (SPICE) (Drouin, 1995), among others 40 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization Many institutions are creating standards to properly evaluate the quality and development processes of the software product, in different domain. The Table shows a set of national and international standards. Standards ISO/IEC 61131 component-based approach for industrial systems RTCA DO 178 B guidelines for development of aviation software ISO/IEC 61508 Security Life cycle for industrial software ISO/IEC 9126 Software Products Quality Characteristics ISO/IEC 14598 Guides to evaluate software product, based on practical usage of the ISO 9156 standard ISO/IEC 12119 Quality Requirements and Testing for Software Packages SQua. RE project (ISO/IEC 25000) Software Product Quality Requirements and Evaluation IEEE P 1061 Standard for Software Quality Metrics Methodology ISO/IEC 12207 Software Life Cycle Process. NBR ISO 8402 Quality Management and Assurance. NBR ISO 9000 -1 -2 Model for quality assurance in Design, Development, Test, Installation and Servicing NBR ISO 9000 -3 Quality Management and Assurance. Application of the ISO 9000 standard to the software development process (evolution of the NBR ISO 8402). CMMI (Capability Maturity Model Integration) SEI’s model for judging the maturity of the software processes of an organization and for identifying the key practices that are required to increase the maturity of these processes. ISO 15504 25/05/2021 Overview Fernando Carvalho (ffc@cin. ufpe. br) It is a framework for the assessment of software 41 processes.

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization ISO/IEC 25000, 2005 / SQua. RE project - Software Product Quality Requirements and Evaluation has been created specifically to make two standards converge: • ISO/IEC 14598, 1998 - define a software product evaluation process, based on the ISO/IEC 9126. • ISO/IEC 9126, 2001 - define a quality model for software product Trying to eliminate the gaps, conflicts, and ambiguities that they present. 42 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization ISO/IEC 25000, 2005 / SQua. RE project The objective is : • To respond to the evolving needs of users through an improved, and • Unified set of normative documents covering three complementary quality processes: • Requirements specification, • Measurement and • Evaluation. The motivation is to supply for developing and acquiring software products with quality engineering instruments supporting both the specification and evaluation of quality requirements. 43 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization SQua. RE include: • • • Criteria for the specification of quality requirements Evaluation of quality requirements, Recommended measures of software product quality attributes. which can be used by: • Developers, • Acquirers, and • Evaluators. 44 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization Quality Requirements Division (ISO/IEC 2503 n) ISO/IEC 25030 - 2007, standard for supporting the specification of quality requirements, either during software product quality requirement elicitation or as an input for an evaluation process: Quality requirements and guide: to enable software product quality to be specified in terms of quality requirements; 45 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization Quality Model Division (ISO/IEC 2501 n) ISO/IEC 25010 – 2005, contains the detailed quality model and its specific characteristics and sub-characteristics for internal quality, external quality and quality in use. This division includes: Quality model and guide: to describe the model for software product internal and external quality, and quality in use. The document present the characteristics and subcharacteristics for internal and external quality and characteristics for quality in use. 46 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization Product Quality General Division (ISO/IEC 2500 n) ISO/IEC 25000 – 2005 contains the unit standards defining all common models, terms and definitions referred to by all other standards in the SQua. RE series. This division includes two unit standards: • Guide to SQua. RE: to provide the SQua. RE structure, terminology, document overview, intended users and associated parts of the series, as well as reference models; • Planning and management: to provide the requirements and guidance for planning and management support functions for software product evaluation. 47 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization Quality Measures Division (ISO/IEC 2502 n) ISO/IEC 25020 - 2007 were derived from ISO/IEC 9126 and ISO/IEC 14598. This division covers the mathematical definitions and guidance for practical measurements of internal quality, external quality and quality in use. It will include the definitions for the measurement primitives and the Evaluation Module to support the documentation of measurements. 48 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization Quality Measures Division (ISO/IEC 2502 n) Ø Measurement reference model and guide Ø Measurement primitives Ø Measures for internal quality Ø Measures for external quality Ø Measures for quality in use 49 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization Quality Evaluation Division (ISO/IEC 2504 n) ISO/IEC - 25040 contains the standards for providing requirements, recommendations and guidelines for software product evaluation, whether performed by evaluators, acquirers or developers: Ø Ø Ø Quality evaluation overview and guide Process for developers Process for acquirers Process for evaluators Documentation for the evaluation module 50 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization ISO/IEC 2501 n (Quality Model Division) ISO/IEC 2501 n is composed of the ISO/IEC 9126 -1 standard, which provides a Quality Model for software product. At the present time, this division contains only one standard: 25010 – Quality Model and guide. This is an ongoing standard in development. Quality Model Division does not prescribe specific quality requirements for software, but rather defines a generic quality model, which can be applied to every kind of software. 51 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization ISO/IEC 2501 n (Quality Model Division) Characteristics and Sub-Characteristics in SQua. RE project The ISO/IEC 25010 defines a quality model that comprises six characteristics and 27 sub-characteristics: 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br) Characteristics Sub-Characteristics Functionality Suitability Accuracy Interoperability Security Functionality Compliance Reliability Maturity Fault Tolerance Recoverability Reliability Compliance Usability Understandability Learnability Operability Attractiveness Usability Compliance Efficiency Time Behavior Resource Utilization Efficiency Compliance Maintainability Analyzability Changeability Stability Testability Maintainability Compliance Portability Adaptability Installability Replaceability Coexistence Portability Compliance 52

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization Quality in Use characteristics and are modeled with four characteristics: • effectiveness, • productivity, • security and • satisfaction The main drawback of the ISO/IEC 25010, is that they provide very generic quality models and guidelines, which are very difficult to apply to specific domains such as embedded components and CBSD. 53 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization ISO/IEC 2504 n (Quality Evaluation Division) The ISO/IEC 2502 n is composed of the ISO/IEC 14598 standard, which provides a generic model of an evaluation process, supported by the quality measurements from ISO/IEC 9126. This process is specified in four major sets of activities for an evaluation: 54 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization ISO/IEC 2504 n (Quality Evaluation Division) The ISO/IEC 2504 n is divided in five standards: • ISO/IEC 25040 – Evaluation reference model and guide; • ISO/IEC 25041 – Evaluation modules; • ISO/IEC 25042 – Evaluation process for developers; • ISO/IEC 25043 – Evaluation process for acquirers; and • ISO/IEC 25044 – Evaluation process for evaluators. 55 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization ISO/IEC 2502 n (Quality Measurement Division) The ISO/IEC 2502 n - 2007 improve the quality measurements provided by ISO/IEC 9126 -2/3/4 (external metrics), (internal metrics) and (quality in use metrics) The most significantly is the adoption of the Goal-Question- Metrics (GQM) paradigm (Basili et al. , 1994), thus, the metrics definition becomes more flexible and adaptable to the software product evaluation context. 56 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Standardization ISO/IEC 2502 n (Quality Measurement Division) The ISO/IEC 2502 n is divided in five standards: • ISO/IEC 25020 - Measurement reference model and guide; • ISO/IEC 25021 – Measurement primitives; • ISO/IEC 25022 – Measurement of internal quality; • ISO/IEC 25023 – Measurement of external quality; and • ISO/IEC 25024 – Measurement of quality in use. These standards contain some examples in how to define metrics for different kinds of perspectives, such as internal, external and quality in use. 57 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Certification “certification, in general, is the process of verifying a property value associated with something, and providing a certificate to be used as proof of validity”. (Stafford et al. , 2001) “Third-party certification is a method to ensure that software components conform to well-defined standards; based on this certification, trusted assemblies of components can be constructed. ” (Councill, 2001) 58 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Certification Third party certification is often viewed as a good way of bringing trust in software components. Components can be obtained from existing systems through reengineering, designed and built from scratch, or purchased. After that, the components are certified, in order to achieve some trust level, and stored into a repository system 59 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 3 – Embedded Software Component Quality and Certification: Certification The CBSE community is still far from reaching a consensus: • how it should be carried out, • what are its requirements and • who should perform it. Some difficulties, was found due to the relative novelty of this area (Goulao et al. , 2002 a). 60 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4 - Embedded software Component Quality Verification Framework In a survey of the state-of-the-art was noted that there is a lack of processes, methods, techniques and tools available for evaluating component quality, specifically for embedded is much more scarce. This necessity is pointed out by different researchers (Voas, 1998), (Morris et al. , 2001), (Wallnau, 2003), (Alvaro et al. , 2005), (Bass et al. , 2003), (Softex, 2007) and (Lucrédio et al. , 2007). Most researchers agree that component quality is an essential aspect of the CBSE adoption and software reuse success. 61 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4 - Embedded software Component Quality Verification Framework Its idea is to improve the lack of consistency between the available standards for software product quality (ISO/IEC 9126), (ISO/IEC 14598), (ISO/IEC 25000), also including the software component quality context and extend it to the embedded domain. These standards provide a high-level definition of characteristics and metrics for software products but do not provide ways to be used in an effective way, becoming very difficult to apply them without acquiring more knowledge from supplementary sources. 62 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 1 - Embedded software Component Quality Verification Framework: Overview robust framework for software reuse context 63 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 1 - Embedded software Component Quality Verification Framework: Overview The framework will allow that the embedded components produced in a Software Reuse Environment are certified before being stored in a Repository System. The Embedded Software Component Quality Verification Framework is composed of four modules: • • an Embedded software component Quality Model, a Maturity Level Evaluation Techniques, a Metrics Approach, and a Component Certification Process. 64 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 1 - Embedded software Component Quality Verification Framework: Overview The framework cover two perspectives of the three considered in SQua. RE project : acquirers and evaluators. acquirer’s perspectives is used to define which component best fits the customer’s needs and application/domain context. evaluator’s perspectives should be considered for evaluation required by companies in order to achieve trust in its components. developer’s perspectives is not contemplate, because it very hard for only one developer to execute all activities, independent of his knowledge 65 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 2 - Embedded software component Quality Model (EQM) ü The evaluation occurs through models that measure quality ü These models describe and organize the quality characteristics that will be considered during the evaluation ü To measure the quality it is necessary to develop a Quality Model ü The EQM proposed is based on SQua. RE project (ISO/IEC 25000, 2005), with adaptations for components and in embedded domain 66 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 2 - Embedded software component Quality Model (EQM) Some definitions: Quality characteristic is a set of properties by which its quality can be described and evaluated, and refined into sub-characteristics. Attribute is a quality property to which a metric can be assigned. Metric is a procedure for examining a component. Quality model is the set of characteristics and sub-characteristics, that provide the basis for specifying quality requirements and for evaluating quality (Bertoa et al. , 2002). 67 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 2 - Embedded software component Quality Model (EQM) Identifying important quality characteristics, classified in different criteria: i. Local or Global characteristics a. individual components (local characteristics ) b. software architecture level (global characteristics). ii. Moment in which it can be measured (Preiss et al. , 2001): a. characteristics at runtime (e. g. Performance) b. characteristics at cycle-life (e. g. Maintainability). iii. Application Metrics a. internal metrics (white-box) b. external metrics (black-box) iv. Marketing characteristics 68 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 2 - Embedded software component Quality Model (EQM) The EQM follow the ISO/IEC 25010, some changes were made to adequate for software components in embedded context. The characteristics : • Relevant were maintained; • Not interesting was eliminated; • The name was changed to adequate it to new context; • New important characteristics was added 25/05/2021 Characteristics Sub-Characteristics Run-time Sub-Characteristics Life cycle Functionality Real-time Accuracy Security Suitability Interoperability Compliance Self-contained Reliability Recoverability Fault Tolerance Safety Maturity Usability Configurability Understandability Learnability Operability Efficiency Time Behavior Resource behavior Scalability Energy consumption Memory utilization Maintainability Analyzability Stability Changeability Testability Portability Deployability Replaceability Flexibility Reusability Marketability Fernando Carvalho (ffc@cin. ufpe. br) Development time Compatibles architectures Cost Time to market Targeted market Affordability Licensing 69

A Embedded software component quality verification framework 4. 2 - Embedded software component Quality Model (EQM) The use of attributes and metrics is used to determine whether a component fulfills in the characteristics and sub-characteristics. The EQM consists of four elements: • Characteristics, • Sub-characteristics, • Attributes and • Metrics. A quality characteristic is a set of properties through which its quality can be described and evaluated An attribute is a measurable physical or abstract property of an entity. A metric defines the measurement method and the measurement scale. 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br) 70

A Embedded software component quality verification framework 4. 2 - Embedded software component Quality Model (EQM) - Quality Attributes are observable at runtime and life-cycle. The table groups the attributes by characteristics and subcharacteristics, and indicates the metrics used for evaluating each attribute. Characteristics Sub. Characteristics (Runtime) Sub. Characteristics (Life-cycle) 1. Response time (Latency) a. Throughput (“out”) b. Processing Capacity (“in”) Real-time 1. Execution time 1. Worst case execution time 1. Dead line Functionality 1. Correctness Accuracy 1. Data Encryption 1. Controllability Security 1. Auditability Compliance Self-contained 25/05/2021 Attributes Fernando Carvalho (ffc@cin. ufpe. br) 1. Standardization 1. Certification 1. Dependability 71

A Embedded software component quality verification framework 4. 2 - Embedded software component Quality Model (EQM) - Quality Attributes Characteristics Sub. Characteristics (Runtime) Sub. Characteristics (Life-cycle) Recoverability Reliability Fault Tolerance Efficiency 1. Error Handling 1. Mechanism availability 1. Mechanism efficiency 1. Environment analyze Safety Usability Attributes 1. Integrity Configurability 1. Effort to configure 1. Understandability Resource behavior 1. peripheral utilization Energy consumption 1. Mechanism availability Data Memory utilization 1. Mechanism availability Program utilization 1. Mechanism availability Memory 72 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 2 - Embedded software component Quality Model (EQM) - Quality Attributes Characteristics Sub. Characteristics (Runtime) Sub. Characteristics (Life-cycle) Attributes 1. Modifiability Stability 1. Extensibility Changeability 1. Customizability 1. Modularity 1. Test suite provided Maintainability Testability 1. Extensive component test cases 1. Component tests in a specific environment 1. Proofs the components tests 1. Complexity level Deployability Replaceability Flexibility 1. Mobility 1. Configuration capacity 1. Domain abstraction level Portability 1. Architecture compatibility Reusability 25/05/2021 1. Backward Compatibility 1. Modularity 1. Cohesion 1. Coupling 1. Simplicity

A Embedded software component quality verification framework 4. 2 - Embedded software component Quality Model (EQM) - Quality in Use characteristics The model is complemented with Quality in Use characteristics (ISO/IEC 25000, 2005) are composed of: • • Productivity, Satisfaction, Security, and Effectiveness. • Bring relevant information for new customers, • This is the user’s view of the component, • Obtained when the component in an execution environment, and • Analyze the results according to their expectations. Quality in Use characteristics are useful to show the component’s behavior in different environments, it is measured through the customer’s feedback 74 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 2 - Embedded software component Quality Model (EQM) - Additional Information Relevant Component Information The Additional Information characteristics complement the model and are composed of: Ø Technical Information is important for developers to analyze the actual state of the component , Ø Organization Information is important to know who is the responsible for that component. Additional Information 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br) Technical Information Component Version Programming Language Patterns Usage Architecture compatible Program Memory used Technical Support Organization Information CMMi Level 75 Organization’s Reputation

A Embedded software component quality verification framework 4. 3 - Maturity Level Evaluation Techniques The quality characteristics proposed not need to be evaluated with the same degree of details and depth for all types of application. (E. g. evaluation of a component used in railway system and game). Different evaluation levels must be used in order to provide degree of confidence for different domains and risk-levels. 76 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 3 - Maturity Level Evaluation Techniques Embedded software component Maturity Model (EMM) The Details of an evaluation is a reflex of the evaluation techniques used. So, an Embedded software component Maturity Model (EMM) was defined. It is based on CMMI (CMMI, 2000) and model for general propose component (Alvaro et al. , 2007 a). 77 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 3 - Maturity Level Evaluation Techniques The EMM is constituted of five hierarchical levels of quality characteristics where the components can be evaluated in different the depth of the evaluation gives different degrees of confidence. Each company/customer decides which level is better for evaluating its components, analyzing the cost/benefits of each level. The evaluation levels can be chosen independently for each characteristic (e. g. functionality → EMM I, reliability → EMM III, usability → EMM IV). 78 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 3 - Maturity Level Evaluation Techniques Guidelines for selecting evaluation level Level Environment Safety/Security Economic Domain EMM I No damage Few material damage; No specific risk Negligible economic loss Entertainment, EMM II Small/Medium damage properly Few people disabled Few economic loss household EMM III Damage properly Large number of people disabled Significant economic loss Security, Control systems EMM IV Recoverable environment damage Threat to human lives Large economic gross Medical, Financial EMM V Unrecoverable environmental damage Many people killed Financial disaster Transportation, Nuclear systems 79 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 3 - Maturity Level Evaluation Techniques The levels and the evaluation techniques selection of EMM must be appropriated to evaluate the quality attributes proposed on the EQM A relation EQM - Quality Attributes X Evaluation Technique EMM is necessary. The objectve is not to propose a large amount of isolated techniques, but to propose a set of techniques that are essential for measuring each quality attribute, complementing each other and, thus, becoming useful to compose the Maturity Level Evaluation Techniques. 80 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 3 - Maturity Level Evaluation Techniques Characteristi cs EMM III EMM IV EMM V Time constraint analysis Requirements and Documentation Analysis Accuracy analysis Evaluation measurement (Time analysis) Functional Testing (black box), Unit Test, Regression Test (if possible) System Test Documents Inspection (check list) Code Inspection Functional Tests (whitebox) with coverage criteria and code inspection Formal Proof Dependability analysis Suitability analysis Programming Language Facilities (Best Practices) Error Manipulation analysis Fault tolerance analysis Error Injection analysis Error recover Reliability growth model Formal Proof Usability Effort to Configure analysis Documentation analysis (Use Guide, architectural analysis, etc) Interfaces inspection provided and required Code and component’s interface inspection correctness and completeness) Analysis of the pre and post-conditions of the component User mental model Efficiency Constraint analyses Accuracy analysis Evaluation measurement (memory, power and resource) Memory Analysis Power consumption Analysis Resource Analysis Tests of performance(memory, power and resource) Algorithmic complexity Performance optimization (memory, power and resource) Performance profiling analysis Formal Proof Customizability analysis Extensibility analysis Inspection of Documents Analysis of the provided test suite (if exists) Code metrics and programming rules Static Analysis of the component development process Traceability evaluation Component Test Formal Proof Component execution in specific environment and architectural analysis Cohesion, Coupling, Modularity and Simplicity analyses Cohesion of the documentation with the source code analysis Deployment analysis Backward compatibility Mobility analysis Configurable analysis Hardware/Software analysis Conformity to programming rules Environment and architectural constraints evaluation Domain abstraction analysis Analysis of the component’s architecture Functionality Reliability Maintainability Portability 81 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 3 - Maturity Level Evaluation Techniques Characteristic Functional ity Sub. Characteristics Real-Time Quality Attributes Evaluation Techniques Response time (Latency) a. Throughput (“out”) b. Processing Capacity (“in”) • Evaluation measurement (Time analysis) • Time constraint analysis • Formal Proof Execution time • Evaluation measurement Worst case execution time • Evaluation measurement • System Test Dead line • Evaluation measurement • System Test Accuracy Correctness • Requirements and Documentation Analysis • Accuracy analysis • Functional Testing (black box), Unit Test, Regression Test (if possible) • Functional Tests (white-box) with coverage criteria Security Data Encryption • System Test • Code Inspection Controllability • System Test • Code Inspection Auditability • System Test • Code Inspection Standardization • Inspection of Documents Certification • Inspection of Documents Dependability • Documents Inspection • Code Inspection Compliance Self-contained 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br) 82

A Embedded software component quality verification framework 4. 3 - Maturity Level Evaluation Techniques Characteristic Reliabil ity Sub. Characteristics Quality Attributes Evaluation Techniques Recoverability Error Handling • Programming Language Facilities (Best Practices) • Error Manipulation analysis • Error Injection analysis • Error recover • Reliability growth model • Formal Proof Fault Tolerance Mechanism available • Suitability analysis • Dependability analysis Mechanism efficiency • Error injection analysis • Programming Language Facilities (Best Practices) • Fault tolerance analysis • Reliability growth model • Formal Proof Environment analyze • Dependability analysis • Environment analyses • System analyses Integrity • System analyses Safety 83 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 3 - Maturity Level Evaluation Techniques Characteristic Usability Sub. Characteristics Configurability Quality Attributes Effort for configure Evaluation Techniques • Effort to Configure analysis • Interfaces inspection provided and required • Code and component’s interface inspection correctness and completeness) • Analysis of the pre and postconditions of the component • User mental model Understandability Documentation analysis (Use Guide, architectural analysis, etc) 84 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 3 - Maturity Level Evaluation Techniques Characteristic Efficiency Sub. Characteristics Quality Attributes Evaluation Techniques Resource Behavior peripheral utilization • Constraint analyses • Evaluation measurement • Resource Analysis • Tests of performance • Performance optimization • Performance profiling analysis • Formal Proof Energy consumption Mechanism available • Constraint analyses • Evaluation measurement • Power consumption analysis • Tests of performance • Performance optimization • Performance profiling analysis • Formal Proof Data Memory Utilization Mechanism available • Constraint analyses • Evaluation measurement • Memory analysis • Tests of performance • Performance optimization • Performance profiling analysis • Formal Proof Program Memory Utilization Mechanism available • Evaluation measurement • Constraint analyses 85 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 3 - Maturity Level Evaluation Techniques Characteristic Maintain ability Sub. Characteristics Quality Attributes Evaluation Techniques Stability Modifiability • Code metrics and programming rules • Inspection of Documents • Static Analysis Changeability Extensibility • Effort for operating • Extensibility analysis Customizability • Customizability analysis Modularity • Code metrics and programming rule Test suit provided • Analysis of the test-suite provided (if exists) Extensive component test cases • Analysis of the component development process Component tests in a specific environment • Traceability evaluation Proofs the components test • Component Test Formal Proof Testability 86 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 3 - Maturity Level Evaluation Techniques Characteristic Portability Sub. Characteristics Quality Attributes Evaluation Techniques Deployability Complexity level • Component execution in specific environments and architectural analysis • Deployment analyses • Environment and architectural constraints evaluation Replaceability Backward Compatibility • Backward compatibility analysis Flexility Mobility • Mobility analyses Configuration capacity • Configuration analyses Domain abstraction level • Cohesion of the documentation with the source code analysis • Domain abstraction analysis Architecture compatibility • Conformity to programming rules • Analysis of the component’s architecture • Hardware/Software analysis Modularity • Modularity analyses Cohesion • Cohesion analyses Coupling • Coupling analyses Simplicity • Simplicity analyses Reusability 87 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 4 - Metrics Approach As with any engineering, software development requires a measurement mechanism for feedback, evaluation, creating a corporate memory and improve of software process (Basili et al. , 1994). Measurement Objectives: • to assess a project progress, • to take corrective action based on this assessment, and • to evaluate the impact of such action Helps of Measurement • Helps support project planning • Allow to determine the strengths and weaknesses of processes/products • Provides a rationale for adopting/refining techniques • Allows to evaluate the quality of specific processes/products 88 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 4 - Metrics Approach There a variety of mechanisms for measurable goals in the literature: • The Quality Function Deployment approach (Kogure & Akao, 1983), • The Goal Question Metric approach (Basili et al. , 1994), (Basili, 1992), (Basili & Rombach, 1988), (Basili & Selby, 1984), (Basili & Weiss, 1984), and • The Software Quality Metrics approach (Boehm et al. , 1976), (Mc. Call et al. , 1977). In this Framework the Goal-Question-Metric (GQM) approach was adopted, which was the same technique proposed to use in ISO/IEC 25000 looking for track the software product properties. 89 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 4 - Metrics Approach The Goal Question Metric (GQM) approach is based upon the assumption that for to measure in a purposeful way it must : 1. Specify the goals for itself and its projects, 2. Must trace those goals to the data that are intended to define those goals operationally, and 3. Provide a framework for interpreting the data with respect to the goals. 90 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 4 - Metrics Approach The resulting measurement model has three levels: 1. Conceptual level - GOAL: A goal is defined for an object(product, process, resource) 2. Operational level - QUESTION: A set of questions is used to characterize the way the assessment/achievement of a specific goal 3. Quantitative level - METRIC: A set of data is associated with every question in order to answer it in a quantitative way (Objective/Subjective) Summarized, A GQM model is starting with a goal. The goal is refined into several questions. Each question is then refined into metrics, some of them objective, some of them subjective. 91 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 4 - Metrics Approach The GQM approach allow metrics to track the properties of: i. Quality attributes of EQM, ii. Evaluation techniques of EMM, and iii. Certification process Objective Metric (i)Metrics to track the EQM Properties Usability Subjective Sub-Characteristic Configurability Quality Attribute Effort to Configure Goal Evaluates the time necessary to configure the component. Question How much time is needed to configure the component in order to work correctly in a system? Metric Time spent to configure correctly Interpretation The faster it is to configure the component the better, but it depends of the component and environment complexity. 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br) Functionality Sub. Characteristic Accuracy Quality Attribute Correctness Goal Evaluates the percentage of the results that were obtained with precision Question Based on the amount of tests executed, how much test results return with precision? Metric Precision on results / Amount of tests Interpretation 0 <= x <= 1; which closer to 1 92 is better

A Embedded software component quality verification framework 4. 4 - Metrics Approach 1. (ii) Metrics to track the EMM Properties Portability Each technique can be measured in different ways and complexity, using different tools, techniques, methods and processes. Quality Attribute Coupling, Cohesion, Simplicity, Reusability and Modularity analyzes EMM level I Technique Coupling, Cohesion, Simplicity, Reusability, Modularity analyzes using Checkstyle tool[2] Goal Evaluates the internal source code of the component Question Is the Checkstyle tool efficient enough to measure those attributes? Metric Analysis of the results and coverage of the Tool Interpretation If the tool can mine these kinds of information from the source code and present them to be analyzed, it is good to evaluate the component. On the other hand, if it is not enough to evaluate some kind of attributes, other tool should be use to complement or to substitute this one. If this tool is good to evaluate the component, an analysis of the metrics collected in the tool can be used to define those attributes from the component. The idea is that the component should have: less coupling, high cohesion, high modularity, ways to perform the function in a simple way 93 and high reusability. [2] Checkstyle – http: //checkstyle. sourceforge. net 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 4 - Metrics Approach (iii) Metrics to track the Certification Process Component Certification Process Properties The idea is to obtain feedback 1. from those metrics in order to improve the activities and steps to assure the efficiency and efficacy of the process The evaluation team should define metrics as much as they think interesting in order to measure the process capacity Goal Adequately evaluate the software component embedded Question Could the evaluation team evaluate everything they planned to execute using the documents developed during the process activities? Metric Total documented functionalities / Total component functionalities (or Total measurement accomplished) Interpretation 0 <= x <= 1; which closer to 1 is better Component Certification Process Goal Analyze the usability of the templates provided Question Has the template helped during the certification development? Metric Evaluation team feedback Interpretation If the template helped during the certification development it is good, if not it should be adapted to improve the time of the component certification process. 94 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 5 - Component Certification Process 1. The Component Certification Process is the default process adopted for embedded component certification in robust framework for software reuse (Almeida et al. , 2004). It define a set of activities in order to guide the evaluation team during the component evaluation, and It could be repeatable and reproducible, each activity contains a well -detailed description: ü inputs and outputs, ü mechanisms to execute, and ü to control. 95 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 5 - Component Certification Process 1. The process is based on SQua. RE project, which provides guidance and requirements for the software product evaluation. A set of works from literature which includes processes for software product evaluation and processes for software component assessment aid during the definition of this process (Mc. Call et al. , 1977) , (Boegh et al. , 1993), (Beus-Dukic et al. , 2003), (Comella. Dorda et al. , 2002) 96 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 5 - Component Certification Process 1. A set of activities to guide the evaluation team during the evaluation were proposed, which is presented using SADT notation (Ross, 1997). 97 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 5 - Component Certification Process 1. 4. 5. 1 - Establish Evaluation Requirements activity 98 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 5 - Component Certification Process 1. 4. 5. 2 - Specify the Evaluation activity 99 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 5 - Component Certification Process 1. 4. 5. 3 - Design the Evaluation 100 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 5 - Component Certification Process 1. 4. 5. 4 - Execute the Evaluation 101 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 5 - Component Certification Process 1. The evaluation team can describe a set of suggestions in order to improve the quality of the component. 1. This information is important in the case that the component is approved and, much more interesting if the component is rejected. The evaluation team is the main responsible for executing this process and should be carefully defined in order to assure that the certification will be efficiently developed. 102 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 5 - Conclusion 1. The demands that industries in embedded system includes: 2. low production costs, addressed by CBD approach 3. short time to market, and 4. high quality. 5. 6. Assessment and evaluation of software components has become a compulsory and crucial part of any CBSD Assessment and evaluation of software components lifecycle. 7. 1. To properly enable the evaluation of embedded software components, supplying the real necessities of the embedded system design of building system fast, cheap and high quality systems, an Embedded software Component Quality Verification Framework is mandatory. 103 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 6 – Experimental Study An experimental process can be divided into the following main activities (Wohlin et al. , 2000): Definition (problem, objective and goals), Planning (design, instrumentation and threats), Operation (measurements are collected) , Analysis and Interpretation (data are analyzed and evaluated), Presented and Packaged (results are presented and packged) Definition, Planning activities will be presented. The complete experiment study will be accomplished and described in next year. 104 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 6 – Experimental Study - Definition According to GQM Paradigm(Basili et al. , 1986), the main objective of experimental study is to: Analyze the capacity to evaluate the quality of embedded software component for the propose of evaluating embedded software component quality verification framework with respect to the efficiency of the framework from the point of view of the researchers, software and quality engineers (customers, evaluators) in the context of the embedded software component quality area. 105 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 6 – Experimental Study - Planning Context – To evaluate the efficiency of framework based a set of embedded component (project between industry and university). Training - The training of the subjects using the process will be conducted in a classroom at the university Pilot Project - Before the study, a pilot project will be conducted, aiming to detect problems and improve the planned material. Selection of Subjects - Ten students of pos-graduation at UFPE were selected by convenience sampling. Subjects - According to its skills and technical knowledge to evaluate the embedded software components. Instrumentation - the subjects will receive a questionnaires about his/her education, experience and satisfaction using the framework, and chapters 4 of this Thesis. 106 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 6 – Experimental Study - Planning Criteria - evaluate quantitatively the real efficiency of the framework and the difficulty of the framework usage. (i) coverage of the EQM, (ii) EMM; and (iii) subjects difficulty to use the framework). Hypothesis is to know and to formally state what is going to be evaluated - Null hypotheses, H 0: determine that the framework is not efficient and it is very difficulty to use: Ho’: coverage of the component quality attributes proposed in the CQM X the quality attributes used during the component evaluation < 90% Ho’’: coverage of the evaluation techniques proposed on the SCMM for the quality attributes defined during the component evaluation < 90% Ho’’’: %Subjects that had difficulty to understand, follow and use the Software Component Quality Framework > 20% - Alternative hypotheses: determine that the framework is efficient and it is easy to use: Ho’: coverage of the component quality attributes proposed in the CQM X the quality attributes used during the component evaluation >= 90% Ho’’: coverage of the evaluation techniques proposed on the SCMM for the quality attributes defined during the component evaluation >= 90% Ho’’’: %Subjects that had difficulty to understand, follow and use the Software Component Quality Framework <=107 20% 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 6 – Experimental Study - Planning Independent Variables -the education and the experience of the subjects, Dependent Variable - the quality of the EQM and EMM developed and the usability of the framework Qualitative Analysis - aims to evaluate the difficulty of the application of the framework and the quality of the material used in the study Randomization - This technique can be used in the selection of the subjects, however, the subjects were selected by convenience sampling Blocking - not identified the necessity of dividing the subjects into blocks Balancing - it is not necessary to divide the subjects, since there is only one group Internal Validity - this study is supposed to have at least between four to eight subjects to guarantee a good internal validity. 108 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 4. 6 – Experimental Study - Planning External Validity - is considered sufficient, since it aims to evaluate the viability of the application of proposed framework. Construct Validity - A relative well-know project will be used (i. e. the experimenter have about three years of experience). This choice avoids previous experience of making a wrong interpretation of the impact of the proposed framework. Validity of the Conclusion of the Study - This conclusion will be drawn by the use of descriptive statistic. The project used in the experimental study was the embedded software components used in cars tracking system developed in a partnership between industry, academy and a research institute. 109 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 5 - Conclusion 1. Contributions 1. To demonstrate that component certification is: 2. Possible, 3. Practically viable, and 4. Applicable in the embedded system industry. üTo show the challenge and specific requirements of the use CBD approach for embedded systems in different domain; üthe realization of a survey related to the state-of-the-art in embedded software component quality and certification research; üTo enable the quality component verification and certification for a Robust Software Framework Reuse (Almeida et al. 2004), and ü To Assess and To evaluate quality of embedded software components 110 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 5 - Conclusion 1. To demonstrate that component certification is: 2. Possible, 3. Practically viable, and 4. Applicable in the embedded system industry. üTo show the challenge and specific requirements of the use CBD approach for embedded systems in different domain; üthe realization of a survey related to the state-of-the-art in embedded software component quality and certification research; üTo enable the quality component verification and certification for a Robust Software Framework Reuse (Almeida et al. 2004), and ü To Assess and To evaluate quality of embedded software components 111 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 5 - Future works 1. During the research it was perceived that there are needs for a number of improvements. Some of them are the following: ü Component Certification Center - component certification standard for Software Factories ü Tool Support - it is needed a tool support in order to aid the usage of the proposed processes, methods, techniques, etc ü Risk and Cost/Benefit Management Model the risk and cost/benefit Model analyze if the costs are acceptable and viable or not. 112 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework 5 - Chronogram 1. The research work to perform up until the defense of thesis can be divided into five distinct activity Activity Deadline Deliverab le Suggest and review metrics for remaining quality attributes Jan 16 - Jan Define a document/form model for register embedded component evaluation 30 - Jan Finalization of case of study for proposed framework evaluation 13 - Feb Completion of doctoral thesis 1 - Mar Doctoral thesis defense 27 - Mar 113 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)

A Embedded software component quality verification framework Thank you ! Questions ? • Fernando F. de Carvalho o ffc@cin. ufpe. br 114 25/05/2021 Fernando Carvalho (ffc@cin. ufpe. br)