WHAT DO WE KNOW ABOUT ALIGNMENT OF REQUIREMENTS

WHAT DO WE KNOW ABOUT ALIGNMENT OF REQUIREMENTS ENGINEERING AND SOFTWARE TESTING Results of a Systematic Mapping Study Pertti Karhapää Alireza Haghighatkhah Markku Oivo

Introduction • Software development Analysis – From initial conceptualization to realization • Important to link different activates • Linking RE and ST important Implementation Design Testing – Requirements • Dictate what a software should do – Testing • Show that it does what it is supposed to do Requirements • Main research question – What is the current state of research on alignment of RE and ST? Testing

Background • Waterfall – RE and ST detached • V-Model – RE and ST connected – Intra organizational distances (Bjarnason et al. 2015) • Agile – No explicit focus on alignment • Flow of information – Linking people – Linking artefacts Royce (1970) Pfleeger & Atlee (2009) “The adjustment of RE and ST efforts for coordinated functioning and optimized product development” Unterkalmsteiner et al. (2014) • Taxonomy and REST-bench • Previous mapping study (Barmi et al. 2011) Unterkalmsteiner et al. (2014)

Research method (1/2) • Systematic Mapping Study Petersen et al. (2015) Plan • • Scope and research questions Search strategy Inclusion / Exclusion criteria Data extraction Conduct Report • Research questions – RQ 1 – What is the intensity of research on alignment of RE and ST? – RQ 2 – What is the foci of research regarding alignment of RE and ST? – RQ 3 – In what context has alignment of RE and ST been applied and what effects in terms of benefits and challenges have been reported? – RQ 4 – What are the most frequently applied research types and methods, and what kind of contributions are provided by the studies? – RQ 5 – What is the scientific rigor and the industrial relevance of the research on alignment of RE and ST?

Research method (2/2) • Search strategy 7228 711 – Database search – Snowballing (Wohlin, 2014) • Backward and forward – Manual search • RET´ 14 and RET’ 15 Unrelated type 2306 Snowballing and Manual search 30 Screening 4868 • Inclusion/Exclusion criteria – Screening – Snowballing and manual search – Full text review • Data extraction Full text review 4 80 2693 1849 1518 464

Findings (1/10) intensity of research • Intensity of research – Conference publications • 44 publications • 36 venues – Journal publications • 22 publications • 14 journals – Workshops • 14 proceedings • 6 workshops

Findings (2/10) methodological data • Methodological data – Research types (Wieringa et al. 2005) • 38 Evaluation research • 29 Solution proposals – Research method • 37 Not reported • 34 Case study – Contribution type (Shaw, 2003) • 59 Framework/Method/Technique • 16 Tool • 13 Lessons learned

Findings (3/10) domains, benefits, challenges • Domains Domain # of studies # of evaluation research Automotive Industry 15 10 Telecommunication and Mobile Devices 11 7 Avionics 6 4 – Greater benefit of alignment of RE and ST • Large development organizations • Complex systems

Findings (4/10) domains, benefits, challenges • Benefits and challenges Challenge category (62 individual challenges) Benefit category (16 individual challenges) Number of benefits Benefits for project quality 4 Benefits for RE 3 Benefits for testing 6 Benefits for product quality 3 Sabaliauskaite et al. (2010) Number of chal. Organization and process related challenges 11 People related challenges 3 Tool related challenges 7 RE related challenges 14 ST related challenges 9 Change management related challenges 6 Traceability related challenges 8 Measurement related challenges 4

Findings (5/10) focus of research • Focus of research Group # V&V in RE 12 Test generation from requirements specification 25 Improved traceability 12 Improved testing 3 Formalization of the requirements 10 Practices to support alignment 5 Assessment of alignment 2 Other approaches 5

Findings (6/10) focus of research • Formalization of requirements – Important in safety critical • Allows formal verification – Synthesize formal models from informal requirements – Create requirements models – Facilitate testing • V&V in RE – Important to test as soon as there is something to test • Late V&V – Need to consider full set of requirements – Finding faults in requirements and improving quality of requirements – Testing requirements models

Findings (7/10) focus of research • Test generation from requirements specification – – Improves traceability and test coverage Facilitates automation Reduce cost of testing MBT • Test models, test cases, and even implementation tests from requirements models – Test cases from use cases – Goal orientation • Not just for context analysis

Findings (8/10) focus of research • Improved traceability – – Understand change impact Full traceability required in safety critical domains MDE, AOSD, BDD After-the-fact traceability • IR to build traceability matrix – Traceability through separation of RE and ST • Minimalistic interface between RE and ST • Changes propagate through changes in interface • Improved testing – Annotating source code with goals • Events and paths emitted when code is running – Improve information gathering

Findings (9/10) focus of research • Practices to support alignment (way of working) – Linking of people • “Human and organizational sides of software development are at the core of industrial alignment” Graham(2002) – Customer defined acceptance tests – Organization size one variation factor for cooperation • Assessment – REST-bench framework – Assess alignment – Elicit improvement opportunities Practice Early tester participation Tester participation in requirements review Linking tester with requirements owners Requirements suggestions made by testers

Findings (10/10) rigor and relevance • Rigor and relevance (Ivarsson & Gorschek, 2011) – 35 Low relevance/Low rigor • Solution proposals • No research method – 25 High Relevance/Low rigor • Mostly case studies in industrial context – 13 High relevance/high rigor • Mostly case studies in industrial context

Conclusions • Summary of findings – – – Increasing interest towards the topic. Many venues It is known what is needed to support alignment The challenges are understood, reported benefits are scares Techniques evaluated in industry, not clear what works in what context and why More rigorous research needed to understand what works and why in what context • Implications – Up to date overview of existing research • For practitioners to understand approaches, benefits, and challenges • For researchers as basis for further research • Future directions – More rigorous research needed, what works in what context and why – Analysis of benefits – Metrics of alignment

References Royce, W. W. (1970, August). Managing the development of large software systems. Proceedings of IEEE WESCON (Vol. 26, No. 8, (pp. 328338). Pfleeger, S. L. , & Atlee, J. M. (2009). Student Study Guide for Software Engineering: Theory and Practice. Unterkalmsteiner, M. , Feldt, R. , & Gorschek, T. (2014). A taxonomy for requirements engineering and software test alignment. Acm Transactions on Software Engineering and Methodology, 23(2), 16. doi: 10. 1145/2523088 Bjarnason, E. , Runeson, P. , Borg, M. , Unterkalmsteiner, M. , Engström, E. , Regnell, B. , . . . & Feldt, R. (2014). Challenges and practices in aligning requirements with verification and validation: a case study of six companies. Empirical Software Engineering, 19(6), (pp. 1809 -1855) Barmi Z. A. , Ebrahim A. H. and Feldt R. , "Alignment of requirements specification and testing: A systematic mapping study, " in Software Testing, Verification and Validation Workshops (ICSTW), 2011 IEEE Fourth International Conference on, 2011. Petersen, K. , Vakkalanka, S. , & Kuzniarz, L. (2015). Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology, 64, (pp. 1 -18). C. Wohlin, "Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering, " in 18 th International Conference on Evaluation and Assessment in Software Engineering (EASE 2014), 2014. Wieringa, R. , Maiden, N. , Mead, N. , & Rolland, C. (2006). Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requirements Engineering, 11(1), (pp. 102 -107). Shaw, M. (2003, May). Writing good software engineering research papers: minitutorial. Proceedings of the 25 th international conference on software engineering (pp. 726 -736). Ivarsson M. and Gorschek T. , "A method for evaluating rigor and industrial relevance of technology evaluations, " Empirical Software Engineering, vol. 16, no. 3, pp. 365 -395, 2011. Graham, D. (2002). Requirements and testing: Seven missing-link myths. IEEE Software, 19(5), (pp. 15 -17). doi: 10. 1109/MS. 2002. 1032845 Uusitalo, E. J. , Komssi, M. , Kauppinen, M. , & Davis, A. M. (2008, September). Linking requirements and testing in practice. International Requirements Engineering, 2008. RE'08. 16 th IEEE (pp. 265 -270). doi: 10. 1109/RE. 2008. 30.

Tanja Sauvola University of Oulu pertti. karhapaa@oulu. fi www. oulu. fi/m 3 s