Bringing Governance into Distributed RD Software Development GANT

Bringing Governance into Distributed R&D Software Development – GÉANT Case Study Branko Marović branko. marovic@rcub. bg. ac. rs Paweł Kędziora kedziora@man. poznan. pl Marek Lewandowski marekl@man. poznan. pl Cezary Mazurek mazurek@man. poznan. pl connect • communicate • collaborate

Motivation Software governance Efficient development and quality Complexity of Organisational structure behind development process – Independent and distributed groups of developers from NRENS Domain where the results are applied – The multi-domain GÉANT network interconnects Europe’s NRENs using combination of routed IP and switched infrastructure. Software participates in providing GÉANT services: Connectivity Services. Network Performance and Operations Services. End User Applications. Need for consistent development methodology, management procedures, tools… connect • communicate • collaborate

Software Governance in GN 3 New activity, independent from developments, directly responsible to project officers Best Practices – collecting and publishing and promoting best practices on software architecture, software development and quality assurance. Development Infrastructure – establishing and maintaining common infrastructure for development. Quality Assurance (QA) – verification and support in application of best practices and common development infrastructure by means of periodical audits. connect • communicate • collaborate

Initial Status How to consolidate established and diverse software developments toward the common ground? Baseline - survey of software developments on practices, technologies, issues and expectations on governance. Developments mostly in Java. Difference in practices and tools. Overlapping efforts, integration problems, uneven quality. Although teams declared use of agile methodologies, most introduced them inconsequently or selectively. Development infrastructure was multiplied. Ineffective usage of manpower. Inadequate access to code and documentation. Interest for guidelines on coding, testing, deployment and support: technological pointers, conventions, lifecycle connect • communicate • collaborate management.

Best Practices Guide the software developments towards a common path across individual developments and methodologies. Common sense, wide-ranging, informative guidelines. From programming and development procedures to access and use of the support infrastructure to support and project closing. Guides: Software Developer. Software Architecture Strategy. Quality Assurance. connect • communicate • collaborate

Best Practices Guides Topics Use of GÉANT development and QA infrastructure Software development process Software design Coding guidelines Version control management Internal and public documentation Code analysis using tools Code reviews Testing Building, integrating and releasing software Agile integration IPR and licensing Change management Configuration management Bug tracking Problem management Knowledge management Closing software projects … connect • communicate • collaborate

QA Audits Periodic (every 4 months) measurement of projects’ progress in terms of: Alignment with best practices. Use of common tools and provided development infrastructure. Progress tracked via metrics. Resulted in numerous recommendations to developers. More advanced or specific topics and tools related to later software lifecycle stages are addressed by: Best practices updates. Infrastructure extension. connect • communicate • collaborate

QA Audits connect • communicate • collaborate

SW Development Infrastructure Based on mainstream development tools. Shared across all distributed teams. Set of commonly used services: Authentication and authorization (Altassian Crowd). Source code management (Subversion). Code reviewing (Altassian Fisheye and Crucible). Artifacts repository for Maven. Issue tracker (Altassian JIRA). Collaborative web spaces (Altassian Confluence). Continuous integration (Jenkins) and code analysis (PMD and Checkstyle). License checking (JFrog Artifactory Power Pro). connect • communicate • collaborate

Adoption of Common Development Infrastructure Project / Tool SVN JIRA Hudson Forge Fisheye / Crucible REPO Auto. BAHN YES YES YES c. NIS YES YES YES HADES NO YES PARTIALLY YES NO NO I-SHARe YES YES YES Oppd YES YES NO PS AS YES YES YES NO PS E 2 E MON MP YES YES NO PS LS YES YES YES NO PS RRD MA YES YES YES NO PS Telnet/SSH MP YES YES YES PS SQL MA YES YES YES NO 10 11 8. 5 11 10 4 No. of tool usages 3 in Perl Proprietary HADES code, connect • communicate • collaborate special sharing of PS modules

Gradual Approach Incremental alignment, continuous detailing of initial recommendations, addition and integration of more advanced tools. Mapping of recommendations on actual use of common tools. Software governance itself is strongly aligned with the principles of agile methodologies promoted for software development. Relies on developers’ feedback and demands. Issues identified through QA. connect • communicate • collaborate

Proactive Attitude Initial results were widely applied by developers, so we extended the scope with: Extension of best practices with guidelines related to the transformation of developments into reliable products. Facilitation of knowledge and experience sharing. In-depth audits on software quality, reliability, efficiency and security. Support in appliance of the recommendations from periodical audits. Support to the Project Office with tools for verification of compliance with the IPR policy. connect • communicate • collaborate

Summer Developers Schools Developers mingle, exchange experiences, learn best programming practices and solve common problems together. Targeted tutorials. Working in small groups on prepared related tasks. Shared development infrastructure used in hands-on sessions. Presentation and analysis of results. Also contribution to two security-related trainings. connect • communicate • collaborate

Summer Developers Schools connect • communicate • collaborate

QA Persons Not all of QA recommendations were implemented. First line of support for each development team established by introducing Quality Assurance Personnel. QA persons: Advisors supporting project teams. Observe the development process and proactively investigate potential issues. Issues reported to lead developers. If some issue is not solved internally, QA persons initiate communication and knowledge exchange with other teams and available experts. connect • communicate • collaborate

In-Depth Audits Performed for two applications under development in addressing architecture, code, reliability, and efficiency; process related suggestions. Security audit for an application in migration from pilot to operational. QA persons served as liaisons between the auditors and developers. Recipients of the results agreed that such detailed and sophisticated analyses would not be possible by using internal reviewing and testing. connect • communicate • collaborate

QA Testbed Infrastructure for demos, developer testing and QA. Number of virtual machines with shared storage. Log term reservation for demonstration and testing by developers. Scheduled pool for distributed QA testing. Shortening release lifecycle by providing environments for: Development (DEV). System Integration Testing (SIT). User Acceptance Testing (UAT). Staging. Production. Stress and Volume Testing (SVT). connect • communicate • collaborate

Common Services VM Admin Management QA Testbed Infrastructure Services User Identity Service Physical Machines Virtualised View QA Testbed VMs Storage SW Development Services Group SWD VMs … Dell R 710 QA Testbed storage (incl. work space, VMs’ images & snapshots) Dell R 710 SWD storage (incl. work space, VMs’ images & snapshots) Dell R 710 Backups

Addressing IPR GÉANT services are developed by distributed teams hired by numerous NRENs. It became crucial to establish GN 3 IPR policy and manage its appliance in development. Separate IPR coordination activity. Artifactory Pro License Control – tool supporting IPR coordination and conformance. Interaction with CI and Maven. Tracking of licenses for built software and dependencies. Repository of allowed and forbidden licenses. Verification of projects’ IPR policy. connect • communicate • collaborate

Conclusions Importance and impact of the software governance in distributed software development in R&D project. Used approach goes beyond the usual methodologies and frameworks and was tailored for the characteristics of GÉANT developments. Everyday procedures (best practices). Formal processes (audits). Support resources and activities (maintenance of development infrastructure). Addressing difficult underlying areas: identification of needs, motivating developers to apply guidelines, trainings, stimulating knowledge sharing. This approach may be applied in other large scale distributed projects. connect • communicate • collaborate

Acknowledgements This work is supported by the GN 3 project, which is in part funded by the European Commission under the Seventh Framework Programme, contract number FP 7 -ICT 238875. connect • communicate • collaborate