LSST Telescope Site Software Testing Rob Bovill Telescope
LSST Telescope & Site Software Testing Rob Bovill Telescope & Site Software Test Engineer LSST 2018 Project and Community Workshop August 14, 2018 #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 1
Agenda - Software testing process - Tools • Automated/Manual • Issue tracking/Project management - Results reporting Status Future goals and plans Questions and Discussion #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 Highlight who I am and what I do 2
Agenda - Software testing process - Tools • Automated/Manual • Issue tracking/Project management - Results reporting Status Future goals and plans Questions and Discussion #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 3
Documentation - LTS-699 – Telescope and Site Software Test Process - Methods • Exploratory, Regression, Functional and Acceptance - Strategies • Manual and Automated - Execution - Management • Roles and Responsibilities - Specific System Test Plans #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 4
Testing Process - - Primary responsibility is to provide data to assess software status Start with manual walk-through • Exploratory testing • “Happy-path” – determine if software does what it is supposed to do Translate this work into test scripts • Iterative process Explore “off-nominal” interactions • Determines if software does NOT do what it is NOT supposed to do - Insufficient arguments, incorrect type, improper configuration, etc. - Run the tests repeatedly • Feedback to/from developers and customers - Bugs, Improvements #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 5
Types of Testing - Software testing teams DO NOT perform Unit testing - Exploratory - Functional • Verify functional requirements - Regression • Verify application performs the same way after a change • Ensures bugs (regressions) do not return - Acceptance • Verify software functionality in a new environment (delivery) - Integration • Testing software applications when combined into the system #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 6
Agenda - Software testing process - Tools • Automated/Manual • Issue tracking/Project management - Results reporting Status Future goals and plans Questions and Discussion #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 7
Automated Testing - Best suited for Regression testing • Quick and repeatable • Validates issues do not return - Does not test new features (until written) • Writing tests takes time - Results stored online • https: //mebsuta. lsst. org/~rbovill/ - Tests currently maintained in Git. Hub • https: //github. com/lsst-ts #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 8
Automated Testing - Tools - Robot-Framework • Open-source • Extendable with test libraries • Well supported - https: //github. com/robotframework - https: //groups. google. com/forum/? hl=en#!forum/robotframewor k-users #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 9
Continuous Integration - Jenkins - an open source continuous integration tool • https: //ts-ci. lsst. codes/ - Integrates with Robot-Framework • SAL, EFD, XML • Scheduler team is exploring build management with Jenkins • Used by Pathfinder teams to verify readiness #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 10
Manual Testing - Only performed when automated testing is not feasible • One-off applications (ILC Calibration) • Simple applications • Tests run infrequently - Results stored online • https: //atlassian. lsstcorp. org/~rbovill/ - Tests currently maintained locally • Docu. Share? Confluence? Other? #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 11
Agenda - Software testing process - Tools • Automated/Manual • Issue tracking/Project management - Results reporting Status Future goals and plans Questions and Discussion #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 12
Issue Tracking & Project Management - Atlassian JIRA • • Task management New features/improvements Bugs/issues Planning - Agile development process - QA closes ALL (TSS) JIRA issues • Except our own tasks #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 13
#lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 14
Jira Workflow - Issue tracking process • Anyone can open a bug or improvement ticket • Ticket is automatically assigned to Component owner • Triage - Validity - Duplicate - Correct attributes • Process through the workflow - External vs Internal facing Jira projects • Tickets from External project duplicated into Internal project #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 15
Agenda - Software testing process - Tools • Automated/Manual • Issue tracking/Project management • Requirements traceability - Results reporting Status Future goals and plans Questions and Discussion #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 16
Testing Results - Results stored online • https: //mebsuta. lsst. org/~rbovill/ • https: //ts-ci. lsst. codes/ - Email sent to all interested parties when results are posted • • • TSSW team CAMs T&S management T&S Systems Engineer T&S Chief Engineer #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 17
Results Deep Dive #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 18
Robot-Framework Results - Pass -> <salgen_results_green. tar. gz> - Fail -> <salgen_results_red. tar. gz> #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 19
Jenkins Continuous Integration - https: //ts-ci. lsst. codes/ #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 20
Agenda - Software testing process - Tools • Automated/Manual • Issue tracking/Project management - Results reporting Status Future goals and plans Questions and Discussion #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 21
Testing Status - Complete • ILC (Inner Loop Controller) calibration software • FAMT (Force Actuator Model Tester) • ILC Firmware - Current • • XML validation SAL (Service Abstraction Layer) M 1 M 3 Support System EFD schema validation #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 22
Agenda - Software testing process - Tools • Automated/Manual • Issue tracking/Project management - Results reporting Status Future goals and plans Questions and Discussion #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 23
Testing Next Steps - Next • Aux. Tel • AOS SW (OFC and WEP) • Scheduler - Future • M 1 M 3 Thermal Control • Summit Facility #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 24
Testing Goals - Software Integration Testing • Dave’s simulation cluster? • Mock control room? • Dedicated software testing environment? - Automate the Pathfinder Activities - Deployment validation • Lab (Patrick Ingraham and Tiago Ribero) • Puppet deployments (Andrés Villalobos) - EFD performance #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 25
What I Need - Feedback - Technical Support • Jenkins - Simulators - Proper Unit testing - Stable code • XML, SAL #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 26
Agenda - Software testing process - Tools • Automated/Manual • Issue tracking/Project management - Results reporting Status Future goals and plans Questions and Discussion #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 27
Questions and Discussion #lsst 2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 28
- Slides: 28