MES Continuous Integration Continuous Deployment 2020 12 16

MES Continuous Integration. Continuous Deployment 2020 -12 -16

MES CI/CD Goals • Support System Integration Testing of modules • Quality • Reliability • Consistency • Reproducibility • Flexibility • Ease of maintenance 2

MES CI/CD Focus Areas • Testing • Security • Automated Build • Build Integrity • Automated Deployment • Deployment Environments • Quality Assurance 3

MES Dev. Ops Overview • Agile Development • Iterative Development (Short Sprints) • Customer Involvement to manage backlog of issues • Demo, retrospective, and release Develop Build Release Test Deploy Monitor Agile Development • Continuous Delivery • Automated project builds, integration testing, acceptance testing • Failure notification Continuous Delivery & Integration Continuous Deployment • Release Repository Manual Step Automated Step Error Reporting • Automatically built and versioned • Automated Release Deployment • Deployment Pipeline • Environments: production, staging, test, and local development • Monitoring • Failures, system load, response times, queue lengths Developer creates pull request or commit Code Repository Automated Build Automated Integration Testing Automated Acceptance Testing Automating Continuous Delivery 4 Project Build in Artifact Repository Dev, Test, Staging, or Production

MES CI - Development • Defined Jira work items • git branching strategy • Develop, Test, Build locally, Test local deployment • Pull Request Management • Development Branches of components 5

MES CI - Jira Items • Jira required metadata • • • 6 Summary Description of item and verification Components Epic Sprint Links to git commits and Pull Requests

MES CI - git branching strategy • git repositories in SDLC account • git feature branches for development • Development branch only updated through Pull Requests from feature branches • Automatically merge, build, and test with latest development branch • Master branch only updated through Pull Request from development branch • Release is a collection of projects from master branches with git tags and hashes 7

Continuous Integration Development Tools Jenkins Manager 8 Anchore Engine

MES CI - Pull Request • Newly developed code that is ready to be merged into the rest of the codebase • Developers create Pull Requests to merge feature branches into development branch • No direct commits to main branches (master or development branches) • Pull Requests go through automated process before manual review • Pull Requests are declined for: • Build failures • Test failures • Failing code metrics • Code Review and Acceptance into Development Branch • At least one reviewer is required to approve the pull request • Perform Static Code Analysis • Builds and tests with other project dependencies 9

MES CI - Pull Request report • Displays overall build status • Links to Jira work items • Reports on dependent repositories • Informs developer of build hierarchy • Itemizes every build’s status • Static code analysis • Docker image vulnerabilities • Dependency resolution mismatches 10

MES CI - Pull Request report sections • Pull Request details • Downstream Pull Request list • Project dependency graph • Project build list and status • Dependency mismatches • Build information • Container security scan • Build artifacts • Project build dependencies 11

MES CI - Pull Request Pull request report dependency graph 12

MES CI - Pull Request 13

MES CI - Pull Request Dependency Check 14

MES CI - Pull Request Declined Pull Request 15 Pull Request Gating

MES CI - Static Code Analysis • Performs a code check against a set of rules • • Ensures code quality Analyzes testing coverage Flags potential vulnerabilities Highlight areas of improvement • Compares previous version of code (a. b. c) to new version of code (x. y. z) • Allows gating for certain conditions, preventing flagged code from moving forward 16

MES CI - Static Code Analysis 17

MES CI - Static Code Analysis 18

MES CI - Static Code Analysis Issue Detection • • Bugs Vulnerabilities Security Issues Deviations from standard accepted practices • Anti-pattern detection 19

MES CI - Static Code Analysis Code Test Coverage • • 20 Coverage percentage Lines of code covered Conditionals covered Highlights uncovered or partially covered code

MES CI - Static Code Analysis Code Quality Badges 21

MES CI - Linked Feedback Bitbucket Pull Request Report Jira Issue 22

MES CI - Quality Assurances • Prevent direct commits to main branches • Jenkins Manager ensures builds run successfully before merging • Sonar. Qube quality gates • Sysdig/Anchore Engine scans Docker images for security vulnerabilities • Custom built easy-to-read reports of pull requests • All developer-facing interfaces link to each other for quick review 23

MES CD • Configuration as code, AWS account abstraction • Scripted Deployment • Multiple environments • SDLC Accounts 24

MES CD - Configuration as Code • git Project to manage deployments 25

MES CD - Configuration as Code • Account abstraction • Variables for • Deployment environment • AWS Account • Other configuration items • Multiple deployment environments • Special purpose deployments 26

MES CD - Scripted Deployment • Documented deployments • Repeatable deployments • No manual configuration/deployments • Deployments are idempotent • Terraform based deployments to AWS • Multiple AWS systems, avoid custom configuration 27

MES CD - Terraform Deployment • Deployments Output • Detailed logging output • Deployments only of tagged git repository 28

MES CD - Terraform Deployment 29

MES CD - Multiple Environments • Environment isolation • Specialized testing • Release progression through environments • PHI limited to specific environments 30 Name Purpose/Activities Initial Development/Module System • Level setting and expectations between module and SI • Simulate what MES-IP is going to look like to modules, file transfers, what kind of files, etc. • No PHI/PII Integration Testing • Ephemeral environments for specific testing • Special purpose testing • Expectation is that running some automated testing • Some certification beyond self-asserting • No PHI/PII Cross Module/Business System Testing • Cross Module Testing for targeting sets of modules • Long running availability with other modules as they come online • No PHI/PII, all testing before moving to PHI/PII MEST Enterprise Stable Build Environment • PHI/PII support • Similar to production • Follows Configuration Management • UAT Testing Production • Production Environment • Operational System Data Migration • PHI/PII support • Support data migration activities

MES CD - SDLC Accounts • Deployments run from AWS SDLC Account • Approved Automated Deployments to environments • Can run scheduled deployments • SDLC Account for monitoring and debugging deployment process Workload Accounts Security/Monitoring Accounts Infrastructure Accounts 31

MES CI/CD Next Steps • Release documentation • Integration Testing with external systems • Deployment report generation • System liveness checks • Components individually define the tests to run after deployment • Availability checks to verify deployment • Data mutating tests • Limited to non-production 32