European International Software Quality Assurance Conference Jake Brower

European International Software Quality Assurance Conference Jake Brower Sr. Engineering Manager, Scrum Master, Certified Agile Facilitator Credit Karma, San Francisco, California The Quality Scorecard Riga. March 22– 23, 2019 sqadays. eu

Who am I? 1994 – 2006: • Taught myself HTML and built/tested web sites for local nonprofit companies • Worked at various companies testing everything from AS 400 mainframe systems to commercial websites and desktop apps • Tested early mobile devices/apps and commercial off-the-shelf software • Team leadership The Quality Scorecard

Who am I? 2006 – Present: • People management • Company-wide quality transformations that focus on pushing quality further left • Process engineering/adoption • Building world-class teams • Utilizing metrics to help teams improve their development processes The Quality Scorecard

What Issues were we Trying to Solve? We didn’t have a consistent way to measure quality for all engineering teams We didn’t have set Quality KPIs that help us align with our eng objectives We didn’t have a way to quickly visualize the quality level for each team We didn’t have a way to identify quality issues earlier so we can take action We didn’t have a system to present metrics in an automated fashion The Quality Scorecard

What is a Quality Scorecard? The Quality Scorecard serves as a visual performance measurement framework, which can help Engineering teams both articulate and act upon the overall vision and strategy by: • Facilitating effective & consistent communication/language across all eng teams • Using Quality KPIs to drive focus for improvements • Enabling frequent reviews to ensure consistent practices • Ensuring organizational alignment The Quality Scorecard

The Journey to v 1 Defining the requirements with engineering leadership; getting buy in The Quality Scorecard

The Journey to v 1 Researching COTS tools that might match our requirements Defining the requirements with engineering leadership; getting buy in The Quality Scorecard

The Journey to v 1 Deciding on the best path forward: off-the-shelf or build in house Researching COTS tools that might match our requirements Defining the requirements with engineering leadership; getting buy in The Quality Scorecard

The Journey to v 1 Documenting the architecture; deciding on the technology Deciding on the best path forward: off-the-shelf or build in house Researching COTS tools that might match our requirements Defining the requirements with engineering leadership; getting buy in The Quality Scorecard

The Journey to v 1 Building out a prototype Documenting the architecture; deciding on the technology Deciding on the best path forward: off-the-shelf or build in house Researching COTS tools that might match our requirements Defining the requirements with engineering leadership; getting buy in The Quality Scorecard

The Journey to v 1 Designing the database Building out a prototype Documenting the architecture; deciding on the technology Deciding on the best path forward: off-the-shelf or build in house Researching COTS tools that might match our requirements Defining the requirements with engineering leadership; getting buy in The Quality Scorecard

The Journey to v 1 Front end/API development; lots of testing Designing the database Building out a prototype Documenting the architecture; deciding on the technology Deciding on the best path forward: off-the-shelf or build in house Researching COTS tools that might match our requirements Defining the requirements with engineering leadership; getting buy in The Quality Scorecard

The Journey to v 1 Soliciting feedback from Engineering leadership Front end/API development; lots of testing Designing the database Building out a prototype Documenting the architecture; deciding on the technology Deciding on the best path forward: off-the-shelf or build in house Researching COTS tools that might match our requirements Defining the requirements with engineering leadership; getting buy in The Quality Scorecard

The Journey to v 1 Building out test/production environments; hook up DB Soliciting feedback from Engineering leadership Front end/API development; lots of testing Designing the database Building out a prototype Documenting the architecture; deciding on the technology Deciding on the best path forward: off-the-shelf or build in house Researching COTS tools that might match our requirements Defining the requirements with engineering leadership; getting buy in The Quality Scorecard

The Journey to v 1 Deploying to UAT for testing/bug fixing and then to Prod Building out test/production environments; hook up DB Soliciting feedback from Engineering leadership Front end/API development; lots of testing Designing the database Building out a prototype Documenting the architecture; deciding on the technology Deciding on the best path forward: off-the-shelf or build in house Researching COTS tools that might match our requirements Defining the requirements with engineering leadership; getting buy in The Quality Scorecard

The Journey to v 1 Communicating to engineering (email, live demos, workshops) Deploying to UAT for testing/bug fixing and then to Prod Building out test/production environments; hook up DB Soliciting feedback from Engineering leadership Front end/API development; lots of testing Designing the database Building out a prototype Documenting the architecture; deciding on the technology Deciding on the best path forward: off-the-shelf or build in house Researching COTS tools that might match our requirements Defining the requirements with engineering leadership; getting buy in The Quality Scorecard

The Quality Scorecard – Technology Front End: • The Scorecard is a Sinatra front end app built in Ruby, HTML, and CSS API/Back End: • The Scorecard collects data from various systems through their various APIs. All queries are stored and executed in the code itself at runtime. • Postgres database that stores up to a year’s worth of data. The Quality Scorecard

The Quality Scorecard – Technology Jira Test. Rail Google Sheets Pager. Duty Git Others The Quality Scorecard API / Collect Metrics Store in Postgres DB Data Pulled and Scoring Applied

The Quality Scorecard

The Quality Scorecard - Dimensions The Quality Scorecard

The Quality Scorecard – Dimension Scores The Quality Scorecard

The Quality Scorecard - KPIs Key Performance Indicators (KPIs) The Quality Scorecard

The Quality Scorecard – KPI Scores Key Performance Indicators Scores (red means the KPI isn’t meeting the target) The Quality Scorecard

The Quality Scorecard – KPI Popup Clicking on a KPI will present a popup that shows score/calc info as well as links The Quality Scorecard

The Quality Scorecard - SDLC With this Dimension and KPIs, we’re trying to understand how teams are at obtaining sign off from supporting teams (Compliance, Security, Legal, QE, Eng) for each epic. For our company, these are FTC compliance related and we must strive to get to 100% approval rate. This data is being pulled directly through the Jira API. The Quality Scorecard

The Quality Scorecard - Coverage With this Dimension and KPIs, we’re using a home-grown tool to see how well teams are at adhering to Git best practices within their Front End and Back End Repos. These Best Practices include things like having a README. md file, approvers, protected default branch, defined owners, etc. We will also use this dimension to track metrics around code coverage and automation coverage. This data is being pulled directly from Github. The Quality Scorecard

The Quality Scorecard – Pre-Production With this Dimension and KPIs, we’re measuring teams’ ability to find Blocker and Major defects in test environments other than Production, proper use of the backlog (we don’t want ANY P 0/P 1), and their ability to resolve defects prior to Production. Since we are also a mobile-first company, we want to see the breakdown of mobile-specific and non mobile-specific backlog items. This data is being pulled directly through the Jira API. The Quality Scorecard

The Quality Scorecard - Production With this Dimension and KPIs, we’re measuring issues that are occurring in Production. Ideally, we want these numbers to be zero. Here, we’re showing both mobile and non-mobile issues. The last KPI in this dimension is used to show any security vulnerability issues for a given team/product/feature. Again, we want this number to also be zero. This data is being pulled directly through the Jira API. The Quality Scorecard

The Quality Scorecard - Platform With this Dimension and KPIs, we’ve added non-defect related measures from the Platform/Site Operations side of the company. These metrics show good each team is at adhering to company-wide standards around operational readiness. We’re also gathering metrics around in incidents and average resolution time. These can things like outages, site issues, breaches, Production defects, etc. This data is being pulled directly through the Jira and Pager. Duty APIs. The Quality Scorecard

The Quality Scorecard – Weighting and Scoring Weighting: • All KPIs in a single dimension are weighted against each other as not all are high impact. For instance, Security vulnerabilities are weighted higher than P 0 or P 1 bugs in Prod as the potential impact is much greater. Scoring: • All weighted KPIs in a single dimension are scored based on a 4. 0 scale. The Quality Scorecard

The Quality Scorecard – Adoption This is harder than implementation Have meaningful discussion with your teams; create a story about what you see Adoption must include using the Scorecard on a regular basis The Scorecard must help you focus on making improvements The Quality Scorecard

The Quality Scorecard – Context and Remediation We’ve created a separate page that gives context to the scores they see as well as remediation steps for lower scores. Often teams don’t know where to start when they encounter low scores so we wanted to provide some actionable steps they can take to improve their processes. The Quality Scorecard

The Quality Scorecard – Trending The Quality Scorecard

The Quality Scorecard – Adoption Score > Conversation > Story > Remediation The Quality Scorecard

The Quality Scorecard – Main Takeaways A Quality Scorecard can help drive focus on making improvements in the SDLC Choose KPIs that make sense for you to track without much team customization You don’t need many resources to make this a reality Remediating low scores should be easy if done on a regular basis The Quality Scorecard

Paldies! Спасибо! Thank you! The Quality Scorecard