Supercharge Your Balanced Scorecard Using Variationbased Metrics Mike

Supercharge Your Balanced Scorecard Using Variation-based Metrics Mike Hyzy August 2001

How to Tie Together? • Balanced Scorecard – Tool for measuring implementation of strategic plan • 4 ‘Balanced’ perspectives: Financial, Customer, Process, Learning & Innovation • Good at identifying metrics to keep focus on long-term objectives and strategy implementation • BUT, poor providing method for analyzing data and taking action • Six Sigma – Method for reducing defects and delivering financial benefits • Focus on customer voice, financial impact, variation/defects • Good at dramatically improving processes and delivering financial return • BUT, often not integrated into corporate strategy • Variation-based Metrics – Powerful approach of applying Six Sigma statistical tools to the Balanced Scorecard August 2001 NCR Confidential 2

Tools for Understanding Variation Based Metrics • Variation Based Metrics (VBMs) are presentations of metrics which allow you to look for patterns in the data • Typically done in two ways: Patterns over Time Patterns not related to Time Run charts Frequency Plots (Histogram) Control Charts: Pareto Charts Individual Charts P, C, U Charts Others Control Charts They help us distinguish between routine variation (common cause) and exceptional variation (special cause) August 2001 NCR Confidential 3

Limitations of Mean-Based Metrics • Be careful of mean-based metrics – Seen in typical monthly reports – Example: • 2 departments delivering product to customers • On the monthly report, they both appear the same (avg = 50) • If you were the customer, would you view them both as the same? ? ! – No comparison between 2 values can be global; data has no meaning apart from their context – Comparisons to beware of: • • • August 2001 Monthly averages Variance from target YTD values YTD variance Variance from same month last year NCR Confidential 4

Limitations of Mean-Based Metrics (Today’s Scorecards) • Any comparisons are LIMITED – Can’t identify patterns or trends • Plans and targets represent the Voice of the Customer (we hope)… not the Voice of the Process • Actions based on this data may be FLAWED • Single points give no indications of the SHAPE, CENTER or SPREAD • Our customers feel and remember VARIATION, not AVERAGES On AVERAGE He’s Comfortable!! August 2001 NCR Confidential 5

Continuous vs. Discrete Data • Continuous Variables are Preferable – Whenever possible, or when given the choice… select/gather continuous variables! • Crucial to consider during data collection planning when flexibility exists to identify and/or define various types of metrics to track for a project – Continuous variables are more “information rich” • Discrete variables capture only yes/no, met/not met, etc. • Continuous variables also capture “by how much” • More flexibility in the use of analysis tools w/ continuous variables • Unfortunately, the tendency with many companies has been to utilize discrete variables – Discrete are usually more easy to define and easier to measure • Majority of Balanced Scorecard metrics in prior years have been discrete August 2001 NCR Confidential 6

Continuous vs. Discrete Example • Example: On-time delivery… Approach #1: – Discrete: Capture Yes or No (1 or 0) as to whether delivery was made on time • Summarized by calculating % deliveries that were delivered on-time • BUT, using this approach we have NO IDEA by HOW MUCH we missed the target for those deliveries that were NOT on-time • Did we miss most deliveries by a few hours or a few weeks? ? ? • NO WAY to determine performance at different target levels, since the target is ‘baked into’ the met/not met determination Week 1 of Reporting Period Target % On Time Delivery = 60% (3 of 5) Can’t tell by how much we missed the delivery target! August 2001 Met target of 70% On-time Deliveries during 17% of the reporting periods (months) NCR Confidential 7

Continuous vs. Discrete Example • Example: On-time delivery… Approach #2: – Continuous: Capture delta time (say, hours) from targeted delivery time • Can now determine BY HOW MUCH we missed our deliveries • Can determine On-time delivery performance at varying target levels • What percentage were delivered no more than 1 day late? • Allows you to identify a lower specification limit as well, since very often the customer would consider it a defect if we delivered too early (they may not be ready or available) Target Range Target Week 1 of Reporting Period % On Time Delivery = 60% (3 of 5) August 2001 OR 16% of deliveries were on time or early NCR Confidential 22% of deliveries were between 3 days early and 2 days late 8

Matching Action to the Type of Variation • Understanding Variation is key – Actions taken are dependent upon the type of variation that exists – Taking the wrong action may make the process worse August 2001 NCR Confidential 9

Why Does It Matter? • Want to Avoid: – Missing signals • Not taking action when action is appropriate – Misreading noise for a signal of a change • Wasting time from taking action when none is appropriate • TAMPERING!! August 2001 NCR Confidential 10

Tampering Drives the Search for Answers… But there aren’t any!! What happened? The Big Gear Turns I’ll go find out. What’s going on? I’m looking! I’m looking The Big Gear Turns A Notch And The Little Gears Whirl In A Frenzy Trying To Keep Up August 2001 NCR Confidential 11

Variance Based Metric Example: Executable Orders • QUESTION: What Action Should You Take? ? ? – NCR Leadership Team Report – Detailed Process Report August 2001 NCR Confidential 12

Variance Based Metric Example Executable Orders • Now what action should you take? ? ? – Red/Yellow/Green color-coding based upon variance from target – 1 problem… where does the 2% value come from? ? ? • Is this level of variance an appropriate threshold for action? ? ? • One would assume that RED bars indicate a SERIOUS PROBLEM and YELLOW bars indicate a POTENTIAL PROBLEM – Looks like you should be acting almost EVERY MONTH! – Time to beat people up for their lousy performance, right? !! August 2001 NCR Confidential 13

Control Chart (Process Behavior) Executable Orders (1/99 - 5/00) Control Chart for % Executable Orders Blue UCL=96. 7% Green 96 Target Yellow 95 94 Avg=93. 6% 93 92 91 LCL=90. 5% 90 Jan 99 • No trends or shifts • EO Results between 90. 5% and 96. 7% are EXPECTED! Red % Executable Orders 97 • Process is STABLE May-99 Oct-99 Mar-00 May 00 Date • Current process only capable of performing in this range • EVEN THOUGH THIS DOES NOT ACHIEVE THE TARGET!! • Reacting to values between 90. 5% and 96. 7% is a COMPLETE WASTE OF TIME!! • Variation within these limits is due solely to RANDOM CHANCE! • Process Improvement required to more consistently meet the target (Six Sigma Project needed) • Existing BSC color-coding is misleading • Has NOTHING to do with actual capability of the process! Existing Balanced Scorecard color-coding for EO metric (+/-2% Variance from 95% target) August 2001 NCR Confidential 14

Frequency Plot (Histogram) Executable Orders (1/99 - 5/00) • QUESTION: Does knowing the average value (93. 6%) provide a complete picture for how this process is performing? ? ? Count (# Values) • Frequency Plot shows # of occurrences at specific values over a period of time (1/99 5/00) • Knowing only the mean (average) blinds you to the actual range of values (from 91% to 95. 5%) • Discover that data is approx. normally distributed % Executable Orders (1/99 - 5/00) August 2001 NCR Confidential 15

NCR Reporting Approach • Modify/Add Charts Contained in Monthly Review Templates to Illustrate Variation – “Bake” variation plots throughout monthly & quarterly reviews -- not just during brief Balanced Scorecard discussion • Use New ‘Exception Chart’ to Gain Insights from Variation – Contains metric details, type of variation, exception type, control chart, frequency plot (histogram), and ACTIONS (appropriate, based upon type of variation) • Better headlights to potential problems ahead • Reporting Approach – Review exceptions on Primary Scorecard during Monthly & Qtrly Reviews – Review exceptions on Secondary Scorecard only during Qtrly Reviews – Only Green or Red coloring to be used for summary view – Metric Exception if any one of the following is true: • Didn’t meet target during reporting period or • Process is unstable / special cause(s) present or • Process not capable of meeting target/spec. (target/spec. inside control limits) August 2001 NCR Confidential 16

SAMPLE Exception Variation Chart (EVC) August 2001 NCR Confidential 17

Exception Criteria for Metric Reporting Not an exception Exception - Fails Crit. 3 Metric Exception Criteria: 1 2 3 Exception - Fails Crit. 1 & 3 Exception - Fails Crit. 2 Exception - Fails Crit. 1, 2, 3 Didn’t meet target during reporting period Process is unstable / special cause(s) present Process not capable of meeting target/spec. (target/spec. inside control limits) Exception determination is made each reporting period for each point Exception - Fails Crit. 2, 3 August 2001 NCR Confidential 18

Where to Collect Data: Process or Result? X’s Y’s Need to measure BOTH X’s (process measures) and Y’s (results measures) August 2001 Customer sees the Y’s (results measures) NCR Confidential 19

Using Headlights to Predict Results • Goal is to Correlate Process Measures to Results Measures – Process measures from S-I-P of SIPOC • Inputs… also known as X’s – Results measures from O-C of SIPOC • Outputs… also known as Y’s – Verify, with data, that a specific set of inputs (X’s) affect key results metrics (Y’s) – Track process measures as leading indicators of results (better headlights) Y Y • Charts (on right) show scatter plots of X’s vs Y’s – Illustrate various levels of correlation Strong Positive Correlation X X Y Y Moderate Positive Correlation Y No Correlation X Y Y = f(X) X Strong Negative Correlation August 2001 NCR Confidential X Moderate Negative Correlation X Other Pattern No Linear Correlation 20

Who Does What ? ? Scorecard and Metrics Team: Process/Metrics Owner: • Conductor - everyone on the same page using the same definition, source, calculation • Musician - playing the instrument/process in order to accomplish goals; and using EDW to track results • Compiler - taking data, action items, notes from all area/process mgrs and putting them all together for overall WCS view to RFG • Consultant - providing training in data gathering, variance analysis, etc August 2001 • Manager - providing action items, leading Six Sigma teams, researching exceptions • Subject Matter expert - utilizing all available sources to record analyze change NCR Confidential 21

NCR Scorecard Best Practices • Each metric has a process owner. • Process owner roles and responsibilities are clearly defined. • Target Setting done with process owner and their management based upon real data • Process owner responsible for the collection and analysis of data, with assistance from MBB’s. • All process owners are GB or BB trained. – Kickoff mtg held for all Process owners. • Monthly meetings with BU VP and BSC Process owner. • Metric/process owners report their own results -- not presented by an independent Quality organization. – Drive greater accountability for their results • Spin off Six Sigma projects to address BSC gaps. • Use of variation analysis not an option. August 2001 NCR Confidential 22

Creating Order From Chaos: What You Can Do • • Increase your personal knowledge of variation Personally apply the tools to make variation visible Don’t tamper or contribute to tampering Partner with the business to get actual data and demonstrate - not just theory Enable your managers and leadership to understand the value of variation Act as a change agent to influence mangers and leadership to make the paradigm shift Create a simple transition plan within your business start now Read this book! Understanding Variation the Key to Managing Chaos, by Donald Wheeler August 2001 NCR Confidential 23