Chicago SPIN Metrics Creating a Balanced Scorecard aligned

Chicago - SPIN Metrics - Creating a Balanced Scorecard aligned to Corporate Objectives and Goals John Durbin Johndurbin@ieee. org At the Chicago SPIN May 2004 © John Durbin Slide 1

Balanced Scorecards at the Operational Level u How to: · Align Metrics to organizational scorecards · Align Bottom-up Balances Scorecard to the next higher level u Do you have points of special interest? u Balanced Scorecard used is just an example for bottom up alignment to “organizational goals” May 2004 © John Durbin Slide 2

Topics NOT covered… u Why Balanced Scorecards were devised? u Good and Bad about Balanced Scorecards u Metrics vs. Performance indicators u Well defined metrics u Successful metrics deployment u Negative Behavioral impact of metrics u Life cycle for metrics for changing behavior u Etc. At the Chicago SPIN May 2004 © John Durbin Slide 3

Review: for Common terminology What is a metric? u. A Quantitative result from one or more measurements u. And is Verifiable from a defined process May 2004 © John Durbin Slide 4

Review: for Common terminology How are metrics used? u. Support Decisions or u. Drive change or u. Diagnose problems At the Chicago SPIN May 2004 © John Durbin Slide 5

Review: for Common terminology Performance Indicators - Flavors Performance Indicators maybe used for: Lessons Learned Execution Planning Performance Indicators Surveys Measurements / Metrics Assessments May 2004 © John Durbin Slide 6

Where to start? u Mission (Values – Interests) · To help our clients create the metrics-driven processes and systems needed to change to be more successful. u Strategic Imperatives u Critical Success Factors (capabilities) u Key Performance Indicators (KPIs) > Goals u Align Management and Decision-Making Processes May 2004 © John Durbin Slide 7

Balanced Scorecard Purpose Strategy Process People Performance measures provide an ongoing mechanism for measuring an organization’s success in channeling its resources toward achieving its near-and long-term strategic goals. Performance measures provide actionable data and use targets for achieving operational excellence in critical processes to focus on proper allocation of resources and appropriate incentive for improvement. Performance measures align the objectives of the individual with the overall organizational strategy ensuring an overall commitment toward common goals and objectives. May 2004 © John Durbin Slide 8

Balanced Scorecard (Norton) First use at Analog Devices in 1987 Financial Objectives Measures Targets Initiatives Customer Internal Business Processes Learning & Growth At the Chicago SPIN May 2004 © John Durbin Slide 9

Define Key Indicators Sales Expertise Integration w/other Business Units Key Performance Indicators Or Goals • Win-Loss Ratio • % Cross-BU Sales • Milestone Completion Build Brand Image Aggressive Marketing Project Management Expertise • # New Customers • On-time Project Completion • # New Distributors • Market Share Increase Value Perception Increase Service Turnaround • Service Response Time • Customer Satisfaction Fix Infrastructure Train People Develop Systems • Inventory Turns • Training Effectiveness • Milestone Completion Strategic Imperatives Be a Full Service Provider Critical Success Factors May 2004 © John Durbin Slide 10

Need for Balanced Metrics for Performance u Avoid missing key issue u Long term focus u Do the Right Things u Avoid disasters (1 -Digital Equip. > Selling incentives on $ of computers; not profits, 2 - Tech. Co. > $ ship product on-time: no $ for/of repair) At the Chicago SPIN May 2004 © John Durbin Slide 11

A - Balanced Scorecard u Customers give us top two scores of five- 70% u Customers recommend our product - 80% u Met or exceed profit targets - $$. u Demonstrate SEI CMM level 4 by… ; -) u… u Employees more than satisfied > 75% u “Grow” 300 new Six Sigma Black Belts At the Chicago SPIN May 2004 © John Durbin Slide 12

A Fed. Gov. Balanced Scorecard Customer Perspective Internal Business Processes Perspective Acquisition Excellence: Effective Quality Control System - Ratio of protests sustained by General Accounting Office and Court of Federal Claims Acquisition Excellence: Effective use of Alternative Procurement Practices - Number of actions using Electronic Commerce Fulfilling Public Policy Objectives - % achievement of socio-economic goals - % competitive procurement of total procurements Customer Satisfaction - % of customers satisfied with timeliness - % of customers satisfied with quality Effective Service Partnership - % of customers satisfied with the responsiveness, cooperation, and communication skills of the acquisition office Learning and Growth Perspective Information Availability for Strategic Decisionmaking - The extent of reliable management information Quality Workforce - % of employees meeting mandatory qualification standards Employee Satisfaction: Quality Work Environment - % of employees satisfied with the work environment Employee Satisfaction: Executive Leadership - % of employees satisfied with the professionalism, Financial Perspective Minimizing Administrative Costs - Cost to spend ratio Maximizing Contract Cost Avoidance - Cost avoidance through use of purchase cards - % of prompt payment interest paid of total $ disbursed culture, values and empowerment At the Chicago SPIN May 2004 © John Durbin Slide 13

Balanced Scorecard For Operational Group u Support Organizational Goals & Results u Aligns to Top Scorecard (s) u Consistent focus and perspective u How do we get a Balanced Scorecard defined to align upward? · GE’s (conceptual) Formula view or · Pyramid conceptual model At the Chicago SPIN May 2004 © John Durbin Slide 14

Cook Book Recipe: Two Dimensions to Build u First, VERTICALLY Build alignment to Goals and Metrics above you. u Second, HORIZONTALLY Derive a balanced set of goals and metrics across top level perspective. u Last, Select and Prune Metrics At the Chicago SPIN May 2004 © John Durbin Slide 15

GE’s Big Y, little y & x Y Results that matter = f (y 1 , y 2 , y 3. . . ) Which Major activities will enable you to achieve those results? Management Stewardship On-going Sponsorship and Review At the Chicago SPIN May 2004 © John Durbin Slide 16

Y = f(y ) i yi Enabling Results that matter = and yi = f(xij) f (xi 1 , xi 2 , xi 3. . . ) Which Operating activities will enable you to achieve those results? Results/ Process Focus Process owner’s Review At the Chicago SPIN May 2004 © John Durbin Slide 17

Conceptual Model for Your Balanced Scorecards Drivers: Topdown Enterprise Balanced Scorecard: Financial, Customer, Internal Business, and Innovation and learning. Stakeholders Organization Enterprise Functional Management Project Manag’t Team or Work Cells Enablers: Bottom-up Metrics Drive Improvements Operating unit Scorecards Operating Units Across different initiatives Few Metrics Deliverable and milestone focused Milestone status Product results Key Performance Indicators Process support Process, diagnostic and Supportive Different Processes Metrics helps you do what you are doing while you are doing it! May 2004 © John Durbin Slide 18

Align Metrics for Consistency of Purpose Drivers: Topdown Stakeholders Customers Organization Team or Work Cells Market Business Unit Vital Few Metrics other layers Across projects and Methods Functional Management Project Manag’t Customer Satisfaction Employee Satisfaction, ROI Profits: $, DM, FF, etc. Customer Survey Cost/Benefits Deliverable and Phase focused Process, Diagnostic and task focused Pla n Enablers: Bottom-up Metrics Drive Improvements An aly ze De si Op Bu er ild Phase status Exit criteria Key Process Indicators Process support RUP, Agile, Waterfallate Hardware Dev. Marketing - Logistics gn Helps you do what you are doing while you are doing it! May 2004 © John Durbin Slide 19

Performance Indicators Metrics else Indicators What? Time & Cost Of resources Inputs Why? Attributes • People • Tools • Techniques • Physical Environment Your Process -Development -Enhancement -Support May 2004 What? Product Size Defects Outputs © John Durbin Slide 20

A Project Process’ Metrics Model -S ch ed Project Metrics ? Ti m Analyze Design Process Attributes • People • Environment • Tools • Techniques Build Output s Size s. Defects Integrate Resources & People May 2004 s le Input s. Time s. Cost Plan sk ab e Productivity Rate Product Test Defect Density Duration Variance Percentage Cost per unit code Ri er i R iv el ? k s D ul e QUALITY Deploy Operate Risk? © John Durbin Slide 21

Balanced Scorecard (Norton) First use at Analog Devices in 1987 Financial Objectives Measures Targets Initiatives Customer Internal Business Processes Learning & Growth At the Chicago SPIN May 2004 © John Durbin Slide 22

Balanced Scorecard Horizontally u Financial Perspective: (ROI) · How do we look to our shareholders? u Internal Business Perspective: (Process) · What Business Processes must we EXCEL at? u Customer Perspective: (Vision) · How should we appear to our customers? u Innovation & Learning Perspective: (Change) · How do we continue to improve and add value? At the Chicago SPIN May 2004 © John Durbin Slide 23

Balanced Scorecard Vertically – A Drill Down u Improve Customer Satisfaction: · Reduce delivered software defects – Increase software Verification t Better Requirements Traceability to …. t Improve Inspections effectiveness to…. – Improve software Validation · Faster feature enhancements – Use Agile paradigm to continually verify … At the Chicago SPIN May 2004 © John Durbin Slide 24

Scorecard trials u Improve Customer Satisfaction · Reduce delivered software defects – Better customer acceptance coverage <? ? ? – Greater storage for test cases <? ? ? – New Configuration Management tools <? ? ? – Use Agile Methods <? ? ? · What is common to the above? (Internal vs goal focus) At the Chicago SPIN May 2004 © John Durbin Slide 25

XYZ Retail Goods Co. u L 1 - Products in every Wal-Mart · L 2 Timely and Accurate market research L 3 Improve link to data sources l 4 Improve extract timing and accuracy of market data. · L 2 Lower Costs – L 3 Reduce delays of misplaced goods l 4 Correct tracking system for Goods availability u L 1 - 20% annual sales growth At the Chicago SPIN May 2004 © John Durbin Slide 26

Critical Success Factors for Metric Definitions u Anticipate “What’s In It For Me? ” · For Sponsors - Owners · For Participants u Seek common and clearly defined terminology (currency for Knowledge) u Must have owner committed not just involved May 2004 © John Durbin Slide 27

Select and Prune Metrics “Not everything that can be counted counts. Not everything that counts can be counted. ” Albert Einstein May 2004 © John Durbin Slide 28

For a small group or persons Minimize number of Metrics u Too many metrics to “focus on” causes “thrashing” or dysfunction behavior. u “Foreground metrics” - people may be measured by << less the 5 (maybe 7) u “Background metrics” – people must not be measured on >> are for heads up and MUST be trusted as non performance measurement – these may be prior metrics that were always under “control” May 2004 © John Durbin Slide 29

Examples of Balanced Set for Software Development u Vital Few Metrics (Like for internal delivery) u Testing Metrics (Like a stand alone group) u SQERT <an FLA> (Like for Contracts) May 2004 © John Durbin Slide 30

The Vital Few Metrics are a Balanced Set u Quality · Process Quality · Product Quality u Performance · Productivity Rate · Delivery Rate u Predictability · Duration Variance Percentage · Effort Variance Percentage May 2004 © John Durbin Slide 31

Example Metrics for Measuring Systems Building Product Size Effort Productivity Rate = (speed of the process) No. of Defects Found in Product Test Product Size Defect Density = Product Size Elapsed Time Delivery Rate = (elapsed time) Cost Per output unit Project Cost Product Size = May 2004 © John Copyright 1996 Durbin Slide 32

Minimize Source Data Measurement Data # of Faults (for 3 month period) Metrics 1. Fault Density # of Defects found During Product Test 2. Defect Density # of Project Function Points Planned Start Date (of analysis) 3. Productivity Rate Planned End Date (of first roll-out) 4. Delivery Rate Actual Start Date (of analysis) 5. Duration Variance Percentage Actual End Date (of first roll-out) 6. Effort Variance Percentage Actual Effort Planned Effort May 2004 © John Durbin Slide 33

A Testing Metrics are a Balanced Set u Productivity · Test Execution Rate · Problem Fix Rate u Process Quality · Repair Effectiveness Percentage · Repair Effort Percentage u Product Quality · Number of problems by module u What could be added? May 2004 © John Durbin Slide 34

S. Q. E. R. T. Balanced Set u. Scope u. Quality u. Effort u. Size u. Risk u. Timeline u. Cost u. Defects May 2004 © John Durbin Slide 35

References… u u u u Cox, B. G. and Chinnappa, B. N. (1995). "Unique Features of Business Surveys. " Ch. 1 in Business Survey Methods. Wiley. NY. Deming, W. Edwards. (1986). Out of the Crisis. MIT Center for Advanced Engineering Study. Cambridge, MA. Hibbard, J. (1997). "Knowing What We Know". Information Week. [Oct. 20]. pp. 46 -64. Juran, J. M. (1989). Juran on Leadership for Quality. Free Press. New York. Gabor, A. (1990). The Man Who Discovered Quality. Penguin Books. New York. Kaplan, R. S. , and Norton, D. P. (1996) The Balanced Scorecard: Translating Strategy into Action. Harvard Business School Press. Boston. Shewhart, W. (1939). Statistical Method from the Viewpoint of Quality Control. in Deming, W. E. , ed. , Graduate School of the Department of Agriculture. Washington, DC. At the Chicago SPIN May 2004 © John Durbin Slide 36

Other sources u Me>> Johndurbin@ieee. org u http: //www. balancedscorecard. org/ u http: //oamweb. osec. doc. gov/bsc/guide. htm u For more than you wanted to know: http: //harvardbusinessonline. hbsp. harvard. ed u/b 02/en/home/index. jhtml? _requestid=20215 u http: //www. sixsigmabenchmarking. com/ u http: //www. som. cranfield. ac. uk/som/cbp/pma/ At the Chicago SPIN May 2004 © John Durbin Slide 37

End of presentation u Remaining slides are FYI extras · Goal Question Metric · Deployment Critical Success factors At the Chicago SPIN May 2004 © John Durbin Slide 38

Goal-Question-Metric Paradigm u Top-down selection process u Linked to organizational goals u Ensures traceable and purposeful metrics u G-Q-M process · Identify your KPI or goals · What questions can you ask to determine whether or not your goal was met? · What metrics answer your question? May 2004 © John Durbin Slide 39

Applying G-Q-M u Must decide responsibilities for these actions · · · Defining the Metrics Communicating Educating Gathering Data Reporting Data Applying Metrics Decision, Change, Diagnosis Victor R. Basili – Univ. Maryland May 2004 © John Durbin Slide 40

G-Q-M Relationship May 2004 © John Durbin Slide 41

G-Q-M Example Goal: Maximize customer satisfaction Question: How long does it take to fix a problem? Metric: Mean time to acknowledge problem. Mean time to deliver solution. Schedule versus actual delivery. Question: Metric: How many problems are affecting customers? Incoming defect rate. Metric: Open critical and serious defects. May 2004 © John Durbin Slide 42

G-Q-M Example Improve Inspection Results Goal: Question: Metric: Is time well focused? Are participants prepared? - Review time per inspection (< 2 hours target) - Preparation per inspection time (1 to 2 hours) - # Interruptions (zero target) - Materials Available (y, n) May 2004 © John Durbin Slide 43

Process Maturity for Metrics Idealistic curve Reduce to none or One Process Metrics to sustain quality or productivity Effectiveness Technology transfer curve or process maturity curve Several metrics to increase both quality and productivity two or less metrics to increase quality Check other processes for metrics conflicts “Doing it” Metrics to support transition & learning Use over TIME May 2004 © John Durbin Slide 44

Metrics Definition Template Why? Who Owns? Defined? When? Œ Œ Ž ‘ �� Suppliers? Baseline & Target? Related Metrics? � �� ’ ’ May 2004 © John Durbin Slide 45

Deployment Critical Success Factors? u Definition u Deployment u Support u Reinforcing u Management Communications May 2004 © John Durbin Slide 46

Checklist for Better Behavior Metrics should: u u u Support goals linked to business objectives Support goals or higher level metrics Assess products and process (not people) Be manageable alongside the other metrics Be meaningful to people reporting Be understood by all affected Encourage correct positive behavior Be a positive (+) indicator (when possible) Be visibly acted upon (decisions and diagnostics) Be collected from within the target process Be about something that the measured group can control May 2004 © John Durbin Slide 47