Jeffrey C Worthington Chair Energy and Environmental Division

Jeffrey C. Worthington Chair, Energy and Environmental Division American Society for Quality OEI Director of Quality Office of Environmental Information U. S Environmental Protection Agency Lorena Romero Cedeno Program Analyst Office of Environmental Information U. S Environmental Protection Agency ASQ 32 nd Annual National Energy & Environmental Conference San Antonio, Texas September 19 -20, 2005 1

Jeffrey Worthington- BIO n n Director of Quality for the USEPA Office of Environmental Information. Jeff has served as the Director of Quality Assurance for the USEPA Office of Research and Development (ORD) National Risk Management Research Laboratory (NRMRL) and as the Director of Quality Assurance for Tech. Law, Inc. He is an American Society for Quality (ASQ) Certified Quality Manager and ASQ Certified Quality Auditor. Jeff, Senior ASQ member, founding member of the Education Division, Chairs the ASQ Energy and Environment Division and participates on the ASQ Division Affairs Council. He is a founding member and serves on the Board of Directors for the recently established International Association for Information and Data Quality (IAIDQ). Jeff is a member of the Editorial Board of Quality Assurance, Science, and the Law and previously served as Editorial Board member for: the Journal of Environmental Forensics, Environmental Laboratory magazine, and Environmental Testing and Analysis magazine. He has been with the Federal Government since 1994. Jeff supported environmental engineering quality at NRMRL, joining a team authoring the combined quality and management system for EPA’s Environmental Technology Verification program. He co-lead the EPA team developing EPA’s Information Quality Guidelines. Jeff co-authored a peer review journal paper and received the USEPA Science and Technological Achievement Award, Level III for equating EPA policies and procedures to U. S. Supreme Court Sound Science Criteria (2002). Jeff has spoken at numerous national and regional conferences on the subjects of quality management, audit management, information quality planning and assessment, data authenticity, data quality, and data integrity. 2

Lorena Romero Cedeno - BIO Lorena R. Romero-Cedeño is the Manager of Quality for the Office of Planning, Resources and Outreach for the Office of Environmental Information. She graduated in 1999 from Colorado State University and currently is finalizing two master degrees in Community and Regional Planning and Latin American Studies from the University of New Mexico. Lorena has a passion for the integrity of environmental data being translated into Spanish targeted to the Spanish speaking public. She is currently conducting research on environmental data for her master thesis and holds two years of experience teaching Spanish Language at the University of New Mexico. 3

DISCLAIMER The opinions expressed in this technical presentation are those of the author and do not necessarily reflect the views of the US EPA. 4

SPONSORED BY MICKEY………. . WHY MICKEY? n n Statue theme in 2005 He’s cool!!! He is a performer… must know something about performance measurements He is a genius n n n Wears the best clothes Never ages Is very popular Is everywhere Has a lot of friends Certified quality inspector (usually wears white gloves) 5

MICKEY FACTS n Cherry Blossoms n Age n Oscar n First words n First cartoon character to ___ n Quotes 6

7

8

9

Selected Performance Measurement Systems Used in EPA and Federal Government n n n GPRA – Government Performance and Results Act FMFIA – Federal Managers Financial Integrity Act (management controls) EVM – Earned Value Management n n CPIC – Capital Planning and Investment Coordination (especially information technology) Balanced Scorecard 10

GPRA – Government Performance and Results Act n n Mandated by OMB Effort to identify and track project/program/process outcomes (vs. outputs) Managed at a high level in the organization In EPA, may include: n n n QA/QC procedures Data quality reviews Data limitations Error estimate Each office negotiates their own list of measures with OCFO and OMB Annual performance plan submittals – www. omb. gov 11

Outcomes vs. Outputs n n Outputs = productivity or efficiency metrics such as the number of reports written per month Outcomes = the impact that the output has on success of the process organization 12

FMFIA – Federal Managers Financial Integrity Act (management controls) n n n FMFIA Act of 1982 Agency weaknesses vs. Annual Management Accomplishments and Challenges report Office’s self-identify adequacy of “management controls” OIG, GAO, and OMB overview comments may be part of overview Action plan to make corrections and identify schedule and measures to correct 13

EVM -Earned Value Management n n Standard - ANSI/EIA-748 -A Standard for Earned Value Management Systems EV = value of completed work expressed in terms of the budget assigned to that work n Objective measure of work accomplished n Based on budgeted value of the work ‘What you got for what it costs you. ” Compliance standard criteria grouped in 5 areas n n n n Planning, scheduling, and budgeting Organization Analysis and management reports Revisions and data maintenance Accounting considerations Performance measurement n n n Establish a baseline Monitor in time units Review both cost variance and schedule variance from baseline 14

Office of Environmental Information Report- Earned Value Management: Fiscal Year 2004 Lessons Learned December 2004 n n Applied to assist IT project managers consistent with the ANSI/EIA-748 standard and in line with requirements of the OMB Exhibit 300. 3 categories of LESSONS LEARNED n n n Refinement of EVM methods Increasing consistency of project reporting Facilitating management analysis of EVM data 15

LESSON LEARNED 1 Refinement of EVM methods n n n Use EVM across all phases of mixed life-cycle projects Separate milestones and associated resources for system life-cycle phase Separate milestones and associated costs by contractor wherever possible Keep milestones from getting too large in duration, cost, or scope Attempt to limit milestones to a single fiscal year (or less) Establish objective measures for determining earned value 16

LESSON LEARNED 2 Increasing consistency of project reporting n Use standard template for reporting n Institute standard reporting cycles 17

LESSON LEARNED 3 Facilitating management analysis of EVM data n n Provide both numerical and graphical representations of EVM data Use color-coded standardized scoring system 18

Balanced Scorecard A Management System (not only a measurement system) that enables organizations to clarify their vision and strategy and translate them into action. n Provides feed back for both internal processes and external outcomes in order to continuously improve strategic performance and results. Balanced Scorecard Institute www. balancedscorecard. org 19

Balanced Scorecard – view organization from 4 perspectives n Learning and growth perspective n Business process perspective n Customer perspective n Financial perspective 20

Balanced Scorecard Business Process Perspective n n Products and services conforming to customer requirements Mission-oriented processes AND support (repetitive) processes n (easier to benchmark repetitive processes) 21

Balanced Scorecard Financial Perspective n n Timely and accurate funding data Cost-benefits 22

Balanced Scorecard Customer Perspective n Customer focus n Customer satisfactions n Matching products and services to customer groups 23

Balanced Scorecard Learning and Growth Perspective n n n Employee training Organizational culture toward personal and professional growth Keeping knowledge workers in a continuous learning mode 24

25

26

BRM – OMB’s Business Reference Model n n From OMB’s Federal Enterprise Architecture Program Management Office (FEAPMO) Identifying and categorizing an organization’s lines of business and subfunctions to relate them to the enterprise architecture 27

PRM – OMB’s Performance Reference Model n n n From OMB’s Federal Enterprise Architecture Program Management Office (FEAPMO) A standardized approach to IT performance Define measurement indicators Establish relationship inputs and outcomes Sets baselines and targets for improvement Closely related to the BRM 28

GENERALIZATION – for various performance measurement systems n n Relationship to organization’s vision and mission may need to be verified Need to verify measures: n n n n Are real Are meaningful Are not redundant Include quality (limitations on use) Are we measuring this item because it is easy to measure? Is the goal convenient or challenging? Applicability - What does measuring this particular thing tell you about the organization’s performance? Completeness – Are enough measures being tracked to characterize a broader sense of organizational performance? 29

How to Speak the Language of Senior Management Stephen George Quality Progress May 2003 OBSERVATIONS n The language of senior management is very different than the language of quality managers n To have their suggestions heard and accepted, quality professionals need to learn management’s financial vocabulary n There are seven steps you can take to improve communication with management 30

Speaking the Language of Senior Management The Seven Steps n n n n PROVE – quality professionals have to prove the need and then prove the value of proposed improvement processes COACH – quality managers must help senior management move goals, objectives, and strategies into action, …. To move from strategies to measures to projects INFLUENCE – involve senior management in quality processes, participate as trainers, incporated quality in reviews PARTNER – with the CEO if possible, or with any other senior manager, focus on the area of greatest need, ask for guidance from senior management PILOT – initiate a project in an area of value to the organization BENCHMARK – with senior managers in other organizations ALIGN – align the quality system language with the senior management language, everyone should be speaking the same language 31

Balanced scorecard for a quality system – HOW? ? n n n Look for ways for the quality system to demonstrate an improvement in each of the four perspectives Try to use a balanced scorecard for the quality system itself – can you really do this? Performance measures of the organization vs. performance measures for individual processes (e. g. , quality systems) – does this mean there is a new customer to consider in the balanced scorecard, the internal customer? n n How much you are spending internally on your own process, for example When do the management system and the quality system come into full alignment? 32

Balanced scorecard for a quality system n OUTPUTS n OUTCOMES 33

Balanced scorecard for a quality system n Learning and growth perspective n Business process perspective n Customer perspective n Financial perspective 34

Mickey says: - arithmetic is counting to twenty without taking off our shoes Quality Mickey says: Performance measurement of quality is measuring both outputs and outcomes 35

HOW CAN AN INTERESTED QUALITY MANAGER TRAIN THEMSELVES? n www. balancedscorecard. com 36

WHAT ELSE IS GOING ON AT EPA FOR QUALITY NOW? n n OEI is re-developing its 5 year QMP OEI has delegated authority for directives for Agency information policy, including quality policy 37

THE END 38

CONTACT INFORMATION Jeffrey Worthington 202 -566 -0997 worthington. jeffrey@epa. gov n Lorena Romero-Cedeno 202 -566 -0978 romeo-cedeno. lorena@epa. gov n 39

40

41

42

43

44

45

46

47

48

49

50

51

OVERVIEW n n n Trends in information quality What is a Body of Knowledge? Who owns a BOK? Basic structure Conclusion Resources 52

Trends in information quality MORE IS BETTER n More information is better? n Faster information is better? n More privacy is better? n More access is better? 53

Trends in information quality WHAT ARE THE TRENDS? n Information technology build-out phase is nearing completion n Increased access n Increased transparency n Recognition that info is a resource n Recognition that IT is not the strategic advantage (IT Doesn’t Matter) n Increased need for security n Increased need for control (? ) 54

What is a Body of Knowledge (BOK)? … an aggregate of what is known and understood within a field of endeavor due to familiarity gained through experience or association www. findmehere. com 55

What is the purpose of a BOK? n To provide a means for people with shared interest to better communicate n To provide formal organization and recognition to a knowledge area n To recognize and improve knowledge n To serve as the basis for a test or certification 56

International Association for Information and Data Quality www. iaidq. org • IAIDQ purpose – to create a world-wide community of people who: • Understand the critical roles data and information play • Recognize the consequences of poor quality data and information • Wish to help organizations enjoy the benefits of improved data and information • IAIDQ mission • Increase the awareness of the impact of poor quality data and information. • Help leaders understand that the high losses can be dramatically reduced. • Provide a network for members to exchange tips and techniques for quality improvement. • Provide opportunities to learn critical skills for making quality information and data a reality. • Membership types • Profession members • Academic members • Lay members • Student members 57

Usability Professionals’ Association http: //www. upassoc. org/ What is usability? The degree to which something – software, hardware, or anything else – Is easy to use and a good fit for the people who use it. 58

Usability Body of Knowledge n n n Methods User interface design principles and guidelines Organizational integration of usability (including managing usability teams, integrating usability into software development, introducing usability to organizations and clients and making a business case for usability) Roles, skills, and job categories for usability professionals Definitions of usability terms 59 Related fields and disciplines

What is usability? n n … a quality or characteristic of a product … whether a product is efficient, effective, and satisfying for those who use it …the name for a group of techniques developed by usability professionals to help create usable products … a shorthand term for a process or approach to creating those products, also called usercentered design 60

Project Management Body of Knowledge Project Management Institute http: //www. pmi. org/info/pp_pmb ok 2000 welcome. asp 61

62

Software Engineering Body of Knowledge www. swebok. org n n n Software configuration management Software construction Software design Software engineering architecture Software engineering management Software engineering process Software evolution and maintenance Software quality analysis Software requirements analysis Software testing ASSOCIATED DISCIPLINES n Cognitive sciences and human factors n Computer engineering n Computer science n Management and management science n Mathematics n Project management n Systems engineering 63

64

EPA Body of Knowledge Areas for Quality What you need to know to work or support quality in the EPA. n Quality Management Systems n Quality Project Planning n Data Quality Objective Planning n Quality Training n Science measures and associated metadata n Environmental methodologies n Concepts – hierarchial quality systems, graded approach…. Did I leave anything out? 65

Who owns a BOK? n Certification program n n n ASQ Certified Quality Engineer (CQE) has a CQE BOK, also for CQA, CQM, and CSQE All are owned by ASQ Professional organizations Government bodies Academic groups 67

How to structure a BOK for information and data quality? Focus on………… n The basics – the basic principles n The what of information and data – identify and describe them n The how of information and data – the processes n The who – what a information and data quality practitioner is and how they do their work 68

Basic structure n n n n n Overarching principles for information and data quality Communication 1: Basic terminology Communication 2: Alignment with the organization Information and data quality features, measures, and acceptance criteria Preliminary review of an organization’s information and data quality Managing and planning for information and data quality Implementing information and data quality Assessing information and data quality Tracking and reporting information and data quality Roles, skills, and job categories for information and data quality professionals 69

What are the quality principles for information and data? n general quality management principles apply to information, data, and information and data quality in the same way they apply to all basic management processes information has unique characteristics the bear on your ability to manage the quality information is a resource of the enterprise and the quality should be managed as any other resource information quality includes the quality of content, format, and n information quality includes: n n n functionality n n n information features (including functionalities) freedom from defect customer service effectiveness and efficiency you must be able to measure the information quality, in order to understand it completely, manage it, and to continually improve quality you must be able to track quality costs (failures, scrap and rework, etc. ) in order to effectively manage the resources used to ensure information quality 70

What is unique about information? …as a resource? n It cannot be used up n It can be copied n It can be shared across large distances n It can be moved easily n It is difficult to keep secure n It may become dated n Age may not affect its quality 71

What is unique about information? …as a product? n The same information can be provided/sold to more than one person n It can be delivered cheaply n Your product can be easily shared by the person who purchased it 72

Communication 1: Basic terminology Terminology needs for information and data quality n Recognition that terminology is discipline-specific n Recognition that there may be conflicting views on the definitions for certain terms (i. e. , information, data) n Multiple definitions may be needed for the same term n More complex definitions may be needed (data quality vs. data entry quality vs. data content transfer quality) n Need to include senior management terminology and align quality terminology with senior management terminology 73

Communication 2: Alignment to the organization n Methodology for tracking to performance measures for the organization Best practices for communicating information and data quality to senior management Models for relationship of information and data to the organization’s product 74

Information and data quality features, measures, and acceptance criteria n n Standardized list of features and functionalities Standardized measures and measurement units for the features and functionalities Suggested acceptance criteria Relationship between concepts and measures clearly mapped (access vs. usefulness vs. usability vs. integrity vs. security vs. transparency vs. objectivity vs. accuracy etc. ) 75

Preliminary review of an organization’s information and data quality n n Techniques to identify priorities for measurement in existing systems. Process to use existing available data about data to determine quality. Best practices in automated information and data quality measurement for existing data systems. Methods to review current quality measure in comparison to organization’s mission and goals. 76

Managing and planning for information and data quality n n n n Identification of standard types of products that should be subject to management and planning For each standard type of information product and data, a standard set of planning criteria Standard techniques for identifying the level of planning needed General planning guidance document. Process to identify goals for general information product and data quality projects. Process to identify acceptable measures to determine conformance with the goals. Processes to crosswalk management/planning to other project management initiatives in an organization Processes to crosswalk information product and data quality objectives to the organization’s performance measurement. 77

Implementing information and data quality n n n Identification of standard implementation phases Best Practices for implementing information and data quality initiatives. Example standard operating procedures (SOPs). 78

Assessing information and data quality n n Identification of hierarchy of assessment. (system vs. program vs. project vs. data system vs. data set vs. data field). Standard procedures for planning, implementing, and reporting assessment information. Suggested procedures for resolving corrective actions for existing data deficiencies. Types of assessments – based on existing BOKs, self-assessment, third-party, conformance, 79

Tracking and reporting information and data quality You can’t manage information and data quality without having the information to manage………. You must be able to track and report your information and data quality in terms of the organization’s performance goals……. (see COMMUNICATION 2 in the BOK) 80

Tracking and reporting information and data quality n n Methodology to identify what to report Standard reporting formats for information and data quality Routine reporting to middle and senior management Techniques to trend data and information quality to track improvement 81

Roles, skills, and job categories for information and data quality professionals n n Information and data quality managers Information and data quality assessors 82

Conclusion • We are already developing a body of knowledge in this area. • Structure to our BOK will help us communicate with each other. • It will help us better: • plan for information and data quality • measure information and data quality • establish organizational performance • align with organizational goal. • The job of quality professionals will be better planned and understood. 83

Resources for information quality & data quality & body of knowledge n n n n Larry English, www. infoimpact. com Dr. Tom Redman, www. dataqualitysolutions. com Michael Bracket Dr. Wang, MIT Data Quality Group American Society for Quality (ASQ) Information Integrity Group International Organization for Information and Data Quality (IAIDQ. org) Project Management Institute 84