Continuous Improvement in Energy Efficiency Helping Clients Shift

Continuous Improvement in Energy Efficiency: Helping Clients Shift from One-Shot Evaluations Toward Assessing the Effectiveness Within and Across Programs Marjorie Mc. Rae, Ph. D. , Principal Jane S. Peters, Ph. D. , President Research Into Action, Inc. , 503. 287. 9136

Energy Efficiency Programs • Programs that encourage people to: § Replace inefficient equipment prior to equipment failure § Purchase energy-efficient equipment § Operate equipment to satisfy human and process needs with minimal energy use § Maintain equipment at maximum efficiency § Build energy-efficient spaces (including renovations, remodels, and tenant improvements)

Continuous Improvement in Managing Energy Efficiency Programs: • …Is considered a best practice • …Necessitates recurring process evaluation • …Has been embraced by a few pioneering organizations • …Has been thwarted by short program cycles and utility regulatory requirements • …Has promise for states embarking on energy efficiency to meet long-term energy use reduction goals

Continuous Improvement as a Best Practice

Continuous Improvement Necessitates Recurring Process Evaluation • Questions relevant at pre-launch and within first six months: § What are the measurable goals? § Does the program logic support goal attainment? § Does data collection support measuring goal attainment? § Is the program being implemented as designed? § Any implementation difficulties? § Are key actors responding as anticipated?

Continuous Improvement Necessitates Recurring Process Evaluation cont. • Questions relevant at one year: § What are the goal measurements; is program on track to attain longer term goals? § What are the process steps; how long do they take; how many people are involved? § What are the “sticking points”, opportunities for improving efficiency? § What are the successes? § What are experiences of key actors? § What are the opportunities for improving effectiveness?

Continuous Improvement Necessitates Recurring Process Evaluation cont. • Questions relevant in subsequent years: § All the questions at previous junctions may be relevant, as indicated by the previous findings. § In-depth looks at: Ø Adequacy of assumptions about market response Ø Reasons for nonparticipation among end-users and trade allies Ø Barriers to repeat participation and larger projects among end-users & trade allies Ø Need for new interventions (training, marketing, networking, etc. )

CI Can Benefit from Process Evaluations Across Programs • Identify themes for program processes § Problems or opportunities common to multiple programs • Better understand markets § End-users can be targeted by multiple programs § Trade allies can support multiple programs

Continuous Improvement Through Process Evaluation • Has Been Embraced by a Few Pioneering Organizations § Northwest Energy Efficiency Alliance (NEEA) § New York State Energy Research and Development Authority (NYSERDA) § Energy Trust of Oregon

Characteristics of Organizations Pursing CI Through Process Evaluations • Management commitment to CI and its concomitant self-appraisal • Organizations geared to fundamental market change (“market transformation”) • Long program cycles—five and ten years • Freedom from typical regulatory constraints faced by utilities

Example 1 of CI Process Evaluation: Building Operator Certification (BOC) • NEEA accepted a proposal of Northwest Energy Efficiency Council (NEEC) to develop the BOC • Intention: to make the BOC self-sustaining • Program trains and certifies building operators in methods to attain energy-efficient high performance • We conducted seven evaluations over five years

Building Operator Certification: Continuous Improvement • Early evaluations addressed: § Quality of course content and teaching methods § Value of certification • Middle evaluations addressed: § Size of market § Appropriate price for the training § Exploration of methods to estimate energy savings • Later evaluations addressed: § Business model for self-sustaining § Estimation of average energy savings per participant

Example 2 of CI Process Evaluation: NYSERDA Photovoltaic Program • Intention: contribute to the development of a sustainable market for PV technologies • Program elements: § Incentives for PV installations § Requirement to use “eligible” installers § Support for the development of accredited PV training programs § Promoting and facilitating nationally recognized certification for PV installers § Providing business development and market support incentives for PV dealers and installers

Photovoltaic Program: Continuous Improvement • First evaluation addressed: § Program process steps and timelines from perspectives of all parties § Satisfaction of installers and end-users • Second evaluation addressed: § Response to a NYSERDA-sponsored RE/EE workforce education conference • Third evaluation addressed: § Relationship between PV installer workforce development activities and PV system outcomes

Example 3 of CI Process Evaluation: Energy Trust Production Efficiency (PE) • Intention: To increase the energy efficiency of industrial processes • Program elements: § Incentives for energy efficiency upgrades to industrial equipment and processes § Technical assistance from industrial process specialists § Dedicated marketing/customer support staff to develop long-term relationships and assist with program processes

Production Efficiency Program: Continuous Improvement • Sixth-month evaluation addressed: § Use of implementation contractor, with subcontractors, for program delivery § Quality of projects; energy impacts • First-year evaluation addressed: § Follow-up on problems identified in 1 st evaluation § Stimulation of process improvements § Progress toward three-year goals • Subsequent evaluations addressed: § Appropriateness of contracting approach § Savings realization rates by technology type

Barriers to Continuous Improvement Evaluation • Short program cycles – 1 to 3 years –focuses evaluations on outcomes, not processes • Lack of organizational commitment to energy efficiency and its continuous improvement • Regulatory indifference • Regulatory requirements focusing on outcomes • Regulatory directives on appropriate topics for evaluation (“micro-managing”)

Continuous Improvement Evaluation Has Promise • Many states are embarking on energy efficiency to meet long-term (10 year) energy use reduction goals § Michigan, Maryland, Ohio, Illinois, … • Aggressive goals go beyond “low hanging fruit; ” might not be attained without continuous improvement • Regulators need to provide encouragement and flexibility

Evaluators Can Educate Regulators and Utilities • Evaluators can educate regulators and utilities on: § Importance of continuous improvement for energy efficiency programs § Role of process evaluations in continuous improvement • Education through § Professional organizations § Public forums, public comment periods

Continuous Improvement in Energy Efficiency Marjorie Mc. Rae, Ph. D. , Principal marjorie@researchintoaction. com Jane S. Peters, Ph. D. , President Janep@researchintoaction. com Research Into Action, Inc. 503. 287. 9136