2015 Comprehensive LGP Process Evaluations Draft Research Plan

2015 Comprehensive LGP Process Evaluations Draft Research Plan with Common Scope of Work John Boroski and Tami Rasmussen, Evergreen Economics

Agenda • IOUs’ approach to staggered LGP Process Evaluations • Overarching guidelines for process evaluation research • LGPs being evaluated in Phase 1 • Common scope of work: Objectives, Methods • Research Team and Timeline • Q&A, LGP information needs • Next Steps 2

Process vs. Impact Evaluations Energy Division conducts impact evaluations • Impact evaluations identify and estimate energy and demand savings from a program. Program administrators may conduct process evaluations: • Process evaluation is a systematic assessment of an energy efficiency program: Ø To document program operations at the time of the examination. Ø To identify improvements that can be made to increase the program’s efficiency or effectiveness for acquiring energy resources. 3

Comprehensive and Staggered LGP Process Evaluations “Comprehensive”: Every LGP activity is documented and evaluated • “Targeted”: A subset of LGP activities are documented and evaluated. ‑ Research Into Action’s current process evaluation is targeted to municipal building retrofits and some Strategic Plan activities. • The current effort is a comprehensive process evaluation. • The last comprehensive process evaluations of LGPs were conducted during the 2006 -2008 program cycle. LGP evaluations will be staggered in phases • Work will be spread out over 4 -5 years, depending on EM&V budget. • Will achieve a census of comprehensive process evaluations. • LGPs through multiple IOUs will be studied once. IOU comprehensive process evaluations will use a common SOW • Ensures consistency across phases

Overarching Guidelines for Process Evaluation Research 1) Focus is on existing program activities only. 2) A logic model is a graphical mapping of key program activities, which are then linked to expected short-term outputs, and then longer-term outcomes. 3) The logic model process drives identification of program delivery needs and gaps, as well as innovative activities LGPs may be conducting. 4) Program implementation recommendations are only made when findings/data show there is an unmet program delivery need or gap – that is, outputs or outcomes are not occurring as planned. 5

Example Logic Model 6

LGPs in Phase 1 • SCG/SCE: Ø Ø Los Angeles County Riverside County San Bernardino County Valley Innovative Energy Watch (VIEW, with PG&E) • PG&E Ø Ø Association of Monterey Bay Area Governments (AMBAG) San Luis Obispo County (with SCG) San Mateo County Sierra Nevada • SDG&E Ø City of Chula Vista 7

Common SOW Research Objectives • Document current LGP activities. • Assess if each LGP is being implemented according to logic model. Ø Are any activities not occurring, or able to be improved? • Document LGP successes and challenges. • Gauge participant satisfaction with LGPs. • Assess if LGPs are on track to meet CPUC- approved program objectives. • Provide customized design/implementation recommendations for each LGP. 8

Methods Overview • Review and, if necessary, help each LGP update their logic model. • Review program documentation and regulatory reports, including marketing collateral promoting core programs. • Interview IOU program managers, support staff, third party implementers and vendors. • Interview participating jurisdiction implementation staff and key stakeholders. • Survey IOU customers/LGP participants. • (Potentially) conduct other activities as needed to address researchable items.

Review/Update LGP Logic Models Objectives: • Understand document program activities, expected outcomes. • Inform data collection. • Update models to reflect actual implementation. Approach/Issues: • Do updates after IOU staff interviews completed, documents reviewed. 10

IOU Staff Interviews Objectives (high level): • Confirm current program activities. • Assess implementation effectiveness, successes, challenges. • Confirm outcomes from LGP activities. • Solicit suggestions for program improvement. • Obtain program materials, participant samples. Approach: • IOUs provide contacts for LGP managers, supporting staff, implementation contractors. • In-depth phone interviews lasting 60 to 90 minutes. 11

Government Partner Interviews Objectives (high level): • Document success and challenges related to program activities, activity gaps and needs. • Confirm outcomes from LGP participation. • Get opinions on IOU services, support, coordination. • Solicit suggestions for program improvement. Approach: • IOUs provide LG partner contacts – who identify key support staff. • In-depth phone interviews lasting 60 to 90 minutes. Ø Follow up inquiries as needed 12

On-site Program Observations Objectives (high level): • Observe first hand how activities are conducted, services are delivered. • Document delivery logistics, event effectiveness, stakeholder engagement. • Document potential improvement areas. Approach: • IOUs and government partners suggest events, meetings worth attending; IOUs approve. • Evergreen spends 1 -2 days on-site documenting activities, effectiveness, key findings. 13

Participant Surveys Objectives (high level): • Document sources of program offering awareness. • Assess ease of program participation. • Assess satisfaction with LGP services, measures. • Solicit suggestions for improved participation. • Assess propensity for future participation. Approach: • Phone surveys lasting 15 minutes or less. • Target completes = 50 to 100 per customer segment, per LGP (TBD after participant samples provided). 14

Notes on Data Collection Tools • Interview guides, phone survey instruments developed after Research Plan finalized. • Evergreen will iterate with IOUs as needed to update and finalize draft guides and surveys. • Will develop an analysis plan for quantitative surveys. • Economies of scale assumed in RFP and work scope. • Need IOU approvals of “core” instruments that can be replicated. • May integrate a few LGP-specific questions. 15

Reporting Objectives/Approach: • Document everything the evaluators did and learned in one report for each IOU. Ø Separate sections for different LGPs • Identify program processes that are working well or need improvement. • Provide higher and lower priority recommendations that are actionable. Ø Who will do what, how and intended result • Give public presentation on draft reports and integrate feedback. 16

Research Team 17

Evaluation Schedule Task Start End Deliverables Task 1 A: Present Draft Research Plan June 13, 2016 June 24, 2016 Draft research plan posted, public webinar, meeting notes for SCG Task 1 B: Review and Update Logic Models July 2016 September 2016 Updated logic models (after IOU staff interviews) Task 2: Revise Research Plan June 27, 2016 July 18, 2016 Final research plan posted, research issues for future LGP studies documented Task 3: Develop Interview Guides, Survey Instruments and Analysis Plan July 2016 August 2016 Draft and final interview guides, draft and final survey instruments and analysis plans Task 4: IOU Program Manager Interviews August 2016 September 2016 Completed interviews, disposition memo Task 4: Local Government Partner Interviews August 2016 October 2016 Completed interviews, disposition memo Task 4: On-site Program Observations August 2016 November 2016 Event attendance, observations/findings memos; events TBD Task 4: Participant Surveys September 2016 November 2016 Sampling plans, disposition reports, topline reports Task 5: Reporting October 2017 December 2017 Draft report template, completed draft reports Task 5: Reporting January 2017 Revised draft report Task 5: Reporting TBD Public webinar Task 5: Reporting February 2017 Final report with documented comments

Q&A and Suggestions 1. Questions on research objectives or work scope? 2. Other process evaluation research issues to consider? • LGPs - What information do you need to help you implement your existing LGP activities more effectively? 19

Next Steps Please submit written comments, suggestions on Draft Research Plan at: • http: //energydataweb. com/cpuc • Search for “Comprehensive Process Evaluations of the PG&E, SCG and SDG&E Local Government Partnerships: First Wave. ” Comments accepted through June 27. Evergreen and IOUs will review comments and develop final research plan with updated schedule. Same core research plan will be used for each IOU/LGP. 20

Next Steps Note that: • Research issues that are out of scope for the LGP process evaluations will be documented for the Joint EM&V Long Term Research Plan and considered for future studies. • Upcoming ED-led LGP evaluation studies may be able to accommodate evaluation questions that do not address existing program activities. 21

Upcoming LGP Evaluations -- From the 2013 -2016 Energy Division & Program Administrator Energy Efficiency Evaluation, Measurement and Verification Plan, Version 6 22

More Questions? John Boroski, Evergreen Economics • boroski@evergreenecon. com 23