Issues with Measures Precision Complexity Operations DOUG MAHONE

Issues with Measures – Precision, Complexity, Operations DOUG MAHONE JENNY ROECKS SEPTEMBER 25, 2014

Overview 2 �Key Questions q q q Policy Precision Complexity � Current Status of measure complexity � Is complexity meaningful and result in greater precision? � Challenges with complexity � Can/should complexity be reduced? � Principles for reducing complexity q Key Issue: Behavior �TF Discussion and Next Steps q q Subcommittee? Policy? Measure Complexity 10/28/2021

Policy Questions 3 �Cal TF focus? q q Continue to focus on work papers brought before the Cal TF, or Should we also proactively seek out issues to address (e. g. complexity in DEER) �How do we address bias in the estimates? q q q Should we bias toward conservatism? Should we bias toward optimism? Can we be explicit about sources of bias? �Cal TF operating procedures? q q Should we establish policies for precision, data, behavior, etc? Should we form subcommittees on these topics? Measure Complexity 10/28/2021

Key Questions: Precision 4 �What is “good enough” for savings estimates? q q q Is it a ratepayer protection issue? Is it a resource adequacy issue? How much effort is worth expending to get a better estimate? �How do we address precision? q q High precision for High Impact Measures? Lower precision for lower impact measures? �How do we deal with ranges in estimates? q q Can we explain that there’s a range? Do we need a point estimate? Measure Complexity 10/28/2021

Key Questions: Precision 5 �What do we mean by precision? q Measure granularity, calculation methods Measure Complexity Accurate & Precise Accurate & Not Precise Not Accurate & Not Precise 10/28/2021

Does more complexity give better accuracy? 6 �Improved accuracy with more measure combinations? q q Variance and uncertainty of data for measure calculations Do measure combination parameters reflect most impactful calculation/model variables �DEER data is mostly modeled, not empirical �DOE-2 has error band of +/- 10%, even in “model case” Measure Complexity 10/28/2021

Challenges with Measure Complexity 7 � In era of “big data, ” does complexity matter? � Challenges with complexity q Planning � q QA/QC � q Is meaningful QA/QC possible with so many measures? Updating Measures � q When you have 200 lines for the same measure, how do you plan? New weather files – every DEER measure had to be updated Transaction Side – Can programs collect information required to differentiate between measure combinations? Customers don’t understand distinctions; contractors may not be willing to collect � Midstream programs can’t collect � � Complexity makes it harder to replicate the database � POU TRM is simpler Measure Complexity 10/28/2021

Principles for Reducing Complexity 8 �Tornado diagrams / sensitivity analysis q Measure combinations should reflect most impactful variables �Appropriate use of measure combinations q q Values should differ by >10% Realistic to collect during program implementation �Engineering calculations instead of building models �Importance of statistical significance q q Data for measure development Data for updates Measure Complexity 10/28/2021

Key Issue: Behavior Data 9 �Many measure savings estimates depend on behavior: What do we assume? q q q How variable is the behavior? How much do we know about population variability? What is the resulting range of savings? �How much data is needed? q q q Are existing sources adequate? Must new data be collected? How much is good enough? �Should we develop policies? Measure Complexity 10/28/2021

Discussion and Next Steps 10 �TF Discussion q q Have other jurisdictions dealt with these issues? How forestall ED perception that “energy efficiency professionals have an optimistic bias”? �Next Steps q q q Who wants to participate in a subcommittee? Cal TF policies on precision, complexity, behavior, etc. ? Proposal: Process for developing policies � Cal TF staff will draft policies based on discussion � Subcommittee review/scrubbing � Draft policy back to full TF Measure Complexity 10/28/2021