Using Logic Models in Managing Performance of Research

Using Logic Models in Managing Performance of Research and Technology Programs: An example for a Federal energy efficiency and renewable energy program IAMOT 13 th International Conference on Management of Technology April 4, 2004 Gretchen Jordan, Sandia National Labs John Mortensen, Independent Consultant John Reed, Innovologie George Teather, Independent Consultant Work presented here was completed for the U. S. DOE Office of Energy Efficiency and Renewable Energy by Sandia National Laboratories. Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy ( DOE) under Contract DE-AC 04 -94 AL 8500. Opinions expressed are solely those of the authors. April 4, 2004

Presentation Overview • Managers are under increasing pressure to demonstrate the value of their programs • Logic models can help programs identify and explain their value • Logic models can include strategies that represent research, development, demonstration and deployment (RDD&D) stages • Logic model example for federal energy program • Logic models help programs identify: – Indicators – Performance targets – Evaluation questions April 4, 2004 2

There is increasing pressure on RDD&D programs to demonstrate value U. S. Requirements: • Government Performance and Results Act of 1993 (GPRA) requires agencies to develop: – Strategic plans – Annual performance reports • OMB’s Program Assessment Rating Tool (PART) scores programs on: – Program purpose and design – Strategic planning – Program management – Program results April 4, 2004 3

Logic models help programs identify value • A logic model is a diagram or table describing how a program will solve identified problems • Elements of a logic model include: – Resources (Inputs) – Activities – Outputs – Customers reached – Outcomes • Short • Intermediate • Long-term – External influences April 4, 2004 4

Basic logic model structure Resources (Inputs) Activities Outputs for Customers Reached Short-Term Outcomes Intermediate Outcomes (through customers) External Influences and Related Programs (mediating factors) April 4, 2004 5 Long-Term Outcomes & Problem Solution

A logic model example • A logic model was developed for the U. S. Department of Energy’s Office of Energy Efficiency and Renewable Energy (EERE) • EERE’s goals are to: – Modernize energy conservation – Increase energy supplies – Modernize our critical energy infrastructure April 4, 2004 6

EERE has 7 strategies for achieving its goals 1. 2. 3. 4. 5. 6. 7. Plan and assess programs Develop and maintain program infrastructure Conduct research Develop technology Demonstrate technology Develop government and market infrastructure Deploy technology April 4, 2004 7

The 7 strategies cover stages in the RDD&D spectrum and are not necessarily linear R Program planning & assessment Develop & maintain program infrastructure Conduct research D D Develop technology Demonstrate technology Feedback Loops April 4, 2004 8 D Developing government & market infrastructure Deploy technology

The strategies are represented as “activities” in the draft logic model Inputs Activities Program planning & assessment Develop & maintain program infrastructure Conduct research Develop technology Feedback Loops Outputs For Outcomes April 4, 2004 9 Demonstrate technology Developing government & market infrastructure Deploy technology

EERE’s draft logic model shows how its strategies/activities are linked to its goals Inputs External Influences Federal, state & local government funding Private funding, Personnel, Facilities, Past R&D results Activities Program planning & assessment Develop & maintain program infrastructure Conduct research Develop technology Demonstrate technology Developing government & market infrastructure Deploy technology Quality of R&D proposals Feedback Loops Benefit estimates, Priorities identified, Budget requests, Program plans Public & private labs and test beds, Knowledge bases, Trained S&T personnel, Partnerships New knowledge, proof of concepts as represented by data, publications For Programs, CFO, OMB, Congress Programs, partners Outcomes Program funding in appropriate areas; Efficiency, Fiscal responsibility Relevant S&T expertise, capabilities and facilities to deliver programs Outputs Performance analysis Test, improve, & validate commercialscale technology, Give industry hands-on experience Codes and standards, Trained personnel, Audits tools, State programs Government purchases, Information disseminated, Early seeding of technologies R&D community R&D Community, Industry Relevant industries Relevant markets Potential purchasers Concepts & designs with possible applications, Knowledge spill-over Potentially commercializable technologies to replace existing or fill a system need Investment by industry in innovative or advanced commercial products Favorable policies, capable delivery channels for EERE products Widespread adoption of EERE products; More productive use of energy National R&D capabilities, including options if circumstances change Technology prototypes -initial -intermediate -commercial Spin-off products and their associated benefits 10 Unpredictable nature of R&D Cost and performance of competing technologies Industry willingness to take risk Energy prices New products & businesses April 4, 2004 Political environment Economic, security, and environmental benefits Technology leadership State of the economy Government policies and regulations

Logic models help programs identify: • Measurement areas for which indicators (metrics) may be developed • Performance targets for each indicator • Evaluation questions April 4, 2004 11

Each box in the logic model is a potential measurement area Inputs Federal funding (millions of nominal $) Private funding (millions of nominal $) Activities Program planning & assessment Develop & maintain program infrastructure Conduct research Develop technology Federal personnel (FTEs) # of RD&D facilities Demonstrate technology Developing government & market infrastructure Cost and performance of competing technologies (varies by technology) Deploy technology Outputs For % program w/program plans Programs, CFO, OMB, Congress Outcomes # of partnerships Programs, partners # of journal articles # of presentations # prototypes -initial -intermediate -commercial Prototype cost & performance # and % of commercialscale technologies validated # codes and standards, # personnel trained, # audits, # state programs # of tech’s purchased by gov’t, # of materials disseminated, # of website hits R&D community R&D Community, Industry Relevant industries Relevant markets Potential purchasers # journal article citations # of potentially commercializable technologies # of innovative or advanced commercial products with improved cost & performance # of recommendations for using advanced commercial products and practices # and % of advanced commercial products by adoption stage # of technology spinoffs April 4, 2004 12 Oil prices ($/barrel) NG prices ($/tcf) Feedback Loops % programs w/benefit estimates External Influences Energy saved (quad. Btu), GW of add’l RE capacity, Expenditure savings ($) Carbon saved (mmtce) Electricity prices (c/k. Wh) Coal prices ($/ton) GDP (billion 1996 $) RE production tax credit (c/k. Wh) EE/RE tax credits ($) CAFÉ standards (mpg) Vehicle & power plant emission standards (varies by pollutant)

Performance targets may also be developed for each box in the logic model Inputs Federal funding (millions of nominal $) Private funding (millions of nominal $) Activities Program planning & assessment Develop & maintain program infrastructure Conduct research Develop technology Federal personnel (FTEs) # of RD&D facilities Demonstrate technology Outputs For % program w/program plans Programs, CFO, OMB, Congress Outcomes # of partnerships Programs, partners # of journal articles # of presentations # prototypes -initial -intermediate -commercial Prototype cost & performance # codes and # and % of standards, commercial# personnel Renewables scale trained, technologies # audits, validated # state programs 5. 00 3. 00 # of tech’s purchased by gov’t, 6. 20 5. 30 # of materials disseminated, # of website hits R&D community R&D Community, Industry Relevant industries Relevant markets Potential purchasers # journal article citations # of potentially commercializable technologies # of innovative or advanced commercial products with improved cost & performance # of recommendations for using advanced commercial products and practices # and % of advanced commercial products by adoption stage # of technology spinoffs April 4, 2004 Cost and performance of Cost of Hydrogen competing ($/gallon gasoline equivalent) technologies (varies by technology) Developing government & Deploy 2003 2006 2010 Oil prices market technology ($/barrel) infrastructure Nonrenewables Feedback Loops % programs w/benefit estimates 13 External Influences Energy saved (quad. Btu), GW of add’l RE capacity, Expenditure savings ($) Carbon saved (mmtce) NG prices ($/tcf) 1. 50 Electricity prices (c/k. Wh) 3. 90 Coal prices ($/ton) GDP (billion 1996 $) RE production tax credit (c/k. Wh) EE/RE tax credits ($) CAFÉ standards (mpg) Vehicle & power plant emission standards (varies by pollutant)

Arrows between the boxes help identify evaluation questions Evaluation Questions: Activities Demonstrate technology Outputs Test, improve, & validate commercialscale technology, Give industry hands-on experience For Outcomes April 4, 2004 • Has EERE engaged the right partners in tests and done it efficiently? • Have relevant industries gained hands-on experience with the technologies? • Has industry experience lead to investment in these innovative or advanced commercial energy products? Relevant industries Investment by industry in innovative or advanced commercial products 14 • Have EERE’s efforts to test and demonstrate energy technologies led to validated commercial scale technologies? (Activities to outcomes)

Potential management / evaluation questions for EERE ask if strategies are working 1. Have program planning and assessment activities increased potential program benefits per federal dollar spent? 2. Has developing and maintaining EERE program infrastructure provided the scientific and technical expertise and facilities required to carryout program activities? 3. Has research conducted by EERE yielded energy-related concepts and designs with possible commercial applications? April 4, 2004 15

Management / evaluation questions (cont. ) 4. Has EERE technology development yielded potentially commercializable energy technologies? 5. Have EERE efforts to test and demonstrate energy technologies led to investment by industry in these innovative or advanced commercial energy products? 6. Have EERE efforts to develop government and business infrastructures led to favorable policies and capable delivery channels for EERE products and practices? April 4, 2004 16

Management / evaluation questions (cont. ) 7. Have EERE deployment activities with end users increased the awareness, appreciation, and adoption of EERE products and practices? And, altogether has the EERE portfolio of RDD& D programs led to adoption of EERE products and practices and yielded economic, security, and environmental benefits that would not have occurred otherwise? April 4, 2004 17

Evaluations can explain why some goals were met and others were not Inputs Federal funding (millions of nominal $) Private funding (millions of nominal $) (1) Funding was about Activities what was expected Program planning & assessment % programs w/benefit estimates Outputs For Develop & maintain program infrastructure # of partnerships Conduct research # of journal articles (2) R&D yielded expected results % program w/program plans Programs, CFO, OMB, Congress Outcomes Programs, partners # of presentations Develop technology Federal personnel (FTEs) # of RD&D facilities (3) But, competing technologies improved more than expected Demonstrate technology Developing government & market infrastructure # prototypes -initial -intermediate -commercial Prototype cost & performance (4) Energy prices were lower than expected # and % of commercialscale technologies validated # codes and standards, # personnel trained, # audits, # state programs # of tech’s purchased by gov’t, # of materials disseminated, # of website hits R&D community R&D Community, Industry Relevant industries Relevant markets Potential purchasers # journal article citations # of potentially commercializable technologies # of innovative or advanced commercial products with improved cost & performance # of recommendations for using advanced commercial products and practices # and % of advanced commercial products by adoption stage (5) Thus, market penetration was slower than expected # of technology spinoffs 18 Cost and performance of competing technologies (varies by technology) Oil prices ($/barrel) NG prices ($/tcf) Feedback Loops (6) And benefits were lower than expected April 4, 2004 Deploy technology External Influences Energy saved (quad. Btu), GW of add’l RE capacity, Expenditure savings ($) Carbon saved (mmtce) Electricity prices (c/k. Wh) Coal prices ($/ton) GDP (billion 1996 $) RE production tax credit (c/k. Wh) EE/RE tax credits ($) CAFÉ standards (mpg) Vehicle & power plant emission standards (varies by pollutant)

In summary, logic models for RDD&D programs help identify… Inputs Federal, state & local government funding Private funding, Personnel, Facilities, Past R&D results Activities Program planning & assessment Develop & maintain program infrastructure Indicators Conduct research Develop technology Performance Targets Demonstrate technology Feedback Loops • Cost of Hydrogen ($/gal gasoline equivalent) Public & Benefit New private labs estimates, knowledge, –Non-renewables and test beds, Priorities proof of Knowledge identified, concepts as Outputs –Renewables bases, Budget requests, Program plans Trained S&T personnel, Partnerships represented by data, publications Evaluation Questions For Programs, CFO, OMB, Congress Programs, partners Program • Did development activities lead to Relevant S&T funding in expertise, potentially commercializable appropriate capabilities Outcomes areas; technologies? and facilities to Efficiency, Fiscal responsibility deliver programs Technology prototypes -initial -intermediate -commercial Performance analysis Test, improve, & Codes and validate standards, commercial. Trained scale Nonpersonnel, technology, renewables Audits tools, Give industry hands-on State programs experience Renewables 2003 2006 Government purchases, Information 5. 00 3. 00 disseminated, Early seeding of technologies 6. 20 5. 30 R&D Community, Industry Relevant industries Relevant markets Potential purchasers Concepts & designs with possible applications, Knowledge spill-over Potentially commercializable technologies to replace existing or fill a system need Investment by industry in innovative or advanced commercial products Favorable policies, capable delivery channels for EERE products Widespread adoption of EERE products; More productive use of energy Spin-off products and their associated benefits 19 2010 Unpredictable nature of R&D Cost and 1. 50 performance of competing technologies 3. 90 Industry willingness to take risk Energy prices New products & businesses April 4, 2004 Developing government & of Hydrogen Deploy Cost Political market technology environment ($/gallon infrastructure gasoline equivalent) Quality of R&D proposals R&D community National R&D capabilities, including options if circumstances change External Influences Economic, security, and environmental benefits Technology leadership State of the economy Government policies and regulations

… and tell the performance story External Influences Inputs Federal, state & local government funding Private funding, Personnel, Facilities, Past R&D results Activities Program planning & assessment Develop & maintain program infrastructure Conduct research Develop technology Demonstrate technology Developing government & market infrastructure Deploy technology Quality of R&D proposals Feedback Loops Benefit estimates, Priorities identified, Budget requests, Program plans Public & private labs and test beds, Knowledge bases, Trained S&T personnel, Partnerships New knowledge, proof of concepts as represented by data, publications For Programs, CFO, OMB, Congress Programs, partners Outcomes Program funding in appropriate areas; Efficiency, Fiscal responsibility Relevant S&T expertise, capabilities and facilities to deliver programs Outputs Performance analysis Test, improve, & validate commercialscale technology, Give industry hands-on experience Codes and standards, Trained personnel, Audits tools, State programs Government purchases, Information disseminated, Early seeding of technologies R&D community R&D Community, Industry Relevant industries Relevant markets Potential purchasers Concepts & designs with possible applications, Knowledge spill-over Potentially commercializable technologies to replace existing or fill a system need Investment by industry in innovative or advanced commercial products Favorable policies, capable delivery channels for EERE products Widespread adoption of EERE products; More productive use of energy National R&D capabilities, including options if circumstances change Technology prototypes -initial -intermediate -commercial Spin-off products and their associated benefits 20 Unpredictable nature of R&D Cost and performance of competing technologies Industry willingness to take risk Energy prices New products & businesses April 4, 2004 Political environment Economic, security, and environmental benefits Technology leadership State of the economy Government policies and regulations

For more information contact: Gretchen Jordan Sandia National Laboratories 950 L'Enfant Plaza, SW Suite 110 Washington, DC 20024 -2123 Phone: (202) 314 -3040 Email: gbjorda@sandia. gov John Mortensen: John Reed: George Teather: April 4, 2004 21 jmort@prodigy. net jreed@innovologie. com gteather@sympatico. ca