Skillbuilding Workshop Using Logical Framework to Identify Outcomes

Skill-building Workshop Using Logical Framework to Identify Outcomes and Develop Performance Indicators in S&T Program Proposals Shan Li & Ling-Chu Lee Science & Technology Policy Research and Information Center AEA annual conference 2011/11/03 1

Two parts of the workshop • Introduction on the steps of the simplified LFA by explaining the case (15 mins. ) • Hand-on practice (30 mins. ) 2

Importance of Ex-ante evaluation The influence of evaluation is to make governments’ resources allocation effective and efficient. How to present the “significant” outcomes and develop the “suitable” indicators is very important for resources allocation. The aim of a “good ex-ante evaluation” is to request the well-built framework and present the kind of logical thinking between the goals and indicators in programs’ proposals. 3

Taiwan’s situation In the 2010 Survey of Government Strategy for Technology Development finds that 31. 9% of respondents believe that the objectives and indicators in government plans are not specific enough (National Science Council, 2010). Government departments are not explicitly defining objectives in their plans, or specifying suitable outputs and outcome indicators so as to reflect the return on the resources invested. 4

The simplified LFA To present the logical relationship between problems, goals/objectives, and indicators Fishbone diagram Simplified LFA SWOT analysis (NORAD) Logical Framework Approach (LFA) Taiwan’s S&T program evaluation framework 5

Objectives of the workshop 1 2 How to identify the problem; How to develop the problem tree and objective tree by using the concept of fishbone diagram; 3 To understand the importance of SWOT analysis in the application of simplified of logical framework; 4 5 How to develop the logframe matrix through the above steps; To understand how to identify outcomes and develop performance indicators. 6

Steps of the simplified LFA Thinking spindle: clarify problem→define the resources→develop solutions→design indicators 1 2 Problem Analysis 3 Strategy Analysis 4 Objective Analysis Development of LFA matrix Logical thinking of S&T programs 7

CASE ：an agency’ program of MOEA(Ministry of Economics Affaires) Thinking spindle: clarify problem→define the resources→develop solutions→design indicators *The content is only provided for the demonstration of the simplified LFA, not representing the position and implementation of Taiwan government agencies’ programs. 8

The agency’s mission • full support to the development of a first-class investment environment; • providing comprehensive assistance to business enterprises to overcome investment obstacles; • providing guidance for the development of new industries as well as the upgrading and transformation of existing industries, so as to lay a firm foundation for long-term national development. • will continue to work closely with the industry to jointly create a new chapter for Taiwan’s industrial development. 9

STEP 1 Problem analysis 1. Identification of the major problems faced by beneficiaries; and 2. Development of a problem tree to establish causes and effects. 10

Definition of the core problem Ideal deviation Status past present Problem

TIPS The classification of problems in S&T programs • Necessary problems (Must) – it is not possible to measure the importance or priority of this kind of problem, its primary goal is its own and not negotiable. Therefore, in discussing this kind of problems, resources are not used to decide strategies, but first to decide the best strategy. • Problems available to be solved with discretion (Desirable) – When some resources remain available after solving the necessary problems, such problems can be analyzed and further sorted for coming up with the one with the highest priority among them which needs to be solved. • Problems available to be solved incidentally (Feasible) – With not much effort needed to solve the problem, which isn’t necessarily related to the objective, but just like taking a free ride. 12

Problem tree To construct the relationship between cause and effect Sub problem 3 Sub problem 1 Core Problem Itemized Problem A Itemized Problem B Detail of the Itemized Problem B Sub problem 4 Sub problem 2 It is required to follow the principles of “mutual Exclusiveness” and “collective exhaustiveness” 13

Decomposition of problems—the principle of MECE • Principle of MECE in problem decomposition: The use of Mc. Kinsey & Company Minto Pyramid Principle in the principle of MECE • MECE has stressed that when facing a conceptual things or idea that needs to be split, sequence in line with structured logic should be used for the classification. • Two terms are required, namely – Mutual exclusiveness – Collective exhaustiveness 14

TIPS Reference on dimension of sub problems The M’s The P’s (Service Industry) The S’s (Service Industry) PEST Analysis Machinery (Equipments) Machinery / location Environment Politics Method (Progress) Progress Supplement Economy HR (labor) Labor System Society Material Policy Skills Technique Natural (environment) Step Management (policy) Price Management Promotion Maintenance Product Market (promotion) 15

TIPS Narrative approach of problems • The narrative of the core problem of is negative thinking. • Narrative form: adjective + noun for the sentence structure, and ought to be a negative narrative. • Every dimension of a Problems Tree is likely to become a single Objectives Tree. 16

Construction of the Problems Tree Production cost too high Lack of Innovation Low R & D investment Increasing of labor hours, labor-intensive Requirement of a larger area Fixed production line Technology-based thinking Declining competitiveness Low degree of internationalization Long product life cycle of the traditional industries WTO opening mechanism Lack of industrial competitiveness 1. Select the dimensions of sub problems 2. Think itemized problem 3. Narrative form: adjective + noun for the sentence structure, and ought to be a negative narrative 4. Clarify the cause-effect relationship 17

STEP 2 Strategy analysis 18

SWOT analysis The SWOT analysis provides information on positive and negative aspects both within and outside the project. Favorable to the achievement of objectives Unfavorable to the achievement of objectives Internal environment Strengths S Weaknesses W External environment Opportunities O Threats T 19

TIPS SWOT analysis ive bject u s l na inter factors the e by l b a l ol s contr anization • org • • • 1. SWOT analysis responding to core problem 2. To make an inventory of organization or programs themselves Strengths Weaknesses Quality of staff and management Quality of services Range of services and referral system Internal procedures and mechanisms • • • exte Opportunities • • • Funding opportunities Policy directions National programmes Change in mandate Trends in society Gaps/overlap in service delivery or target group coverage by other providers Weaknesses of organisations outside the project Access to means (finance and technology) External coordination Source: NLR. (2007). Organization structure Internal coordination Financial management rna fact l objec tive con ors the trolla not b org ani le by Policy directions zati ons Decrease in subsidies or other Threats • • funding Increasing demand for other services Other organisations providing the same services Developments in society that increase the leprosy burden Changes in donor priorities Decrease in resources (staff, inputs) 20

SWOT matrix Strategic option: First explain that the focus of the group is now changing from analysis into action and that for effective actions we will need to have a vision about where we wish to go. Inner Analysis Opportunity (O) Weakness（W） So Strategy (Max-Max) Strengthen advantages Exploit opportunities WO strategy (Min-Mix) Reduce threats, Exploit opportunities Threat (T) Outer Analysis Strength（S） ST Strategy (MAX-Min) Strengthen advantages Reduce threats WT Strategy (Min-Min) Lower threats, Reduce disadvantages source：Weihrich, 1982 21

TIPS SWOT matrix—strategic thinking Successful strategies build upon strengths and takes advantage of opportunities, while it overcomes or minimizes the effects of weaknesses and threats. Inner Analysis Strength Weaknesses Outer Analysis Opportunity Threat Will this strength help us to make use of this opportunity? Will this strength help us to ward off this threat? Does this weakness prohibit us to make use of this opportunity? Does this weakness prohibit us to ward off this threat? Source: NLR. (2007). 22

SWOT Analysis Strength 1. 2. 3. 4. Accumulate abundant manufacturing energy, technical competence and management experience Industrial structure with a tight integrity; SMEs with a flexible organizational structure Mature industrial technology will help quickly develop heterogeneous products and promote Industry - university - academy joint R & D cooperation Government policy support, and list subsidies for traditional industries as a policy program Weakness 1. Emigration of production base and value chain 2. Difficult to Finance 3. Low percentage of total R & D personnel into the traditional industries 4. difficult to obtain key The key technologies 5. Lack of R & D strategic planning 6. Lack of independent R & D energy 7. Lack of clustering energy of traditional industries and hence difficult to produce overall performance 8. Lack of product competitiveness of traditional industries Opportunity Threat 1. With a sound innovation infrastructure, strengthening the link between traditional industries would further improve the innovation capability of our country 2. Increasing market demand for products with a larger variety and low in quantity to create new business opportunities for the industries 3. A substantial increase in market demand due to the steady global economic growth 4. Traditional industries facing global competition and the formation of global expansion and brand competition after Taiwan’s accession to WTO 1. Market magnetic effect of the mainland China and the dumping of low-priced products 2. Emerging developing countries absorb low-cost resources for production to attract the emigration of traditional industries, resulting in the gradual disintegration of industry clusters and the OEM model facing a bottleneck situation 3. Domestic market is facing the pressure of foreign competition after Taiwan's accession to WTO 4. Rise of raw material prices caused by the global inflationary pressure is threatening the competitiveness of industries 5. Competition of cheap products produced by the emerging developing countries 23

SWOT Matrix Internal Analysis SWOT Matrix 1. Opportunity 2. External Analysis 3. Strength Weakness SO Strategy WO strategy Provide R & D subsidies to encourage traditional industries to be actively involved in research & development and innovation. Promote university-industry joint R & D cooperation to help traditional industries update their products corresponding to the market demands. Combine correlated key industries in joint development to expand the industrial chain synergy. 1. 2. 3. ST Strategy 1. Threat 2. Strengthen the leading technology of the traditional industries, with the full effectiveness of industry clusters, to obtain competitive advantage through high-quality products. Make use of R & D Program and government resources to enhance the industrial added value, free from lowcost competitive pressure and establish innovative products with differentiation. Provide subsides for R & D research to assist the industries in obtaining R & D resources, share research & innovation risk and improve the added value of the products. Help traditional industries with R & D, positive transformation, attracting talented people and capital inflows. Introduction of design aesthetic to the traditional industries for promoting the uniqueness and differentiation of the new products. WT strategy 1. 2. Enhance R & D capabilities of the traditional industries and create product differentiation for improving the competitiveness of the traditional industries. Apply the capabilities of Flexibility and quick response to the market demands of our industries, combined with the innovation energy which guides the traditional industries for maintaining the 24 leading position.

STEP 3 Objective analysis 25

Objective tree To determine the relationship of means and methods. Objective 1 Objective 3 Goal Sub Objective A Sub Objective B Itemed Objective 2 Objective 4 26

TIPS Narrative approach of objectives • Method of objective presentation: statement structure of the Objective Tree: is almost equivalent to that of the Problems Tree, except converting the opposite viewpoints of the Problems Tree into positive objective statements. – Narrative: verb + object (person / thing / object) for sentence structure, and use a positive statement. – Statement of principles: corresponding indicators can be developed for measuring the objectives. 27

Construction of the Objectives Tree Global division of labor Establishment of Joint Development Model Vertical Integration Independent R&D Development led by the key industries Improve the competitiveness of the traditional industries Combination with design services Transfer of hightech industry experience Establishment of intellectual property and accounting systems SO 2 Industry-university-academic joint development Innovation of Industrial production technology SO 3 R & D subsidies provision Enhancement of product uniqueness & differentiation Improvement of product competitiveness 1. Goal and objectives are transferred from the problem tree 2. Itemized objectives come from SWOT matrix 3. Narrative: verb + object (person / thing / object) for sentence structure, and use a positive statement. 28

STEP 4 Development of LFA matrix 29

LFA matrix Content Goal (impact) Objective (outcome) Output Activity Input Indicators Measure the contribution of the programs to the goals Sources of verification Assumption Methods for measuring the contribution Factors for the definition Information required for of achieving the presenting the outcomes objectives Quantitative presentation for the outputs List of the outputs for the purpose Program Whether the activities are necessary and sufficient for producing the outputs? Resources required for implementing the activities Whether the outputs are reasonable and in line with the activities? Whether the preconditions being relied on are reasonable? 30

TIPS Assumptions are statements about external factors influencing the project positively or negatively, but are outside the project’s control. These project risks are to be formulated during project planning and monitored throughout implementation. Assumptions are stated as positive conditions. Source: NLR. (2007). 31

Development of LFA matrix -vertical logic + horizontal logic. Items Indicators 14 Sources of verification 15 Assumption 1 Goals 13 1 2 Objectives 2 14 15 10 11 7 8 13 9 9 3 Output Input 3 7 4 5 Activity 6 6 8 Vertical logic Resource Horizontal 32

TIPS Transformation of outputs to outcomes • Outputs: quantitative items of programs • Outcomes: must consider “anticipated changes of target population” • “anticipated changes of target population”: knowledge, skills, attitudes, and behaviors – Example: Output: papers Target: researchers Outcomes: citation of papers 33

TIPS Types of indicators Direct outcomes: change of behavior or concept of a target objective due to output Output: materials or things created by project activities • • Paper Technology Patent Teaching material Database Information System Academic Activities • • • Theoretical applications Financial intermediation Cost reduction Industry-University mode Market development Talent Training Knowledge diffusion • • • Paper citation • Patent purchase • Technology application material application • database application • Knowledge diffusion • Decision making Tool Application Business model Increased investment Industry-University Cooperation R & D team New Product New Service Socio-economic impact: social and economic side effects arising from the proliferation of indirect benefits Indirect outcomes : Changes created by the diffusion, extension and integration of direct benefits • • Technical Service Utility method Investigation report analysis report Benchmark/norm Policy recommendation Competitiveness enhancement Extended value Service Innovation Knowledge Fusion Decision-making results • • Value change Improved talent quality Increase in employment • Improved efficiency • Development of new industries Improved quality of life • Efficient use of resources Balanced distribution of benefits Implementation of environmental safeguard Improvements in global issues Benefits of the policy implementation 34

LFA matrix Item Goal Objective Indicator Sounes of verification Assumption Better market 1. Increase the added value of 1. calculated by EVA definition acceptance of product Enhance the enterprises ($ billion) 2. Domestic market is defined diversification competitiveness of the 2. Domestic product market share (%) as xx strategy with the traditional industries 3. Global trade market share (%) 3. Global trade is defined as xx increase of personal disposable income 1. Goals come from goal in objective tree 1. Establish product differentiation 2. Reduce production cost 1. Create output performance ($ billion) 2. Reduce production cost ($ billion) 1. Total corporate revenue Successful integration 2. Based on enterprise gross of key technologies profit margin* revenues as and patents subsidies 2. Objectives come from objectives in objective tree Output Input 1. Calculation based on the R & D budgets of company financial statements 2. Number of Patents approved 1. Industry R&D funding ($ billion) by R. O. C or the United States Promote industry 2. Number of corporate patents Sufficient supply of R 3. Number of Industry 3. Number of joint development cases investment in R & D talents university-academic 4. Number of technology transfer activities development cases 4. Consideration relationship obtained in technology transfer from university-academy to 3. Outputs come from itemized objectives in objective tree industry Subsidy case**～**（ name of a case） 1. Input Funding ($ thousand/4 years) 2. Successful integration of key technologies and patents 4. Assumption is the necessary condition of successful program 35

Hand-on practice The procedures of the session are: 1 2 3 4 5 The problem identification (5 mins); The development of the problem tree (5 mins); The application of SWOT analysis (5 mins); The identification of the objective tree (5 mins); The development of the logframe matrix (5 mins); 36

Requirements • Open atmosphere for communication • Free talk without any rules or disturbances 37

Any issues you can choose… • The topics on aging population – health issues: These include hearing and sight loss, memory loss and increased health problems. – elderly person's income and economic welfare: No longer being in the workforce, these individuals will need to rely on their pensions and Social Security. – Loneliness: While some children are a short distance from older parents, some live too far a distance away to provide the proper support in emergencies. – ……OR you can choose any issues which happen in your life or your job 38

2 Template for problem tree cluture difference Young childre n are work outside Middle class don’t have much money to pay insurance 39

3 Template for SWOT analysis 40

Template for SWOT matrix 41

4 Template for objective tree Increase vareities of insurance Increase The salary of middle class 42

5 Template for LFA matrix Increase The salary of middle class 43

References • NORAD (1999). The logical framework Approach: handbook for objectives-oriented planning. • Netherlands Leprosy Relief (NLR). (2007). Guidelines for Logical Framework Planning Workshops. • Thailand Bureau of the Budget. (2004). The Logical Framework Approach - a Step-by-Step Guide to Objective. Oriented Project Design. PPD Refreshers Workshop. • Y. C. Shiue. (2008). Strategy Planning and Management. Taipei: Yeh Book Gallery. 44

Thank you for Attention! Shan Li, ssli@stpi. narl. org. tw Ling-Chu Lee, lclee@stpi. narl. org. tw