IFI 8109 Planning and Running Research and Development

  • Slides: 39
Download presentation
IFI 8109 Planning and Running Research and Development Projects Evaluation of R&D-projects Peeter Normak

IFI 8109 Planning and Running Research and Development Projects Evaluation of R&D-projects Peeter Normak 1

Plan 1. Homework: evaluation of the project proposal “The framework for supporting and analysing

Plan 1. Homework: evaluation of the project proposal “The framework for supporting and analysing self-directed learning in augmented learning environment”. 2. The objectives of evaluation (reviewing) of R&D project proposals. 3. Critical aspects in evaluation of R&D projects: • • The purpose (objective) of the project Methodology Available resources Sustainability 2

Homework: evaluation of the project proposal “The framework for supporting and analysing self-directed learning

Homework: evaluation of the project proposal “The framework for supporting and analysing self-directed learning in augmented learning environment” 3

Grantitaotluse hinnang 1. 2. 3. 4. 5. 6. 7. 8. 9. Scientific quality and

Grantitaotluse hinnang 1. 2. 3. 4. 5. 6. 7. 8. 9. Scientific quality and innovativenss of the problem setting. Competency of the Principal Investigator and the team. Actuality of the subject. Project elaboration and justification of the methodology. The involvement of graduate students. The strengths of the application. The weaknesses of the application. Comments. Overall assessment of the application (is not necessarily the arithmetic average of the previous scores). 4

Scientific quality and innovativenss of the problem setting 1. Scientifically acceptable in every sense,

Scientific quality and innovativenss of the problem setting 1. Scientifically acceptable in every sense, an international level project proposal. 2. Importance in education could actually transcend national boundaries. 3. Essentially, it is a design-based research in a topical research area – a physical learning environment that is augmented by a virtual one. Score: 4, 5. 5

Competency of the Principal Investigator and the team 1. Highly competent team. 2. There

Competency of the Principal Investigator and the team 1. Highly competent team. 2. There exist relevant international co-operation. Score: 3, 5. 6

Actuality (topicality) of the subject The subject is clearly highly topical. Score: 4, 5

Actuality (topicality) of the subject The subject is clearly highly topical. Score: 4, 5 7

Project elaboration and justification of the methodology 1. In spite of some generality of

Project elaboration and justification of the methodology 1. In spite of some generality of the description, the methodology seems to be appropriate and reasonable. 2. The research tasks are well presented and are able to direct adequately the work. Score: 3, 5 8

The involvement of graduate students Involvement of graduate students in the project is a

The involvement of graduate students Involvement of graduate students in the project is a strengths of the proposal. Score: 5 9

The strengths of the application 1. The international level, an important topic. 2. Design-based

The strengths of the application 1. The international level, an important topic. 2. Design-based research has a strong performance potential in the field of technological and methodological innovation. 3. Graduate students' involvement and good potential for sustainable development. 10

The weaknesses of the application 1. There were not enough hypotheses-type research tasks. 2.

The weaknesses of the application 1. There were not enough hypotheses-type research tasks. 2. Contingent with whom the tests are conducted, is not specified. 3. The locations where the research will be conducted is not specified. 11

Comments 1. The foreign reviewer is favorably optimistic. 2. The local reviewer proposes to

Comments 1. The foreign reviewer is favorably optimistic. 2. The local reviewer proposes to fund the proposal in full (as requested). 12

Overall assessment of the application (is not necessarily the arithmetic average of the previous

Overall assessment of the application (is not necessarily the arithmetic average of the previous scores) 5 - Outstanding (is among the best 10% of applications); 4 - Very Good (falls among the best 25% of applications); 3 - Good (among the best 50% of applications); 2 - Satisfactory (is among the best 80% of applications, the funding is not feasible); 1 - Weak (belongs to the 20% weakest applications, certainly not to finance). Score: 4, 0. 13

Why this example? 1. It is interdisciplinary (ICT + education). 2. The scope of

Why this example? 1. It is interdisciplinary (ICT + education). 2. The scope of the project was very broad (learning takes in today’s society mostly place in an augmented learning environment). 3. The threshold for being financed was exactly 4, 0. 14

Discussion What we learned from this example? 15

Discussion What we learned from this example? 15

The objectives of evaluation (reviewing) of R&D project proposals 16

The objectives of evaluation (reviewing) of R&D project proposals 16

(Peer-)Reviewing of research proposals The objective of peer-reviewing is to evaluate and rank the

(Peer-)Reviewing of research proposals The objective of peer-reviewing is to evaluate and rank the project proposals, that would allow the decision-makers to conclude on funding. Method of rating: parametric (fixed or non-fixed weights). The basic budgeting methods: • Formula based • Application-based (with possible corrections to individual items or activities) • Proportional (applying the same or decreasing coefficients) • A combined method. 17

Reviewing the final report The objective: • Evaluation the quality and quantity of the

Reviewing the final report The objective: • Evaluation the quality and quantity of the outcome • Assessment of the compliance to initial project proposal. Method of evaluation: parametric (fixed or non-fixed weights) as for evaluation of the project proposals. Indirect objective (targeted mainly to the applicant): to provide input for composing possible follow-up project proposals. 18

Critical aspects in evaluation of R&D projects 19

Critical aspects in evaluation of R&D projects 19

Project objective is the key factor that determines planning (and execution) of a project.

Project objective is the key factor that determines planning (and execution) of a project. The general SMART requirements are applicable for objectives: 1. 2. 3. 4. 5. Specific – states the exact purpose of the project. Measurable – concrete criteria for measuring progress/outcome. Agreed (attainable) – stakeholders recognize its importance. Realistic (relevant) – enough resources are available. Time-bound (timed) – grounding within a time frame. 1. The weight of these requirements are quite different for R&D projects. 20

Critical aspect 1 – the objective of a project 1. Target level (compared to

Critical aspect 1 – the objective of a project 1. Target level (compared to the projects done elsewhere). Example: 2. Measurability (clearly defined) and realistic. Examples: 3. Justification (importance), including: • • • Relevance of the literature cited in justifying the research objectives Clarity of research questions and/or hypotheses Description of theoretical framework What the study supports and for whom it is important To what extent the study raises new questions, or opens up new research directions. 21

Analysis of the project objective Forward looking! Analysis of possible trends. Taking into account

Analysis of the project objective Forward looking! Analysis of possible trends. Taking into account current and emerging factors, requirements and limitations, for example: 1. 2. Institutional and national strategies and development plans. 3. 4. Emerging/changing social demand. 5. Possible activities of the competing research groups/schools. Changes of technological solutions and/or methodological approaches ( Example: increase of computing power). Legislative constraints (Example: ethical issues, usage of sensitive data) Image of a sandglass: absorb as much knowledge as possible (upper part) for having impact as broad as possible (lower part). 22

Critical aspect 2 – the methodology 1. The clarity of description and adequacy of

Critical aspect 2 – the methodology 1. The clarity of description and adequacy of the methodology, including • • • Unambiguous definitions and concepts Well structured research compliance with the objectives of the research. 2. Methodological Innovation. 3. Adequacy of the methods used. 4. The methodology must be described in detail, so that it would be possible to decide to what extent achieving the objectives of the project is realistic. Example: 23

Critical aspect 3 – availability of resources 1. Formal competency indicators of the Principal

Critical aspect 3 – availability of resources 1. Formal competency indicators of the Principal Investigator (Project manager’s) and the project team members, including: • • Existence of degrees and relevant certifications. Existence of high-quality publications and / or patents. 2. Non-formal competency indicators of the Principal Investigator and the project team members, including: • • Experience in completing similar projects. Previous cooperation experience between the team members. 3. Existence of necessary infrastructure, equipment and other tools. 24

Critical aspect 4 – sustainability Definition 1: sustainability = long-term survival capability. Definition 2:

Critical aspect 4 – sustainability Definition 1: sustainability = long-term survival capability. Definition 2: sustainability = conformity to present needs without compromising the ability to conform the future needs. 1. Ensuring continuity of people, particularly through inclusion of graduate students in the project implementation. 2. Maintaining/improving the academic environment (i. e. the project should not exhaust the resources). 3. Growth of reputation, particularly through applicability and high impact of the project outcomes. 4. ? 25

Home assignment 1. Identify and formulate the objective of a research project in the

Home assignment 1. Identify and formulate the objective of a research project in the area of your thesis and perform the following tasks: • • Describe the expected outcome of the project. Explain the necessity (topicality) of achieving the objective of the project. Describe the innovativeness of the expected outcome. Analyze the possible applications of the expected outcomes. 2. Compose a 5 -7 -minutes presentation about the items described above. 26

Next class: Tuesday, February 11, at 12: 00 Seminar: Identifying the objective and outcomes

Next class: Tuesday, February 11, at 12: 00 Seminar: Identifying the objective and outcomes of a research project 27

Example: fixed weights in rating project proposals 1. The scientific and/or technological level •

Example: fixed weights in rating project proposals 1. The scientific and/or technological level • • • The level of project objectives Correspondence of the project’s concept to the objectives Innovativeness with respect to the current international level 2. Implementation of the project • • • 30% Research/technological development methodology Compliance of activities and budget with the objectives The partners involved and the level of their experience Project management structures, adequacy and thoroughness The adequacy and reasonableness of resources involved 3. Impact of the project • • • 40% 30% The project's impact on science and technology development Communication plan of the results The use of the intellectual property created in the course of the project 28

Example: non-fixed weights in rating project proposals 1. Scientific justification of the project -

Example: non-fixed weights in rating project proposals 1. Scientific justification of the project - 3, 5 2. The competence and expertise of the PI and research staff - 3 3. The importance of the research for Estonian and EU society, economy and culture -4 4. State-of-art of research environment -4 5. Correspondence to national strategic development plans and to the institution’s development plans and strategies - 4, 5 6. Justification for accommodating other available R&D resources into the budget -4 Overall assessment - 3, 5 29

Example: a formula based budget The simplest case: the formula contains three parameters only:

Example: a formula based budget The simplest case: the formula contains three parameters only: 1. 2. 3. X – Number of researchers’ positions Y – Number of other academic staff’ positions P – proportion of experimental research, 0 ≤ P ≤ 1. The formula for calculating Estonian target financing grant (yearly, €): S = 15050 X + (X+Y) (4215 P+1690 (1 -P)) Additional possible parameters for more complex formula: • • Number of graduate students Number of technical staff Equipment … 30

Application-based budget calculation Fiscal position Application Awarded Salaries (including taxes) 2 556 Stipends 6

Application-based budget calculation Fiscal position Application Awarded Salaries (including taxes) 2 556 Stipends 6 000 Equipment - Mobilities 3 068 2 000 Other costs 1 534 1 444 TOATAL 13 158 12 000 31

Proportional budget calculation Rank of the application Awarded Coefficient 1 584 600 526 000

Proportional budget calculation Rank of the application Awarded Coefficient 1 584 600 526 000 0, 9 2 599 000 539 000 0, 9 3 221 000 188 000 0, 85 4 582 300 466 000 0, 8 5 460 600 322 000 0, 7 6 315 000 0 0 7 640 000 0 0 8 227 000 0 0 9 482 000 0 0 10 325 000 0 0 11 315 000 0 0 32

Combined budget calculation Rank of the application Awarded Coefficient 1 584 600 450 000

Combined budget calculation Rank of the application Awarded Coefficient 1 584 600 450 000 0, 77 (0, 9) 2 599 000 539 000 0, 9 3 221 000 188 000 0, 85 4 576 920 300 0, 52 (0, 8) 5 460 600 322 000 0, 7 6 315 000 0 0 7 640 000 0 0 8 227 000 0 0 9 482 000 0 0 10 325 000 0 0 11 315 000 0 0 33

Objective – good examples 1. Learning Layers: Development of a set of modular and

Objective – good examples 1. Learning Layers: Development of a set of modular and flexible technological layers for supporting workplace practices in SMEs that unlock peer production and scaffold learning in networks of SMEs, thereby bridging the gap between scaling and adaptation to personal needs. 2. Development of change-aware collaborative decision-making algorithms and tools based on dynamically integrated big data. 3. Design and validation of methods and web based software tools for language learning and testing the language proficiency level A 1 -A 2 -B 1 -B 2 -C 1 -C 2 on the basis of Estonian Inter-language Corpus. 34

Example: improper research objective Development of a web-based and competence-driven learning environment for personal

Example: improper research objective Development of a web-based and competence-driven learning environment for personal learning which is open to all educational institutions in Croatia. What is wrong with this objective? 1. What is “competence-driven learning environment ”? 2. R & D should be problem-based, not country-based. 3. Openness to institutions is an administrative, not a research issue. 4. “to all educational institutions” refers to the universality of the proposed learning environment. Why only for educational institutions (LLL, workplace learning, …)? Back … 35

Example: rated as not feasible (unrealistic) The main objective: to create theoretical foundations and

Example: rated as not feasible (unrealistic) The main objective: to create theoretical foundations and artificial intelligence support systems prototype for (1) describing situations and (2) deducing decisions based on the written texts. Sub-goals: 1. To model artificial intelligence support systems for representing the situation descriptions and situation constraints in natural language with the help of formal structures. 2. To model artificial intelligence support systems for representing arguments in natural language with the help of formal structures. 3. To create a web-based software prototype which allows to transform the descriptions of the situations in Estonian language to the logic formulas automatically. 4. To model artificial intelligence support systems for deriving decisions and checking for derivability based on situation descriptions and situation constraints. 5. To validate the created systems by imitating various management processes. 36

Example: description of research methodology The research methodology bases on the principles of design

Example: description of research methodology The research methodology bases on the principles of design research (The Designbased Research Collective, 2003; Herrington & Oliver, 2000; Mc. Mahon & Oliver, 2004): intertwined development of learning environments and learning design theories; repeated and spiral cycles of design, development based on analysis and evaluation; development of learning design theories that support practical applications, authentic design. Secondly, the principles of action research (Whitehead & Mc. Niff, 2006) will be used to inform our methodology which call for involving the target group to the study as researchers and interpreters of their own learning processes. The study is divided into three sequential cycles with the following phases: 1) needs analysis and the analysis of current practices; 2) planning of design process; 3) performing the design, evaluating the results and 4) implementing the outcomes. The results of the first cycle are used in executing the second and third cycles. The soft ontology methods for multiperspective analysis and corresponding software will be used that are developed by the research group [Kaipainen et al. , 2007]. We would like to emphasize the new opportunities that the use of modified Petri nets will offer for describing and analysis of activity theory based learning processes. 37

Example: description of a research cycle Goal of the cycle: Developing the pedagogical pattern

Example: description of a research cycle Goal of the cycle: Developing the pedagogical pattern language. Phase Activities 1 The focus group study with the aim to map teachers’ pedagogical behaviour patterns in dealing with learning objects and e-learning tools as well as terminology used in relation to these patterns. 2 The application of soft ontology methods on the collected data corpus for defining the elements of pedagogical patterns and for development of the principles for describing pedagogical patterns. 3 Visualization, multi-perspective exploration and management of pedagogical patterns using soft ontology and grounded theory approach. 4 Grounded theory methodology is followed in all phases of the 1 st cycle in order to critically evaluate the developed pattern language until theoretical saturation point will be achieved. Multi-perspective exploration as a mixedmethod (qualitative-quantitative) tool of evaluation will be applied as well. 38

Example: extract from a review Yet, while this general framework is well-presented, the specific

Example: extract from a review Yet, while this general framework is well-presented, the specific research design and practical research tasks remain disappointingly vague. The field of pedagogical pattern language is broad and well-developed, but the proposal gives little indication as to even the kinds of patterns that might be suitable for the language they wish to create. The project cycles are also extremely nebulous, giving no clue as to the time frame of each cycle or the practical activities involved (for instance, where will the focus groups among teachers be conducted? how many? ). Likewise, it is not clear whether the project plans to develop a prototype of a distributed learning environment (proposed for the 3 phase of the III cycle) or a distributed learning environment itself (to be tested in the 4. phase of the IV cycle). 39