Brokerage Event Call 2011 Barcelona REMINDER The Evaluation

Brokerage Event Call 2011 Barcelona

REMINDER The Evaluation Process ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 2

Evaluation FPP – the process Selection of experts § The Executive Director (ED) appoints himself or one of § § his staff as the Call Coordinator The ED approves the ‘global list’ of experts to be used for the evaluation (PO and FPP phase). The list includes experts that are registered in the FP 7 - EMM database AND Ø are suggested by the PAB or Ø are suggested by the IRC Ø § The Call Coordinator selects experts and assigns them to proposals ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 3

Evaluation FPP – the process: Opening and registration of proposals § Full Project Proposals must be submitted electronically, § § using the Electronic Proposal Submission System (EPSS). Proposals arriving by any other means are regarded as ‘not submitted’, and will not be evaluated For Call 2010 -1 FPP only 2 cases of incomplete submission Case 1 – minor impact: only 1 page update Ø Case 2 – major modification discarded because new version was not submitted in time (Sept 1 st - 5 PM) Ø ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 4

Evaluation FPP – the process: Eligibility checks § The ARTEMIS JU checks the Proposals’ conformity to the eligibility criteria for Full Project Proposals (form E-1). Proposals that fail this eligibility check do not go forward to any of the following steps and will be notified of the outcome of this check Ø In 2010 -1 FPP – all proposals were found eligible § Eligible proposals are sent to National funding Authorities (via CIRCA) to be checked against national eligibility criteria for funding (form E-2). Ø REMINDER: Feedback mandatory (to be sent by evaluation close) § The ARTEMIS JU checks the eligibility criteria for funding (form E-3). Ø Only minor discrepancies found ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 5

Evaluation FPP – the process: Assignment of experts § The Call Coordinator assigns 4/5 experts to each eligible FPP proposal Ø Two from the list of experts suggested by the Industrial and Research Committee (IRC) Ø Two from the list of experts suggested by the Public Authorities Board (PAB). Ø One of the experts included in the list of experts approved by the Executive Director is appointed as proposal rapporteur § All experts are registered in the EC FP 7 EMM expert database § Experts are selected so that they cover the technical content of proposals to the best possible extent Ø rules for conflict of interest described in Annex B of the ARTEMIS Evaluation and Selection procedures are followed. Ø ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 6

Evaluation FPP – the process: Individual assessment of FPPs § No proposal material is sent or made available to an expert until the Conflict of Interest and Confidentiality declaration in the appointment letter have been completed, signed and returned to the Coordinator. § Individual readings are done remotely. The proposals, instructions and other key documents and forms are provided to the experts by an electronic remote evaluation tool (RIVET). Ø Each expert assigned to a FPP proposal reads it in detail and makes an individual evaluation without discussion with other experts. Ø Scores and comments are given for each of the evaluation criteria Comments are supported by concise, detailed and factual arguments. Scores are consistent with comments. The evaluator records scores and comments on the Individual Assessment Report (IAR) in RIVET and finalize it. § The Coordinator checks that each IAR form is complete and finalised on time before the panel meeting. ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 7

Evaluation FPP – the process: Consolidation of individual assessments § Proposal rapporteurs are responsible for § § consolidating the individual assessment reports (IAR) and for drafting the corresponding Synthesis Report (Consensus Report). This step can be done remotely (via the RIv. ET tool) or at the panel meeting venue previous to "Panel discussions". If the consolidation is done at the panel meeting and the proposal rapporteur is not present in this meeting, the Coordinator will assign another expert attending the panel as proposal rapporteur. ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 8

Evaluation FPP – the process: Panel meeting § A subset of the experts employed in the remote evaluations is invited in Brussels to the Panel session. Ø Ø All proposal rapporteurs are normally present if this is not the case for a given proposal, the Coordinator will appoint a proposal rapporteur from the panel experts. § One of the panel experts is selected to act as “panel rapporteur” who will be § § responsible for providing input to the panel report (PR). The panel discussions are chaired by a Moderator (the Executive Director, the Coordinator or other staff appointed by the Executive Director). Each panel member is provided[1] with the technical summaries of all proposals and the draft CR. Copies of the Full Project Proposals are made available for on-site consultation[2]. Panel members are given the time to make themselves familiar with FPPs for which they have not intervened as experts. [1] except in case of conflict of interest [2] except in case of conflict of interest ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 9

Evaluation FPP – the process: sub-panels § Depending on the number of FPP to be evaluated the ED may decide to implement one of more sub-panels Ø Each sub-panels runs in parallel Ø Each sub-panels review a number of ESR Ø For each sub-panel, one JU staff is appointed as a moderator ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 10

Evaluation FPP – the process: sub-panels § Each sub-panel reviews a number of the FPPs § Any expert in the sub-panel who has a conflict of interest with a proposal is § § § requested to leave the room when that proposal is discussed in the subpanel The sub-panel discussions are be based on the draft ESR (Evaluation Summary Report). The proposal rapporteur is in charge of reflecting in the ESR the final score and comments for each assessment criterion, as well as any other salient point as recommended by the panel. The rapporteur: Ø Ø introduces briefly the FPP He/she provides an overview on the ESR draft He/she provides an overview on the consensus / Reports dissenting views Points out inconsistencies (scores vs. comments) Ø If necessary, the Moderator may make available to the panel the content of the individual IARs when discussing a particular proposal. § The sub-panel discusses and agrees on the ESR (scores and comments) ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 11

Evaluation FPP – the process: off-line checks on ESRs § Once an ESR is discussed in the subpanel The moderator put any relevant info in a ESR forms The ESR is printed out and sent to the PM for first level check Ø Once PM approved the ESR goes to the ED for a second level check Ø The ESRs are signed by the respective proposal rapporteurs and the Moderator Ø Ø § In case of observations, the ESR is sent back to the § sub-panel for analysis Only after the 2 level checks are successfully passed, the ESR can be finalized Ø A final spell-check is done off-line (after the evaluations session) ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 12

Evaluation FPP – the process: Panel discussions § Once all ESR are discussed, the sub-panels convene in a “plenum” § § under the chairmanship of the Moderator (the ED). The goal of the panel discussions is not to obtain a consensus on the comments to be included in the individual ESR, but rather to seek consistency in the evaluations (e. g. scores vs. comments) and to ensure that the same standards of quality are applied across FPPs. The panel may make observations on the overall quality of proposals, the coverage of the objectives and priorities of the Call or on any other issue it deems important The panel also may point out any overlaps, synergies or other relations between proposals. . Any expert in the panel who has a conflict of interest with a proposal notifies the Moderator and either leave the room or remain silent when that proposal is discussed in the panel. ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 13

Evaluation FPP – the process: Elaboration of the Panel Report (PR) § The panel rapporteur prepares a report on the panel deliberations and conclusions. The following issues are recorded: • Coverage of the objectives or priorities of the AWP by the proposals (ASP coverage) • Links, synergies or overlaps between proposals and how these could be addressed. • Comment on any other issue deemed significant by the panel. § This content of the Panel report is included in the overall Evaluation Report that will be submitted by the Executive Director to the Public Authorities Board. ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 14

FPP evaluation – statistics experts IRC vs PAB § Experts Ø Remote 67, of which • 32 IRC • 30 PAB • 5 both PAB and IRC Ø Panel 25, of which • 12 IRC • 12 PAB • 1 both IRC and PAB ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 15

FPP evaluation – statistics Experts Nationality - Remote ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 16

FPP evaluation – statistics Experts Nationality - Panel ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 17

REMINDER The Evaluation Criteria ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 18

Call 2010 Evaluation Criteria § No changes to the 5 evaluation criteria since Call 2009 § Comments were added clarifying the criteria, for Proposers and for Evaluators 1. Relevance to the Call • 2. R&D Innovation and Technical Excellence • 3. Clarifies “State of the Art” as being what is commercially in the field today (not only what is in labs) S&T approach and work-plan • 4. Clarify ARTEMIS as a downstream programme, so ASP targets take precedence of Industrial Priorities (more generic) Clarifies need for practicable demonstration/prototypes Market Innovation and Impact • • Has weight of 2, so good understanding is vital Sub-criteria re-ordered and clarification of their relative weight » important, very important or very important • 5. “Open Source business-models” added as a note for sub-criterion f) (IPR). Quality of consortium and management • No clarification is felt to be needed • • Generation and protection of Intellectual Property: redefined in criterion 4 Management of Intellectual Property Rights: added in criterion 5 § Changes criteria 4 and 5 (minor) ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 19

Evaluation Criteria Call 2010: the 5 evaluation criteria were: 1. 2. 3. 4. 5. Relevance and contributions to the objectives of the Call. R&D innovation and technical excellence. S&T approach and work plan. Market innovation and market impact. Quality of consortium and management. Ø Ø Ø § Evaluation scores will be awarded for each of the five criteria, and not for the subcriteria. Each criterion will be scored out of 10. Criteria 1, 2, 3, and 5 will have a weight of 1 and criterion 4 will have a weight of 2. The threshold for the individual criteria (1), (2), (3), (4) will be 6. There is no threshold for the individual criterion (5). The overall threshold, applying to the weighted sum of the five individual scores, will be 40. Each criterion has explanatory notes, to harmonise interpretation Ø ARTEMIS Joint Undertaking Helps both proposers and evaluators ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 20

Some further explanation on the evaluation criteria: 1. Relevance and contributions to the objectives of the Call. Relevance will be considered in relation to the topic(s) of the work programme open in a given call and to the objectives of a call. Ø Relevance and contribution to the overall ARTEMIS targets listed in section 4 of the AWP. Ø [It must be noted that ARTEMIS puts high emphasis on the downstream relevance of proposals. Therefore, the contribution to specific ASP targets takes precedence over contribution to the Industrial Priorities, though both must be addressed]. ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 21

Criterion 1 - Comments § More guidance is needed on this one § A good project should address the objectives of 1 subprogramme address the objectives of 1 industrial priority contribute to the “overall ARTEMIS targets” objectives should be presented with quantitative and measurable figures Ø cross domain objectives are a “plus” if clear, understandable and justified Ø unsubstantiated claims are definitely a “minus” Ø Ø § The term "embedded systems" is sometimes not clearly defined or understood Ø pure application software projects are eligible if system aspects are tackled with standard hardware Ø pure application software projects are not eligible if totally independent from any hardware (system) aspect ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 22

Some further explanation on the evaluation criteria: 2. R&D innovation and technical excellence. Soundness of the concept Ø Clarity and quality of the objectives and expected results Ø Progress beyond the state-of-the-art. Ø [Here, the State of the Art refers to that which is currently seen in an 'industrial' context, i. e. existing, commercially viable solutions, technologies or applications. ARTEMIS views the novel application or integration of existing technologies, alongside new technologies, in new domains or for improving efficiency in existing domains as valid advances on the State of the Art under this interpretation] ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 23

Criterion 2 - Comments § Description of state-of-the art with sufficient references is essential Ø Including relation, complementarities to other ongoing projects § Clear objectives are essential, should be § quantified and measurable and demonstrate the advance compared to state-of-the art. Many proposals present qualitative objectives but no quantitative objectives ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 24

Some further explanation on the evaluation criteria: 3. S&T approach and work plan Ø Quality and effectiveness of the S&T methodology Ø Quality of the work plan. [The S&T methodology and work plan must also consider the specificities of the downstream nature of the ARTEMIS programme, addressing also prototype and/or demonstration phases] ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 25

Criterion 3 - Comments § Relation with the objectives should be clear § Role of partners should be clear § Demonstrators are important (proving industrial orientation). ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 26

Some further explanation on the evaluation criteria: 4. Market innovation and market impact Contribution, at the European and/or international level, to the expected impacts listed in the work programme under the relevant sub-programme Ø Degree of application innovation in the context of the sub -programmes addressed Ø Market impact and quality of the exploitation plans of the industrial partners; quality of the market analysis section including competitor descriptions and market opportunities. Ø Appropriateness of measures for the dissemination of project results. Ø Contribution to standards. Ø Management of intellectual property. Ø ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 27

Some further explanation on the evaluation criteria: Criterion 4 (cont. ): § Applicants and reviewers/evaluators are reminded that this criterion has a weight of 2, emphasising the market innovation orientation of the ARTEMIS programme. Proposals should highlight these items and also foresee continuous evaluation of the expected results in evolving markets. Ideally, the project should be able to demonstrate predicted impact through practical demonstrators as part of their dissemination plan. As a guide, the following gives an idea of the relative importance of each of these sub-criteria: Ø Ø Ø Market impact: Degree of application innovation: Contribution to the work programme: Dissemination measures: Contribution to standards: Management of intellectual property: very important very important. Remember that "Management of intellectual property" embraces its generation and protection, and may also contribute to concepts of IP business models, such as Open or Community Source, so can go beyond the management of the IPR within the project boundaries. ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 28

Criterion 4 - Comments § Market analysis (study) is essential (size and growth potential) Ø Competitor description should be included § Impact should be quantified with some credible § § explanation Exploitation plans are needed, including exploitation outside consortium Dissemination and standardisation is often addressed in the same way, showing lack of commitment (avoid the use of copy-paste!) ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 29

Some further explanation on the evaluation criteria: 5. Quality of consortium and management[1]. Ø Appropriateness of the management structure and procedures Ø Quality and relevant experience of the individual participants Ø Quality of the consortium as a whole including complementarities, balance and involvement of SMEs Ø Appropriateness of the level, allocation and justification of the resources to be committed (budget, staff, equipment) [1] This evaluation criterion corresponds to the selection criteria in the meaning of the general financial regulation (article 115) [OJ L 248, 16. 09. 2002, p. 1] and its implementing rules (article 176 and 177) [ OJ L 357, 31. 12. 2002, p. 1] and of the financial rules of the Joint Undertaking (article 101). It will also be the basis for assessing the 'operational capacity' of participants. The other four evaluation criteria (1 -4) correspond to the award criteria. ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 30

Criterion 5 - Comments § Management is often "cut and paste" from other § projects. Not a bad thing, it shows that they know the game. Makes it difficult to understand if proposals poorly address this type of things. Sometimes an overkill compared to the size of the project ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 31

Call 2009 Some numbers ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 32

ARTEMIS Call 2009 Evaluation averages and standard deviations ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 33

ARTEMIS Call 2009 Evaluation averages and standard deviations Analysis § For all proposals: Ø Criterion 1 is the less discriminating Ø Criteria 3, 4 and 5 discriminate the top scored proposals (1 -12) the runner up scoring proposals (13 -24) from the low scoring ones (2544) Ø Criterion 4 does not discriminate strongly between very good (1 -12) and good (13 -24) proposals ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 34

ARTEMIS Call 2009 Evaluation : additional comments § From the panel : In general, high-scoring projects have taken the PO phase feedback into account Ø Pre-evaluation phase is beneficial for programme execution quality Ø Some sub-criteria should be revised or better commented From the ED : Ø The final selection shows 24 very good proposals Ø The first 12 proposals remain almost the same if we do a sensitivity analysis towards “very much or less downstream research” • by dropping criteria 2 and 3 Ø For criteria 4, the “business plan” aspects should be explained and emphasized Ø § ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 35

Call 2010 Statistics AVERAGE Rank C 1 C 2 C 3 C 4 C 5 TOTAL 1 to 12 8, 92 7, 83 8, 00 7, 92 8, 42 49, 00 13 to 28 7, 75 6, 81 6, 50 6, 63 7, 81 42, 13 29 to 47 6, 53 5, 26 5, 11 5, 26 6, 37 33, 79 1 to 47 7, 55 6, 45 6, 32 6, 40 7, 38 40, 51 STDEVP Rank C 1 C 2 C 3 C 4 C 5 TOTAL 1 to 12 0, 76 0, 80 0, 71 0, 86 0, 76 2, 68 13 to 28 0, 75 0, 73 0, 50 0, 60 0, 53 1, 45 29 to 47 1, 67 0, 96 1, 07 1, 25 1, 31 6, 29 1 to 47 1, 54 1, 35 1, 42 1, 44 1, 30 7, 49 ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 36

Average scores for best 12 and runner up proposals Call 2009 § § § C 1 C 2 C 3 C 4 C 5 1. 16 1. 08 0. 67 0. 83 0. 59 Call 2010 1. 17 1. 02 1. 50 1. 29 0. 61 delta=0. 83 delta=0. 46 § Total 5. 16 6. 87 delta=1. 71 § Conclusion: improvement comes from C 3 and C 4 0. 83 + 2 x 0. 46 = 1. 75 ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 37

Observations from Call 2010 Evaluations & Suggestions for Call 2011 § Criterion 1 (“relevance to the Call”) offers little § differentiation Proposal: Ø Use criterion 1 to “steer” proposals to better match specific high-level goals of the ARTEMIS strategy (what makes an “ARTEMIS” project different). • Add to the sub-criteria / expand the guidelines to include: » Has the proposal synergies with other existing ARTEMIS projects? » Is there a good balance between focus on one ASP and the cross-sectorial aspects? » Continuation / refinement / … leading to increased valorisation of previous projects’ results » Does the proposal reference the MASP as well as the AWP? ARTEMIS Joint Undertaking ARTEMIS Brokerage Event Call 2011 Barcelona 15 -12 -2010 - 38