NETLIPSE Infrastructure Project Assessment Tool Stuart Baker Deputy

NETLIPSE Infrastructure Project Assessment Tool Stuart Baker, Deputy Director of National Rail Projects Department for Transport, UK Vilnius, May 18, 2010

Contents 1. Background IPAT 2. IPAT – The assessment tool 3. IPAT – The assessment process – pilot cases 4. Lessons learned 5. The future of IPAT 6. Discussion

What is the context of IPAT? § Predecessor: Netlipse I research § One of the main deliverables of Netlipse II § Funding by the European Commission / TEN-T Agency in order to: § Improve the current management and organisation of LIPs § Get insight in ‘the vitality’ of projects on certain moments, e. g. financing (gate review) § Get better insight in the progress of LIPs § Benchmark projects

What is the IPAT? § The IPAT is an assessment tool § The IPAT indicates the level of confidence the IPAT assessors have that the PDO and its Client have created the necessary conditions for successfully executing the next project phase § The IPAT may be used by the European Commission /remark MHE/ as one of its tools for deciding what projects will receive their support.

When to apply the IPAT?

What is the added value of the IPAT? § Project delivery organisations: to increase the certainty of successful execution of projects, resulting in particular in reduced cost overruns and time delays, and; § Clients and funders: to understand the deliverability of projects by the project delivery organisations § EU, local governments and financial institutes: to monitor and evaluate projects (ex ante and ex post) in a systematic way § NETLIPSE and research institutes: the collection of information on research forecasts and future research demands.

Focus of the IPAT A Clear Context and Purpose, with a competent Organisation will bring the Required Results Organisatio n

IPAT – Conceptual model and themes Political context (Usually a Ministerial client) T 1 Client / official sponsor T 2 T 6 T 12 Stakeholders T 3 T 6 Related projects T 4 T 7 Project Delivery Organisation T 8 T 9 T 12 T 5 T 10 T 11 Private companies (contractors, advisors, operators, etc. ) T 1 Political Context T 2 Objectives, Purpose and Business Case T 3 Functional Specifications T 4 Interfaces T 5 Stakeholder Management T 6 Finance T 7 Legal procedures T 8 Technology T 9 Knowledge T 10 Organisation & Management T 11 Contracting T 12 Risks

IPAT – model: Project Phases of the Project (relate to Milestones): M 1 M 2 M 3 M 4 M 5 M 6 M 7 Initiation of the project Funding assembly Official approval official planning authority Start of execution Completion Start operation 5 years after start of operation

Levels of Importance by Milestone and by Theme M 1 M 2 M 3 M 4 M 5 M 6 T 1 T 2 T 3 T 4 T 5 T 6 T 7 T 8 T 9 T 10 T 11 T 12 Prioritisation: Level 1: Minimal importance Level 2: Little importance Level 3: Medium importance Level 4: Important Level 5: Crucial M 7

IPAT – The assessment tool Three elements: § Questionnaire § Scoring model § Assessment process

A pilot assessment § EX ANTE EVALUATION § Femern Bælt, Denmark

A pilot assessment: Femern Bælt, Denmark § STEP 1: Search for background information

A pilot assessment: Femern Bælt one of a long term series of key Danish links Great Belt Øresund Fehmern Bælt

19 km coast to coast connection (Bridge or Tunnel? ) § Bridge § Tunnel

Project Organisation § Femern A/S: subsidiary of Sund & Bælt

Budget & Financing • a cable-stayed bridge: 4. 5 billion Euro • an immersed tunnel: 5. 5 billion Euro (prices as • hinterland infrastructure on the German side: 840 million of 2008) Euro (cost as of 2003). • hinterland infrastructure on the Danish side: 1 billion Euro Budget Financing 70 -75%: by Danish loans, to be refinanced by toll income 25 -30%: European Commission (TEN-T) funds for the period from 2007 to 2013 of 339 million Euro. For the period from 2014 to 2020, EU support will be requested

Planning Present stage of project. Investigations: - route corridor - geotechnical - maritime - environmental Time of decision to build, i. e. , the go/no-go decision 2012: Anticipated presentation of a construction law in the Danish Parliament Time of start of construction 2013: start of construction phase for the coast-to-coast segment Time of delivery 2018 Commencement of operations End of 2018

A pilot assessment: Femern Bælt, Denmark § Step 1: Search for background information § Step 2: Site visit - interviews

Site visit § Assessors team: § Stuart Baker § Marcel Hertogh § Ellen Gehner § Interviewees: § Peter Lundhus – Managing director Femern A/S § Steen Lykke – Project director Tunnel § Open interview on 12 themes § Duration: 2 x 2 hours

A pilot assessment: Femern Bælt, Denmark § Step 1: Search for background information § Step 2: Site visit – interviews § Step 3: Scoring

IPAT- Questionnaire (Theme 1) 1 Political context M 1 1 Is there a political consensus about the need and general purpose of the project? 0 2 Is there a political consensus on the defined outcomes from the project? 0 3 Is there a political consensus on the type of infrastructure solutions proposed by the project? 0 4 Is there a political consensus on the defined route of new infrastructure? 0 5 Are there clear interfaces between the politicians and the clienting of the project? 0 6 Is there political consensus on the funding solutions? 0 7 Are there arrangements to handle any political interfaces, including differences of view between the various levels of (inter)national, regional and local, city level? 0 8 How is the relation between the Sponsor/Client and the politics/ and how do you handle differences? 0 Total score (Σ score 1 -8) Average score (Total score / number of questions) 0 0, 0

Scoring process 1. Score all questions in the IPAT Questionnaire § § § on a scale from 1 to 4 individually Questions that are not relevant to a project phase or not applicable to a project must be indicated as such. 2. Note relevant arguments and comments that support the scoring as input for the assessment report 3. Organise a meeting with the assessors team to create consensus about the scores

Scores § § The objective of scoring the criteria is to identify the main strengths and weaknesses of the project and its organisation within each theme. Scoring table reflects weakness of the project (low score) and strength of the project (high score): Score Qualification This reflects Score 1 Very negative effect on the project being successful a vital need to improve immediately Score 2 Negative contribution to a successful project an need to improve a weak area Score 3 Positive contribution to a successful project Further progress can be made to achieve a better result Score 4 Very positive contribution to a successful project a clear strong area for the project (probably close to best practice)

IPAT – Scoring model PROJECT NAME Theme PROJECT PHASE: M 1 Weight Score Max. score Weighted score Max. weighted score 1 Political Context 3, 0 … 4, 0 = Weight x Score = Weight x Max. score 2 Scope 3, 6 … 4, 0 “ “ 3 Functional Specifications 4, 8 … 4, 0 “ “ 4 Interfaces 4, 8 … 4, 0 “ “ 5 Stakeholder Management 4, 2 … 4, 0 “ “ 6 Finances 4, 2 … 4, 0 “ “ 7 Legal Procedures 3, 6 … 4, 0 “ “ 8 Technology 4, 4 … 4, 0 “ “ 9 Knowledge 4, 6 … 4, 0 “ “ 10 Organisation and Management 5, 0 … 4, 0 “ “ 11 Contracting 4, 8 … 4, 0 “ “ 12 Risks 4, 4 … 4, 0 “ “ = Total Weighted Score = Total Maximum Weighted Score Total Weighted Percentage Score = Total Weighted Score / Total Maximum Weighted Score

IPAT – Scoring model PROJECT NAME Theme PROJECT PHASE: M 2 Weight Score Max. score Weighted score Max. weighted score 1 Political Context 4, 0 3, 3 4, 0 13, 0 16 2 Scope 4, 9 2, 8 4, 0 13, 4 19, 6 3 Functional Specifications 4, 2 3, 5 4, 0 14, 7 16, 8 4 Interfaces 1, 9 1, 1 4, 0 2, 1 7, 6 5 Stakeholder Management 2, 4 1, 0 4, 0 2, 4 9, 6 6 Finances 4, 1 3, 8 4, 0 15, 4 16, 4 7 Legal Procedures 3, 6 2, 0 4, 0 7, 2 14, 4 8 Technology 3, 4 3, 1 4, 0 11, 0 13, 6 9 Knowledge 2, 8 3, 0 4, 0 8, 3 11, 2 10 Organisation and Management 3, 3 2, 8 4, 0 9, 4 13, 3 11 Contracting 2, 6 1, 3 4, 0 3, 35 10, 4 12 Risks 4, 2 3, 0 4, 0 12, 6 16, 8 = 112, 9 = 165, 7 Total Weighted Percentage Score = 68, 1%

A pilot assessment: Femern Bælt, Denmark § § Step 1: Search for background information Step 2: Site visit – interviews Step 3: Scoring Step 4: Analysis of the results

Result 1: Weight – score matrix (sub)themes

Result 2: Overall score total weighted percentage score

A pilot assessment: Femern Bælt, Denmark § § § Step 1: Search for background information Step 2: Site visit – interviews Step 3: Scoring Step 4: Analysis of the results Step 5: Writing the assessment report

Two pilot assessments § MONITORING / EX POST EVALUATIONS § Gotthard Base Tunnel § West Coast Main Line (Pre 2002)

Overall score: Total percentage score West Coast Main Line – Failed Project Pre 2002

(Sub)theme scores: Weight – score matrix West Coast Main Line – M 1

(Sub)theme scores: Weight – score matrix West Coast Main Line – M 1, M 2

(Sub)theme scores: Weight – score matrix West Coast Main Line – M 1, M 2, M 4

Overall score: Total percentage score Gotthard Base Tunnel

(Sub)theme scores: Weight – score matrix Gotthard Base Tunnel – M 1

(Sub)theme scores: Weight – score matrix Gotthard Base Tunnel – M 1, M 2

(Sub)theme scores: Weight – score matrix Gotthard Base Tunnel – M 1, M 2, M 3

(Sub)theme scores: Weight – score matrix Gotthard Base Tunnel – M 1, M 2, M 3, M 4

Lessons learned from (pilot) pilots § Validation and fine-tuning of the questionnaire § § § Development of the scoring methodology § § § Questionnaire was comprehensive Questions were relevant Questions were recognisable for the interviewees Questions could be answered by the interviewees Some questions are reformulated The assessors encountered little problems in scoring the questions The assessors were generally unanimous about the scoring The methodology of calculating the overall score was considered and changed ahead of the pilots Significance of levels of importance appears to be little Development of the final analysis of the IPAT: § Still to do: what are appropriate ‘pass scores’? ? ?

What’s next until end 2010? § Pilots still to undertake: § Lock - Flanders (June 2010) § French Project possibly Figueras - Perpignan (Sep 2010) § Koper – Ljubljana (June 2010) § Operational version of the IPAT (including assessors manual) § List of competent assessors § Training programme

Discussion (1) § Role of the assessors during the assessment process: how to guarantee the quality of the assessment?

Discussion (2) § Pass score: what is a ‘good enough’ project? § (sub)theme score – position of green area? ? ? § Overall score > 50% ? ? ? Bandwidth? § Extra hurdle: the overall score might be considered good enough if on each (sub)theme level the score is in or above the green area? ? ?

Discussion (3) § Added value of the application of IPAT: what makes the IPAT attractive to Clients and PDOs to do an assessment?

Time for your input now please. . . Having heard all this, do you broadly understand the IPAT and are you now convinced that you we can do a reliable, fair and comparable assessment of Projects with the IPAT?