FAIRMODE meeting Norrkoping June 2011 Institute for Environment

FAIRMODE meeting, Norrkoping, June. 2011 Institute for Environment and Sustainability SG 4: Benchmarking of air quality models http: //ies. jrc. europa. eu/ http: //www. jrc. europa. eu/ 1

Agenda FAIRMODE meetind, Norrkoping, June 2011 Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion User’s presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Inclusion of planning application Benchmarking service Ensemble Review of possible TSAP Deliverables 2

SG 4 – Presentations on the Questionnaire FAIRMODE meetind, Norrkoping, June 2011 Short ü ü ü presentations (max 10 min) by: Ana Miranda (PT) David Carruthers (UK) Hans Backström (SE) Helge Olesen (DK) Marcus Hirtl (AT) Mihaela Mircea, Guido Pirovano (IT) 3

4 FAIRMODE meetind, Norrkoping, June 2011 Background

The benchmarking procedure 5 FAIRMODE meetind, Norrkoping, June 2011 JRC USER Data Extraction Facility Model results DELTA BENCHMARKING service Model performance Evaluation reports

Since the Oslo meeting (sep. 2010) FAIRMODE meetind, Norrkoping, June 2011 6 - Document “ DELTA concepts” sent to SG 4 participants (March 11) - Distribution of DELTA tool & utilities (about 20 users) - SG 4 Web page created (http: //aqm. jrc. it/DELTA) - Use of DELTA on different datasets (POMI, Madrid, London) - User feedback questionnaire

The DELTA tool 7 FAIRMODE meetind, Norrkoping, June 2011 • Intended for rapid diagnostics by single users (at home) • Focus mostly on surface measurementmodel pairs “independence” of scale • Focus on AQD related pollutants on a yearly period (but AQ related input data also checked) • Exploration and benchmarking modes

Agenda FAIRMODE meetind, Norrkoping, June 2011 Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion User’s presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Inclusion of planning application Benchmarking service Others (Ensemble…) Review of possible TSAP Deliverables 8

Outline FAIRMODE meetind, Norrkoping, June 2011 9 1. Content of the performance report 2. Links between Target and more “traditional” 3. 4. 5. 6. indicators (an analysis based on 3 datasets) A “first-guess” for criteria/goals Comparison with RDE Observation uncertainty Proposed update to the report template

FAIRMODE meetind, Norrkoping, June 2011 Content of the performance report 10

Content of the performance report (1) FAIRMODE meetind, Norrkoping, June 2011 11 Constraints • Should include a set of statistical indicators and diagrams complete enough to capture the main aspects of model performance but limited enough to fit in one page summary • Keep a similar template for all pollutants and spatial scales (but differences in terms of criteria/gals). • Restricted to AQD needs. Currently proposed for O 3 8 h daily max, NO 2 hourly and PM 10 daily. • Developed (at least first) for assessment purposes • Should include performances criteria and goals

Content of the performance report (2) 12 FAIRMODE meetind, Norrkoping, June 2011 0 < F E M RM SE /σ o R=0. 7 OU Criteria: Acceptable performance for a given type of application (e. g. PM: MFE=75%, MFB=+/-60%) Goal: Best performance a model should aim to reach given its current capabilities (e. g. PM: MFE=50%, MFB=+/-30%)

Content of the performance report (3) 13 FAIRMODE meetind, Norrkoping, June 2011 90% concept for indicators Checks on data availability for each stations • 75% for time averaging (e. g. 18 h at least per day) • 90% available on total (e. g. >328 days/year) MFB=0. 67

FAIRMODE meetind, Norrkoping, June 2011 Links between Target and more “traditional” indicators (an analysis based on 3 datasets) 14

Links between Target and more “traditional” indicators (1) 15 FAIRMODE meetind, Norrkoping, June 2011 Bias Target indicator = RMSE / SIg. O R Sig. M/Sig. O CRMSE FAC 2

Examples on 3 datasets 16 FAIRMODE meetind, Norrkoping, June 2011 61 monitoring sites suburban, urban and rural background 5 models: CHIMERE, TCAM, CAMX, RCG, MINNI Year: 2005 Domain resolution: 6 x 6 km 2 O 3 – PM 10 Madrid Po - Valley 10 monitoring sites urban background 1 model: WRF-CMAQ Year: 2007 Domain resolution: 1 x 1 km 2 O 3 – NO 2 London 107 monitoring sites suburban/urban background, kerbside and roadside 1 model: ADMS Year: 2008 NO 2 – O 3 – PM 10

Links between Target and more “traditional” indicators (3) FAIRMODE meetind, Norrkoping, June 2011 Methodology to fix “first guess” criterias • Based on real datasets, start by analysing how the bias criteria (MFB) proposed by Boylan and Russel (2005) compares to Target. • Fix a criteria for the target indicator which is consistent with the MFB criteria • Fix values for the other statistical indicators (R, Std. Dev ratio, FAC 2) to be consistent with the assigned criteria on the Target value 17

How to connect Target to more accessible indicators? (3) EXAMPLE 1: Po_valley (PM 10) 18 FAIRMODE meetind, Norrkoping, June 2011 Crit Target=1 T: 58% RDE: 83%

How to connect Target to more accessible indicators? (3) EXAMPLE 2: Po_valley (PM 10) 19 FAIRMODE meetind, Norrkoping, June 2011 Crit Target=1 T: 32% RDE: 95%

How to connect Target to more accessible indicators? (3) EXAMPLE 3: London (PM 10) 20 FAIRMODE meetind, Norrkoping, June 2011 Crit Target=1 T: 96% RDE: 100%

How to connect Target to more accessible indicators? (3) EXAMPLE 4: Po_valley (O 3) 21 FAIRMODE meetind, Norrkoping, June 2011 Crit Target=0. 8 T: 70% RDE: 96%

How to connect Target to more accessible indicators? (3) EXAMPLE 5: Madrid (O 3) 22 FAIRMODE meetind, Norrkoping, June 2011 Crit Target=0. 8 T: 66% RDE: 100%

How to connect Target to more accessible indicators? (3) EXAMPLE 6: London (NO 2) 23 FAIRMODE meetind, Norrkoping, June 2011 Crit Target=1 T: 77% RDE: 94%

How to connect Target to more accessible indicators? (3) EXAMPLE 7: Madrid (NO 2) 24 FAIRMODE meetind, Norrkoping, June 2011 Crit Target=1 T: 60% RDE: 100%

FAIRMODE meetind, Norrkoping, June 2011 A “first-guess” for criteria/goals 25

A “first-guess” for criteria/goals (1) 26 FAIRMODE meetind, Norrkoping, June 2011 MFB Target R FAC 2 SM/SO PM 10 Daily 60% 1. 0 ~ 0. 50 50% ± 50% O 3 8 h 30% 0. 8 ~ 0. 65 50% ± 50% NO 2 1 h 30% 1. 0 ~ 0. 55 50% ± 50% NOTE: Boylan and Russel MFB criteria • is proposed based on urban to regional scale modelling (from 4 to 36 km spatial resolution) • addresses only O 3 and PM 10

A “first-guess” for criteria/goals (2) 27 FAIRMODE meetind, Norrkoping, June 2011 • Different criteria are currently proposed for O 3 -8 h, PM 10 -daily and NO 2 -hourly. Although spatial-scale and time average dependency are possible, they are not considered up to now (point of discussion) • Scale is intended in terms of spatial resolution, linked to monitoring station type: • Regional • Urban • Local Rural background Urban & suburban background All urban stations (incl. roadside & kerbside) • Criteria probably need to be developed for yearly averaged values • Performance goals have arbitrarily been fixed to a 20% more stringent value • 3 datasets is not ENOUGH!

FAIRMODE meetind, Norrkoping, June 2011 28 How do these criteria compare to RDE?

29 FAIRMODE meetind, Norrkoping, June 2011 Target R PM 10 FAC 2 RDE SM/SO MFB

30 FAIRMODE meetind, Norrkoping, June 2011 Target R O 3 FAC 2 RDE SM/SO MFB

31 FAIRMODE meetind, Norrkoping, June 2011 Target R NO 2 FAC 2 RDE SM/SO MFB

How these criteria compares to RDE? (2) 32 FAIRMODE meetind, Norrkoping, June 2011 Station Osio Sotto, POMI (NO 2 - RCG) Target: MFB: FAC 2: R: Sig. M/Sig. O: 2. 19 73% 41% 0. 39 1. 49 RDE=11%

How these criteria compares to RDE? (3) 33 FAIRMODE meetind, Norrkoping, June 2011 Station EA 1, London (NO 2 - ADMS) Target: MFB: FAC 2: R: Sig. M/Sig. O: 0. 82 8% 89% 0. 73 1. 14 RDE = 56%

FAIRMODE meetind, Norrkoping, June 2011 About observation uncertainty 34

About observation uncertainty (1) 35 FAIRMODE meetind, Norrkoping, June 2011 AQD Pollutant OU O 3, NO 2 15% PM 10 25% S 0 O 3 NO 2 PM 10 RMSE / SO PO 43 0. 30 LOND 24 0. 35 MAD 31 0. 36 PO 22 0. 29 LOND 27 0. 31 MAD 30 0. 25 PO 26 0. 48 LOND 13 0. 56 MAD 16 0. 57 RMSE/S 0

About observation uncertainty (2) 36 FAIRMODE meetind, Norrkoping, June 2011 NO 2 ADMS, London, 2008 O 3 PM 10

FAIRMODE meetind, Norrkoping, June 2011 37 Proposed update to the report template

Proposed update to the report template 38 FAIRMODE meetind, Norrkoping, June 2011 ✘ ✘ ✘ Sig. O > Sig. M ✘ Sig. O < Sig. M Sig. O/Sig. M ✓

Agenda FAIRMODE meetind, Norrkoping, June 2011 Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion User’s presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Inclusion of planning application Benchmarking service Others (Ensemble…) Review of possible TSAP Deliverables 39

DELTA: Exploration mode (1) FAIRMODE meetind, Norrkoping, June 2011 Exploration: • Time selection (period, averaging time, season, day/night-time, max/min/mean) • Information overlay (models, scenarios, variables, stations) • Spatial analysis (color codes vs. 2 D maps) 40

DELTA: Exploration mode (2) FAIRMODE meetind, Norrkoping, June 2011 41

DELTA: Exploration mode (3) FAIRMODE meetind, Norrkoping, June 2011 42 Model V 1 vs. Model V 2 Upgrade

DELTA: Exploration mode (4) FAIRMODE meetind, Norrkoping, June 2011 43 Upgrade

Agenda FAIRMODE meetind, Norrkoping, June 2011 Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion User’s presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Inclusion of planning application Benchmarking service Others (Ensemble…) Review of possible TSAP Deliverables 44

Agenda FAIRMODE meetind, Norrkoping, June 2011 Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion User’s presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Inclusion of planning application Benchmarking service Others (Ensemble…) Review of possible TSAP Deliverables 45

Agenda FAIRMODE meetind, Norrkoping, June 2011 Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion User’s presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Benchmarking service Others (Ensemble…) Review of possible TSAP Deliverables 46

DELTA developments FAIRMODE meetind, Norrkoping, June 2011 Short term (Autumn 2011) • Flexible use of benchmarking mode and production of “pdf” or postscript reports • On-click mouse information • Windows/Linux portability • Station grouping mode Longer term (2011 -2012) • Inclusion of planning applications • Extension of benchmarking for annual averages (? ) • Inclusion of PM 2. 5 47

DELTA developments 48 FAIRMODE meetind, Norrkoping, June 2011 Model responses to emission reductions depend on the geographical location, the model scale, meteorological year… 1. Require a series of simulations with fixed emission reductions for main precursors (NOx, VOC, NH 3, SO 2, PPM) and analyze difference in behavior. Problem: • • No observations available Reference model ? Joint exercices 2. Analysis of spatio-temporal emission patterns in provided data (e. g week vs. week-end day, DEFRA 2011) DELTA expl. Mode (Links with SG 3)

Agenda FAIRMODE meetind, Norrkoping, June 2011 Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion User’s presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Benchmarking service Others (Ensemble…) Evaluation Datasets Review of possible TSAP Deliverables 49

50 FAIRMODE meetind, Norrkoping, June 2011 JRC USER Data Extraction Facility Model info Model results DELTA BENCHMARKING service Model performance Evaluation reports Deadline: end 2012

Agenda FAIRMODE meetind, Norrkoping, June 2011 Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion User’s presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Benchmarking service Others (Ensemble…) Evaluation Datasets Review of possible TSAP Deliverables 51

Deliverables for TSAP review FAIRMODE meetind, Norrkoping, June 2011 • Common template for reporting model performances for O 3, NO 2 and PM 10 as complement to the current RDE indicator. • Recommendations on quality objectives (performance criteria) to be achieved • Benchmarking procedure including the following elements: DELTA Benchmarking service ENSEMBLE + data preparation facility 52

53 FAIRMODE meetind, Norrkoping, June 2011 Annexes

Criteria & goals 54 FAIRMODE meetind, Norrkoping, June 2011 Air Quality (Regional scale modelling) Species Metric Criteria Goal Boylan and Russel, 2005, EPA report 2007 Main PM constituents (> 30% total mass), PM 2. 5 MFE MFB Minor PM constituents (< 30% total mass) Ozone 75% ± 60% Exp variations to reach 100% / 200% at 0 concentrations MFE MFB 35% 15% Evaluating the Performance of Air Quality Models, AEA (2009) Any pollutant FAC 2 NMB Half points within -0. 2 < MFB < 0. 2 Air quality model performances evaluation, Chang et Hanna (2004) NOx, CO, PM 10 FAC 2 FB NMSE Half points within -0. 3 < FB < 0. 3 NMSE < 4 50% ± 30%

How these criteria compares to RDE? (2) FAIRMODE meetind, Norrkoping, June 2011 • RDE takes closest to exceedance values. In the case the max value does not exceed the threshold, it is relative to the closest one. Lack of comparability of this measure among stations • MFB can be taken as 60%, 30% … No change because low values are within range of stable MFB • No assumption on station representativness • Screen monitoring data and drop extreme artificial events (e. g. fireworks) • Adapt templates for annual values 55

SG 4 participants - Eo. I 56 FAIRMODE meetind, Norrkoping, June 2011 Person Institute Marco BEDOGNI AMAT (IT) Jana MATEJOVICOVA HMI (SK) Bertrand BESSAGNET INERIS (FR) Anna MIRANDA University of Aveiro (PT) David CARRUTHERS CERC (UK) Dietmar OETTL Government of Styria (AT) Koen DE RIDDER VITO (BE) Christakis PAPADOPOULOS DLI (CY) Dick DERWENT AEA (UK) Guido PIROVANO ERSE (IT) Hilde FAGERLI Met Inst (NO) Peter ROBERTS CONCAWE (BE) Alenka FRITZEL Ministry of Environment (SI) Martijn SCHAAP TNO (NL) Stefano GALMARINI EC-JRC, Ispra Camillo SILIBELLO ARIANET (IT) Emilia GEORGIEVA EC-JRC, Ispra Alexandros SIRAKOS UOWM (GR) Maria GONCALVES BSC (ES) Philippe THUNIS EC-JRC, Ispra Pernige GRENNFELT IVL (SE) Keith VINCENT AEA (UK) Ari Karppinen FMI (FI) Rahela ZABKAR University of Ljubljana (SI) Fernando MARTIN CIEMAT (ES) John BARTZIS UOWM (GR) Jose BALDASANO BSC (ES) Nicolas MOUSSIOPOULOS AUT (GR) John DOUROS AUT (GR) Leonor TARRASON NILU (NO) Bruce DENBY NILU (NO) H. BECKSTROEM SMHI (SE) Joanna STRUZEWSKA WOT (PL) Nutthida KITWIROON HERTS (UK) Marcus Hirtl ZAMG(AT) Ruben Beijk RIVM (NL) Joost Wesseling RIVM (NL)

User Name FAIRMODE meetind, Norrkoping, June 2011 User affiliation Questionnaire Presentation Giudo Pirovano RSE Mihaela Mircea ENEA Marcus Hirtl ZAMG Addo Van Paul RIVM Dietmar Oettl Graz University Rafael Borge UPM David Carruthers CERC Suzanne Yongen VITO Kevin Delaney Irish EPA Stefan Andersson SMHI Claudio Carnevale Brescia University Ana Miranda Aveiro University Helge Olesen DMU V V V v x ? v v v V x V V V x x ? v x V v 57