PP INSPECT report Dmitry Alferov 1 Elena Astakhova

PP INSPECT report Dmitry Alferov (1), Elena Astakhova (1), Petra Baumann (4), Dimitra Boukouvala (2), Anastasia Bundel (1), Ulrich Damrath (3), Pierre Eckert (4), Flora Gofa (2), Alexander Kirsanov (1), Xavier Lapillonne (4), Joanna Linkowska (5), Chiara Marsigli (6), Andrea Montani (6), Anatoly Muraviev (1), Elena Oberto (7), Maria Stefania Tesini (6), Naima Vela (7), Andrzej Wyszogrodzki (5), and Mikhail Zaichenko (1), André Walser (4) (1) RHM (a. bundel@gmail. com), (2) HNMS, (3) DWD, (4) MCH, (5) IMGW-PIB, (6) ARPA-SIMC, (7) ARPA-PT COSMO GM, 11 -15 September 2017, Jerusalem, Israel

Spatial methods Filtering methods Many√ • Neighborhood (Ebert, 2008) Displacement methods • Features-based √ • Scale Decomposition DIST method √√ • √√ √ √√√ üContiguous Rain Area (CRA) (Ebert and Mc. Bride, 2000) üMethod for Object-based Diagnostic Evaluation (MODE) (Davis et al. , 2006) ü Structure, Amplitude, Location (SAL) (Wernli et al. , 2008) • Field deformation Now almost all the categories of spatial methods are applied by INSPECT participants. Flora Gofa: “It gave us a chance to learn methods that before we just knew by name”. Thus, the main benefit of INSPECT “that the wide range of spatial verification methods available will become commonly used within the COSMO community” is achieved.

Status highlights • The PP INSPECT is extended until the end of 2017 due to delays in some tasks, with 0. 3 FTE shifted and 0. 2 FTE added • Research tasks are mostly completed except for some tasks involving ensembles. • The reports describing the properties of each spatial method are written by participants. The report summarizing them is under preparation.

Reruns for Meso. VICT test cases • MCH: COSMO-1 reruns for ALL Meso. VICT cases are done and available at WG 5 repository (Petra Baumann) • ECMWF-IFS reruns (51 member) for cases 1 and 2 (8 initial dates) (Andrea Montani), • COSMO-E reruns (21 member) for cases 1 and 2 (8 initial dates) (André Walser), • COSMO-Ru 2 -EPS (51 member) for case 1 (1 initial date) and case 2 (is running now) (Dmitry Alferov) All reruns are interpolated to VERA grid by Manfred Dorninger (Austria) -> easy to use and compare!

Tasks involving development of routines for neighborhood, CRA, SAL, and MODE • For the most part, the software is based on free R Spatial. Vx package (developed by E. Gilleland). For SAL (D. Boucoucala) and Neighborhood (J. Linkowska) comparisons are made with alternative packages -> bug fixing of Spatial. Vx • VAST development by N. Vela (ARPA-PT): inclusion of time dimension and variables besides pecipitation, namely, cloud cover and wind speed

Neighborhood applications. Analysis of long time series (DWD, MCH) Relax the requirement for an exact match by evaluating forecasts in the local neighborhood of the observations

Uli Damrath’s method to aggregate the scores • Getting Equitable Threat Score (ETS) for upscaling and Fractions skill score (FSS) as monthly values from fuzzy verification • No averaging over daily values but calculation of scores from the contingency tables of the whole month • Calculation of running means of the results over one year • Presentation of mean values and mean averages

Comparison of COSMO-EU to COSMO-DE – upscaling Equitable Threat Score (ETS) Threshold 0. 1 0. 2 0. 5 1 2 5 10 20 50 W i n d o w s I z e 1. 625 0. 825 0. 425 0. 225 0. 125 Mesh width of COSMO-EU 0. 075 0. 025 Precipitation amount

Compact visualization of total precipitation FSS: to focus on the useful scale for a given lead-time and threshold (MCH) FSS: Fractions Skill Score P is the event fraction in the neighborhood. Threshold Lead time 0. 1 mm/h 1 mm/h 2 mm/h 01 -12 2. 2 km 19. 8 km 59. 4 km 13 -24 2. 2 km 33. 0 km 99. 0 km

Intensity-scale: filtering method (F. Gofa, HNMS) Error image is expressed as the sum of components on different spatial scales by performing a 2 -dimentional discrete Haar wavelet decomposition. The spatial scale refers to the spatial scale of the error and not that of the precipitation features as it happens in the neighborhood methods. COSMO 1 Meso. VICT case 1: 20070621 -15, Intensity Scale Skill COSMO 2 Small scales have skill close to zero, while large scales exhibit large skill. COSMO 1 ISS graphs exhibit that errors due to displacements of small spatial scale features are more important compared to those of COSMO-2

Object-based methods by IMGW-PIB • MODE, CRA, SAL Case study VERA 2007. 09. 25. 06, 6 h precipitation, threshold>=5 mm COSMO 2 Selected feature pairings based on total interest obs feature mod feature total interest 1 1 0. 898 unmatched object (false)

SAL: Maria Stefania Tesini and Daniele D'Alessandro A single parameter to evaluate the structure, amplitude, or location error in forecast is not enough when the precipitation field complexity is too high Precipitation intensity is overestimated in Germany and underestimated in France, but the amplitude A is close to zero

Object matching for EPS, Meso. VICT case 1 (A. Bundel) Probability of each observed object is found and the ensemble skill can be estimated using the BSS, for example 2007062021 COSMO-E ensemble, first 6 of 21 members, precip threshold >0. 5 mm/1 h Probabilities of each of 5 observed objects: 1/21 20/21 19/21 14/21 Difficulty of such an approach: we have estimate only probabilities of objects CRA errors In spatial shift, precip volume, and fine-scale pattern can be

SAL for EPS: ways to display large amount of ensemble data (Dimitra Boucouvala) 20/6

SAL for EPS: introducing observation uncertainty (Dimitra Boucouvala) Objects comparison for probability of precipitation >= 2 mm Probability threshold =1 LEPS Observations Preci >= 2 mm for all 16 members 3 h Precipitation 21/6 12 UTC S=1, A=0. 38, L=0. 3

Andrea Montani: Sensitivity of COSMO-LEPS forecast skill to the verification network: application to Meso. VICT cases

New ensemble precipitation observation product by Meteo. Swiss • Available for the past data (e. g. for Mesovict cases) • Available for Swiss + whole alpine domain for daily accumulation

Main results • Several ways of compact visualization of long time series of neighborhood scores are proposed (DWD, MCH) • The object-based SAL (Structure-Amplitude-Location) method is easier to implement as it doesn’t require pair-wise matching of observed and forecast objects. However, it can give misleading results as it estimates characteristics in the whole domain, e. g. if precipitation intensity is overestimated in one place and underestimated in another, the amplitude value will be close to zero. • The object-based MODE and CRA methods provide more information as the matched pairs of observed and forecast objects are compared. However, it’s difficult to find a best universal matching function, in particular for high-resolution fields with objects of complex shape. • The DIST, SAL and CRA methods are applied to ensembles (ARPAE-SIMC, HNMS, RHM) • The experiments are started on introducing observation uncertainty into the spatial methods (ARPAE-SIMC, HNMS, MCH) The main benefit of INSPECT is achieved: the most wide-spread spatial methods enter into the everyday practice within the COSMO Community

THANK YOU FOR YOUR ATTENTION !