Methodology and preliminary results of the Hsaf precipitation

Methodology and preliminary results of the H-saf precipitation products validation Silvia Puca In collaboration with RMI (Belgium), BFG (Germany), OMSZ (Hungary), Uni. Fe and DPC (Italy), IMWG (Poland), SHMI (Slovakia), ITU TMS (Turkey) Dipartimento della Protezione Civile Italiana

outline • H-saf Precipitation Products program and members of Calibration and Validation Group; • H-saf Precipitation Products; • Precipitation ground measurements used for the validation; • Validation Methodology: common and specific validation; • Up-scaling techniques; • Preliminary feedback on AMSU and AMSU+SEVIRI products; • Conclusions; • Future steps. WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

Precipitation Product Validation Program: • characterize the product error structure whose knowledge is needed for correct utilisation; • supporting algorithms and models tuning (i. e. , calibration) during their development process; • collecting routine reporting from end-users and special reporting from experimental activities; • continuing calibration/validation activities during the preoperational phase. Workshop on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007, Geneva,

H-saf Precipitation Products Cal/Val Group WP-2300 Calibration/validation Italy (DPC) WP-2310 Cal/val Belgium WP-2320 Cal/val Germany WP-2330 Cal/val Hungary WP-2340 Cal/val Italy WP-2350 Cal/val Poland WP-2360 Cal/val Slovakia Belgium: Royal Meteorological Institute (IRM) Germany: Federal Institute of Hydrology (Bf. G) Hungary: Országos Meteorológiai Szolgálat-Távérzékelési Osztály- (HMS) Italy: Università di Ferrara (Uni. Fe), Dipartimento della Protezione Civile (DPC), Poland: Institute of Meteorology and Water Management (IMWM) Slovakia: Slovak Hydrometeorological Institute (SHMÚ) Turkey: Istanbul Technical University, Meteorology Department (ITU), Middle East Technical University (METU) WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007 WP-2370 Cal/val Turkey

H-saf Precipitation Products; Table 1 - List of H-SAF products and indication of the Units responsible of algorithm development. Cod Responsible Acronym Product name e of algorithm PR-OBS- Precipitation rate at ground by MW conical scanners (with Italy, CNRH-01 1 indication of phase) SSMI - CRDB ISAC PR-OBS- Precipitation rate at ground by MW cross-track scanners (with Italy, CNRH-02 2 indication of phase) AMSU - NN ISAC PR-OBS- Precipitation rate at ground by GEO/IR supported by LEO/MW Italy, CNRH-03 3 SSMI+AMSU+ SEVIRI - blending ISAC PR-OBS- Precipitation rate at ground by LEO/MW supported by GEO/IR Italy, CNRH-04 4 (with flag for phase) SSMI+AMSU+ SEVIRI - morphing ISAC PR-OBSItaly, H-05 Accumulated precipitation at ground by MW+IR and MW only 5 CNMCA PR-ASS- Instantaneous and accumulated precipitation at ground computed Italy, H-06 1 by a NWP model CNMCA WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

Precipitation ground measurements used for the validation In situ data: § automatic rain gauges with different time resolution: 5 min, 10 min, 15 min. , 30 min. § synoptic stations with different time resolution : 6 hour, 24 hour cumulated values; § Meteorological Radars with different time resolution: 5 min, 10 min, 15 min. , 30 min. Someone gauge adjusted Data for cloud type classification, containing information about water content in vertical column and for the discrimination of the synoptic situation: NWC-saf, NWP models, MSG composite image. WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

PPV Raingauge network is composed by 4100 stations: Data Sources Instrument characteristics raingauges Telemetric and mechanic time domain Near real time, case (near real time/ studies case studies) time resolution (15 min, 30 min) 10 – 30 min (telemetric), 3 – 24 h (mechanic) spatial distribution (whole national territory/ limited area) Whole national territory ~390 mechanic (RMI) + 12 number of station (please telemetric (RMI) + attach a map) 4160 telemetric (SETHY) Operational (RMI) operational/ for + research (other research only networks) Telemetric: automatically checked / data quality check mechanic: autom. + manually checked

PPV Radar network is composed by 40 C-band: Data Sources radars Beam width ~1°, max range ~150 Km, 250 m, Instrument characteristic C-band, single polarization, s Doppler polarimetric time domain near real time/ case studies time resolution 5 min, 15 min, 30 min, 1 h, 24 h spatial distribution Whole national territory number of station 33 C band +1 Ka band operational/ for research only Operational data quality check Permanent ground clutter removed; monitoring of electronic calibration

Validation Methodology: common and specific validation Common methodology: • rain gauges and radar data; • multi categorical and continuous statistics evaluated on long time series; Contingency tables and errors are evaluated in the same way by all the institutes involved using the same up-scaling techniques. Statistical error evaluated by all the institute in the same way on long time series Each Institute in addition to the common validation methodology has developed a specific validation methodology based on its own knowledge and experience. • lightning data, numerical weather prediction and nowcasting product • case studies: convective/stratiform precipitation, day/night, land/ocean deep analysis met. events, important Calibration task WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

Common validation Continuous verification statistics: calculating Mean absolute error, root mean square error, correlation coefficient, standard deviation etc; Multi-Categorical statistics: calculating the contingency table (which allows for evaluation of false alarm rate, probability of detection, equitable threat score, Heidke skill score, etc ). All the tools used for this validation are automatic tool. WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

Multi-Categorical statistics For the rain rate (RR) derived from MW imagery (spatial resolution of 16 -40 Km) the precipitation classes of the multi categorical statistics were defined following the hydrologist, developers requirements and project plan: Project Plan Developers request Class 1 RR ≤ 0. 25 mm/h Class 1 RR < 1 mm/h Class 2 1 mm/h ≤RR < 10 mm/h Class 3 10 mm/h ≤ RR Class 2 0. 25 mm/h < RR ≤ 0. 5 mm/h Class 3 0. 5 mm/h < RR ≤ 1 mm/h Radar classes Class 4 1 mm/h < RR ≤ 2 mm/h Class 1 Class 5 2 mm/h < RR ≤ 4 mm/h Class 2 0. 5 mm/h < RR ≤ 2 mm/h Class 6 4 mm/h < RR ≤ 8 mm/h Class 3 2 mm/h < RR ≤ 10 mm/h Class 7 8 mm/h < RR ≤ 16 mm/h Class 4 10 mm/h < RR ≤ 16 mm/h Class 8 16 mm/h < RR ≤ 32 mm/h Class 5 8 mm/h < RR ≤ 16 mm/h Class 9 32 mm/h < RR ≤ 64 mm/h Class 6 16 mm/h < RR ≤ 32 mm/h Class 10 64 mm /h < RR Class 7 64 mm /h < RR RR ≤ 0. 5 mm/h WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

H-saf Precipitation Products; Common validation: PR-OBS-2 v 1. 1: August 2006 - August 2007; (AMSU: neural network algorithm trained on radar data (NEXRAD)) PR-OBS-2 v 2. 1: August 2006 - August 2007; (AMSU: neural network algorithm trained on numerical model (MM 5)) PR-OBS-3 v 1. 0: June 2007 - August 2007 (AMSU + IR: Blending techniques) Specific validation: 12 case studies selected by the involved institues. Radar, rain gauge, convective detection, numerical model data have been used. Descrimination between convective/stratiform precipitation, day/night, land/ocean: • PR-OBS-1 v 1. 0 (Rain Rate: SSMI) • PR-OBS-2 v 1. 1 (Rain Rate: AMSU) • PR-OBS-2 v 2. 1 (Rain Rate: AMSU) • PR-OBS-3 v 1. 0 (Rain Rate: AMSU + SEVIRI) • PR-OBS-5 v 1. 0 (Cumulated Precipitation: AMSU + SEVIRI) WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

UP-SCALING on SATELLITE NATIVE GRID: A common up-scaling method has been applied to raingauge and radar data vs AMSU -B grid: a Gaussian filter, Take into account that precipitation data in the AMSU retrieval product (H 02) follows the scanning geometry and IFOV resolution of AMSU-B scan, so that each pixel along the scan has a precipitation value representative for an elliptical region with different size. WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007 Fig 1 – Left) Gaussian filter – Right) section of gaussian filter. If the Radar resolution is 1 km, 1 px=1 km

Precipitation Product: PR-OBS-02 v 11 type of observation: RADAR date: 200608 ATION CLASSES REQUIRED BY PROJECT PLAN , 923 0, 204 0, 005 , 028 0, 893 0, 999 , 900 0, 917 0, 966 , 950 1, 913 3, 269 0, 634 0, 058 0, 026 899 0, 075 0, 001 , 102 0, 053 -0, 005 288 0, 146 -0, 021 , 185 0, 101 -0, 010 892 4, 135 0, 175 ATION CLASSES SIMILAR TO RADAR CLASSES , 913 0, 033 0, 026 0, 032 0, 000 , 039 0, 971 0, 987 1, 000 , 883 0, 936 0, 970 0, 966 0, 971 0, 996 0, 999 , 950 1, 138 2, 013 2, 471 3, 116 4, 210 5, 646 0, 596 0, 035 0, 020 0, 025 0, 022 0, 003 0, 001 880 0, 015 0, 009 0, 000 , 130 -0, 001 0, 002 -0, 005 -0, 001 -0, 000 317 -0, 003 0, 006 0, 007 -0, 022 -0, 003 -0, 001 , 230 -0, 002 0, 004 -0, 011 -0, 000 101 0, 924 1, 326 1, 305 0, 001 0, 000 : observations satellite number of pixels : 54906 raining points (rr>0, 25 mm/h): 6685 4455 class more frequent : 2 5 max class : 10 10 PRECIPITATION CLASSES REQUIRED BY DEVELOPERS POD 0, 901 0, 005 0, 009 0, 017 0, 026 0, 017 0, 003 0, 000 FAR 0, 057 0, 993 0, 990 0, 988 0, 987 0, 993 0, 999 1, 000 ACC 0, 859 0, 962 0, 966 0, 969 0, 970 0, 974 0, 977 0, 986 0, 996 BIAS 0, 956 0, 717 0, 938 1, 410 2, 013 2, 462 2, 678 3, 579 4, 210 POFD 0, 618 0, 016 0, 017 0, 018 0, 020 0, 019 0, 017 0, 011 0, 003 CSI 0, 855 0, 003 0, 005 0, 007 0, 009 0, 005 0, 001 0, 000 ETS 0, 130 -0, 006 -0, 004 -0, 001 0, 002 -0, 001 -0, 004 -0, 002 -0, 001 HK 0, 283 -0, 011 -0, 007 -0, 002 0, 006 -0, 002 -0, 014 -0, 011 -0, 003 HSS 0, 230 -0, 013 -0, 008 -0, 002 0, 004 -0, 001 -0, 008 -0, 005 -0, 001 OR 5, 631 0, 324 0, 558 0, 897 1, 326 0, 905 0, 186 0, 000 1, 000 0, 999 5, 646 0, 001 0, 000 -0, 001 -0, 000 CONTINGENCY TABLE 45467 989 736 572 351 87 20 0 0 722 7 5 11 60 58 19 2 0 0 652 11 9 18 50 98 67 11 0 0 683 18 16 12 22 117 91 42 9 0 759 29 33 14 14 48 114 69 21 3 777 65 62 29 10 7 35 38 10 4 752 64 60 33 18 1 1 0 2 1 456 37 44 20 19 4 1 0 0 0 151 7 9 5 4 0 0 0 33 6 3 1 1 0 0 0 WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007 0

Preliminary results of the common validation on OBS 02 v 1. 1 v OBS 02 v 2. 1: Data used: Radar and rain gauge Period : August 2006 -August 2007, Score : Accuracy

Preliminary results of the common validation on OBS 02 v 1. 1 v OBS 02 v 2. 1: Data used: Radar and rain gauge Period : August 2006 -August 2007, Score : Probability Of Detection

Preliminary results of the common validation on OBS 02 v 1. 1 v OBS 02 v 2. 1: Data used: Radar and rain gauge Period : August 2006 -August 2007, Score : False alarm ratio

Table 8 - Multi category scores of PR-OBS-2 v 1 evaluated over one year of radar and rain gauge data for three precipitation classes radar PR-OBS-2 v 1 rain gauges class 1 class 2 class 3 Accuracy 0. 97 0. 99 Probability Of Detection 0. 96 0. 20 0. 01 0. 98 0. 13 0. 17 Critical Success Index 0. 92 0. 07 0. 00 0. 96 0. 07 0. 03 False-alarm rate 0. 02 0. 82 0. 99 0. 02 0. 78 0. 96 Frequency Bias 0. 98 2. 41 2, 47 0. 98 1. 3 5, 42 Table 7 - Minimum, mean and maximum statistical scores of PR-OBS-2 v 1 evaluated over one year of radar and rain gauge data PR-OBS-2 v 1 radar rain gauges min mean max Mean error (mm/h) -0. 02 0. 18 0. 77 -3. 67 -1. 21 3. 81 Mean absolute error (mm/h) 0. 05 0. 29 0. 92 1. 60 3. 46 8. 05 MSE (mm/h)2 0. 01 1. 32 7. 37 10. 81 62. 32 281. 11 RMSE (mm/h) 0. 08 1. 10 3. 83 2. 51 5. 67 12. 32 Multiplicative bias 0. 20 1. 55 5. 02 0. 11 1. 55 6. 03 Correlation coefficient 0. 07 0. 25 0. 42 -0. 08 0. 09 0. 44 WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

Table 10 - Multi category scores of PR-OBS-2 v 2 evaluated over one year of radar and rain gauge data for three precipitation classes PR-OBS-2 v 2 radar rain gauges class 1 class 2 class 3 Accuracy 0. 97 0. 99 0. 93 0. 99 Probability Of Detection 0. 97 0. 23 0. 05 0. 94 0. 28 0. 06 Critical Success Index 0. 96 0. 00 0. 93 0. 12 0. 05 False-alarm rate 0. 01 0. 92 0. 99 0. 02 0. 84 0. 80 Frequency Bias 0. 95 3. 74 12. 90 0. 96 3. 6 0. 26 Table 9 - Minimum, mean and maximum statistical scores of PR-OBS-2 v 2 evaluated over one year of radar and rain gauge data PR-OBS-2 v 2 radar rain gauges min mean max Mean error (mm/h) 0. 02 0. 09 0. 29 -0. 56 0. 13 1. 01 Mean absolute error (mm/h) 0. 08 0. 17 0. 44 0. 23 0. 53 1. 12 MSE (mm/h)2 0. 09 0. 32 0. 80 5. 43 16. 5 RMSE (mm/h) 0. 19 0. 37 0. 79 0. 89 1. 92 3. 34 Multiplicative bias 0. 9 2. 34 11. 3 0. 94 4. 61 21. 6 Correlation coefficient 0. 08 0. 28 0. 47 -0. 03 0. 15 0. 41 WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

Preliminary results of the common validation on OBS 03 v 1. 01: Data used: rain gauge only Period : June 2007 -August 2007, Score : Accuracy, Heidke skill score

Preliminary results of the common validation on OBS 03 v 1. 01: Data used: rain gauge only Period : June 2007 -August 2007, Score : Probability of Detection, False Alarm Ratio

Preliminary results of the common validation on OBS 02 v 1. 1 v OBS 02 v 2. 1: Data used: Radar and rain gauge Period : August 2006 -August 2007, Score : Corr. Coef. , Mean absolute error , Mean Error

Preliminary results of the common validation on OBS 02 v 1. 1 v OBS 02 v 2. 1: Data used: Radar and rain gauge Period : August 2006 -August 2007, Score : Root Mean Sq. Error, Standard Deviation, Multi. Bias

AMSU + SEVIRI: blending technique Table 12 - Multi-category scores of PR-OBS-3 evaluated over three months of rain gauge data for three precipitation classes PR-OBS-3 raingauge Table 11 - The mean statistical scores of PR-OBS -3 evaluated over three months rain gauge data PR-OBS-3 Rain gauges Mean error (mm/h) 0. 45 0. 68 0. 95 Mean absolute error (mm/h) 0. 07 MSE (mm/h)2 25. 6 RMSE (mm/h) 4. 96 Multiplicative bias 4. 67 Standard deviation (mm/h) 3. 12 class 1 class 2 class 3 Accuracy 0. 81 0. 86 Probability Of Detection 0. 86 0. 18 Critical Success Index 0. 83 0. 06 0. 03 False-alarm rate 0. 06 0. 90 0. 94 0. 92 presented 2. 29 4. 29 The scores here were evaluated using precipitation 0. 11 and Correlation coefficient no-precipitation data. The large amount of no-precipitation data influenced the statistical results!!! Frequency Bias WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

Preliminary results on OBS-02 and OBS-03 from statistical error evaluation and case studies analysis • Precipitation clusters are well detected by both OBS 2 v 11, OBS 2 v 21 and OBS 3 v 10; • OBS 02 v 1 seems underestimate the RR more than OBS 2 v 2 (ME and MAE better v 2 than v 1); • Similar values of accuracy, FAR and POD for both (2 v 1 that 2 v 2). V 1 is very good for class 1 better during the autumn and winter periods than summer one; • Frontal cases are well described in both obs 02 v 1 and obs 02 v 2, but the RR is more realistic in obs 02 v 1. 0; • obs 02 v 2. 0 is better during the summer period, in detecting very high convective cells and convective systems than low rain rate ; • obs 02 v 2. 0 some times detects precipitation where clouds are not present, background problems; • H 03 good performances for class 1 but not enough data are available for a significant statistical validation (raingauge only); WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

It is important to take into account that: - The quality of H 02 products depends on the pixel number: the closer to the ends of AMSU scan line, the lower quality. - Majority of radars used for validation are not gauge adjusted; - time sampling of radar and raingauge used are different (10 min 15 min, 30 min); - interpolated gridpoints are not at same altitude of the nearby stations; - orographic precipitation enhancement depends on meteorological structure; - Different channels are used from the same product OBS 2 v 2. 1 for different satellite pass. WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

Conclusions: • Radar and raingauge archive from October 2006 for all the institutes involved in the PP Cal. Val is available; • Common up-scaling techniques and share of procedures; • Preliminary results on OBS 02 and OBS 03 indicate a a good discrimination between rain/no-rain but a general underestimate of the precipitation rate; Remarks: • Which precipitation classes can the mw and infrared data discriminate? Work on high precipitation rate. • Standardization of radar and raingauge data is necessary. WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

Future steps: • Take into account the orography; • Add new classes as land or sea, day or night also in the common validation; • automatization of validation procedure described; • Parallax and geolocation errors should be accounted; • Quality control of the data used for validation (radar, raingauges); • Evaluation of continuous and multi-category statistic on all products: H 01, H 02, H 03 and H 05; WS on the Evaluation of High Resolution Precipitations Products, WMO 3 -5 December 2007

Visiting Scientist on refinement and operational implementation of a rain rate algorithm based AMSU/MHS, and SEVIRI data within the Hydrological-SAF Antonelli P. , Bennartz R. Puca S. , Tassa A. , Zauli F. NEFODINA 01: 45 AMSU 01: 52 paoloa@ssec. wisc. edu http: //lupin. ssec. wisc. edu/paoloa/HSAFoper/ NEFODINA 02: 00 AMSU 01: 52 CO RR (01: 45 UTC)=48 mm/hr RADAR on SEVIRI GRID CO RR (02: 00 UTC)=38 mm/hr Radar RR=42 mm/hr Redistribution of AMSU-B averaged RR ifov on SEVIRI under the hypotesis that most of the precipitation occores in convective region

Thank you!!