Monitoring of the observing system at ECMWF Cristina
Monitoring of the observing system at ECMWF Cristina Prates 2 nd WIGOS Workshop on Data Quality Monitoring and Incident Management 15 -17 December, Geneva, Switzerland © ECMWF September 24, 2020
Outline • Importance of Observation Monitoring for NWP • ECMWF Monitoring capabilities: – Automatic data checking – Alert messages – Geographical coverage, time series of area averages and time-averaged geographical mean – Blacklisting/Whitelisting • WIGOS pilot project on data quality monitoring • Summary 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 2
Why monitoring the quality and availability of observations is essential for NWP? NWP models rely on Data Assimilation (DA) systems to provide them with the initial atmospheric state (analysis) necessary for the integration of the model equations that can lead to the best prediction. DA systems are based on methods that combine prior knowledge of the atmosphere (First Guess and Background) with observations in a optimal way taking into account statistical information about the errors of both pieces of information (Kalney, 2003). The quality and availability of observations impact the quality of the analysis therefore it is important an early detection of any observational data issue that can potential degrade the analysis. 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 3
What type of system is required? DA systems have the ability to provide diagnostic facilities to monitor the performance of the observational network (Hollingsworth et al. , 1986). Volume of data used by NWP DA systems is huge: ECMWF operational 12 -hour assimilation cycle can have as many as 90 million pieces of data available. Wide variety of observations: in situ measurements (groundbased or aircraft) and remote sensing space-borne platforms (satellites). 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 4
What type of system is required? DA systems have the ability to provide diagnostic facilities to monitor the performance of the observational network (Hollingsworth et al. , 1986). Volume of data used by NWP DA systems is huge: ECMWF operational 12 -hour assimilation cycle can have as many as 90 e million pieces of data available. dur ce o r p l Wide variety of observations: in situ measurements (grounda u n based or aircraft) and remote sensing ma space-borne platforms a on (satellites). y l le o s ely r t o n n ca e W 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 5
ECMWF Monitoring capabilities Automatic data checking system to trigger warnings if there is a sudden change in quality or availability of data actively assimilated by 4 D-Var system (started with satellite observations in 2008 and extended to in situ measurements). The selected statistic quantities are checked against an expected range. The alert message is sent if statistics are outside the specified limits. ------ Soft limits (calculated from past statistics) _______ (fixed) Hard limits Average of the bias correction Alert Standard deviation of first-guess departures Average of first-guess departures Number of observations 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 6
ECMWF Monitoring capabilities Automatic data checking for in situ observations: PGE, mean and rootmean-square of innovations and bias correction. Each Individual Station ID • I Probability of Gross Error (PGE) PGE > 0. 75 triggers a warning. Average of bias correction rms of FG and AN departures Parameters checked: Temperature, pressure humidity and wind. Average of first-guess departures Number of observations 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 7
ECMWF Monitoring capabilities Automatic data checking for in situ observations: PGE, mean and rootmean-square of innovations and bias correction. Geographical areas & WMO Block Each Individual Station ID W warnings Severity levels : Alert Slightly Considerable Severely persistent 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 8
Alert messages are sent and also published on ECMWF webpages 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 9
Alert messages are sent and also published on ECMWF webpages 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 10
Alert messages are sent and also published on ECMWF webpages Alert 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 11
Visualization of warnings (being tested) 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 12
ECMWF Monitoring capabilities 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 13
ECMWF Monitoring capabilities 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 14
ECMWF Monitoring capabilities 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 15
ECMWF Monitoring capabilities 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 16
ECMWF Monitoring capabilities 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 17
Blacklisting/whitelisting Monthly: per Station Id & parameter Quality of observations: First : a list is generated automatically by the data checking system, which checks the statistics computed for each station used (blacklisting) and also for the ones previously blacklisted (whitelisting). Second : a subjective analysis is done to decide if the automatic suggestion is right Updating station metadata: WMO catalogue station position & height is checked against google API information. New stations are always blacklisted and monitored by the data checking system before being used by DA Final: A list of Ids is produced and implemented operationally. 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 18
Monthly Monitoring Report • WMO monitoring of conventional observations based on monthly reports produced by leading Centres following the recommendations on Attachment II. 9 of WMO Manual GDPFS (No. 485). 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 19
WIGOS pilot project on data quality monitoring • The aim is to move towards a near-real-time (e. g. daily) monitoring of the status of the GOS in terms of availability and data quality, which would help WMO to take actions. Quality Information about SYNOP observations: every 6 hours Header 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 20
WIGOS pilot project on data quality monitoring • The aim is to move towards a near-real-time (e. g. daily) monitoring of the status of the GOS in terms of availability and data quality, which would help WMO to take actions. Quality Information about SYNOP observations: every 6 hours Header 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 21
WIGOS pilot project on data quality monitoring Wikipage at ECMWF https: //software. ecmwf. int/wiki/display/WIGOS+pilot+project+on+data+quality+monitoring 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 22
Summary • The ECMWF monitoring system was designed to ensure that DA system uses good quality observations. • Computes NWP-based statistics on quality and availability of the different components of the GOS. • Fully automated. • Alarm system triggered by few checks: send e-mails. • Timely detection of quality issues and data outages. • Influence the usage decision (blacklisting/whiteliting). • Monitor data impact on forecast and analysis. 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 23
Thank You for listening! Questions? References: Hollingsworth, A. , Shaw, D. , Lönnberg, P. , Illari, L. , Arpe, K. , & Simmons, J. (1986). Monitoring of observation and analysis quality by a data assimilation system. Amer. Meteor. Soc. , 114, 861 -879. Kalnay, E. (2003). Atmospheric modelling, data assimilation and predictability. Cambridge University Press. 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 24
Extra information Automated checking system limits Soft limits Updated automatically using statistics from last 20 days (last 2 days and extremes are excluded). Calculaded as: Upper soft limit=mean+5 x stdev Lower soft limit = mean-5 x stdev Hard limits Estimated during the process of adding new data and updated manually. Adjusted occasionally when a drift is noticed or during IFS upgrades. 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 25
Extra information Severity levels • Slightly: outside +- 5 stdev from the mean. • Considerable: outside +- 7. 5 stdev from the mean. • Severely: outside +- 10 stdev from the mean. • Severely persistent: Severe problems occurring frequently during the past 10 days. 2 nd WIGOS Workshop on Data QM & IM, 15 -17 Dec 2015, Geneva, Switzerland EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 26
- Slides: 26