Visualizing Uncertainty in Mesoscale Meteorology APL Verification Methodology

Visualizing Uncertainty in Mesoscale Meteorology APL Verification Methodology 21 May 2002 Scott Sandgathe

A New Paradigm for Weather Forecasting Automated Meteorological Information Evaluation System Human Forecaster Decision Interface Decision Driven Auto-Product Generation Decision Interface is comprised of: Verification of Global and Meso Models past forecasts Evaluation of Current Met. Analyses Rule-based evaluation of model predictions MURI/UW 2/13/02

Verification and Evaluation NOGAPS ETA centroid Global and Regional RMSE Evaluation – Recent, Historical and Synoptic MM 5 Output MM 5 Eval. MM 5 evaluation Wt. evaluation Evaluation of Ensemble members and combinations – recent, historical and synoptic Ensemble Products Wt. Evaluate current analyses based on observations, satellite and radar Reform ensemble or chose “mostrepresentative” member based on user evaluation Products are automatically generated based on user-selected ensemble or member.

Global RMS Error 00 Z 06 Z 12 Z 18 Z OBS OBS Analysis Forecast 72 -120 hr Forecast 6 hr Forecast 72 -120 hr Forecast 6 hr 00 Z. . . OBS Analysis Forecast 72 -120 hr

Regional RMS Error Global models are often “tuned” to the region of national interests or to the predominate natl weather pattern, and global skill may not reflect regional skill. The region between 110 E and 110 W accounts for the majority of 0 – 48 h weather.

Mesoscale Verification O O F All equivalent? ! POD=0, FAR=1 O (Brown, 2002) F F

Mesoscale Verification POD >> 0, FAR < 1 OO F Is this a better forecast? Or is this? O F (After Brown, 2002)

Mesoscale Verification

Mesoscale Verification • Total Error = Displacement Error + Amplitude Error + Residual Error – MSE and CCF give equivalent results – Hoffman, et. al. , 1995, for satellite data assimilation. • MSE(tot) = MSE(disp) + MSE(amp) + MSE(pattern) – Elbert and Mc. Bride, 2000, for precipitation pattern verif. • Implementation (Du and Mullen, 2000): – Calculate MSE(tot) = (Forecast – Analysis)2 – Shift forecast field to minimize total MSE and calculate MSE(disp) = MSE(tot)-MSE(shift) – Adjust amplitude to minimize MSE(shift). MSE(amp) = MSE(shift) – MSE(shift-min) – MSE(residual) = MSE(tot) – MSE(disp) – MSE(amp) • MSE(res) =? MSE(pattern) =? MSE(rotation)

Phase Shift

Phase and Amplitude Error

Rotational Error

Total Error

Future Research Issues • Need to test on “real” data. • Many computational solutions: – Correlation Coef. , Mean absolute difference, etc. – Rapid ‘image motion’ search techniques • Map verification or “feature” verification: – Phase and amplitude suitable for both – Rotation requires “feature” and more complex search • Need to examine usefulness • Evaluation of “goodness” – Relative weight of phase vs amplitude vs rotational err – Will test “table” approach often seen in software or “service” evaluation.

Questions and Comments?

References • Hoffman, R. N. , Z. Liu, J. -F. Louis, and C. Grassotti, 1995: Distortion representation of forecast errors. Mon. Wea. Rev. , 123, 2758 -2770. • Brown, B. , 2002: Development of an Object-based Diagnostic Approach for QPF Verification. USWRP Science Symposium, April 2002. • Ebert, E. E. , and J. L. Mc. Bride, 2000: Verification of precipitation in weather systems: determination of systematic errors. J. Hydro. , 239, 179 -202. • Du, J. , and S. L. Mullen, 2000: Removal of Distortion Error from an Ensemble Forecast. Mon. Wea. Rev. , 128, 3347 -3351. • Chan, E. , 1993: Review of Block Matching Based Motion Estimation Algorithms for Video Compression. CCECE/CCGEI. • Lim, D. -K. , and Y. -S. Ho, 1998: A Fast Block Matching Motion Estimation Algorithm based on Statistical Properties of Object Displacement. IEEE.

BACKUP SLIDES • SLIDES FROM 31 Jan 2002 Meeting

36 km Ensemble Mean and Selected Members SLP, 1000 -500 mb Thickness 2002 Jan 2200 Z

12 km Ensemble Mean and Selected Members SLP, Temperature, Wind 2002 Jan 2200 Z

Verification of Mesoscale Features in NWP Models Baldwin, Lakshmivarahan, and Klein 9 th Conf. On Mesoscale Processes, 2001

Tracking of global ridge-trough patterns from Tribbia, Gilmour and Baumhaufner

Current global forecast and climate models produce ridgetrough transitions; however, the frequency of predicted occurrence is much less than the frequency of actual occurrence

Creating Concensus From Selected Ensemble Members - Carr and Elsberry

Necessary Actions for Improved Dynamical Track Prediction Small Spread (229 n mi) Large Error No forecaster reasoning possible. Help needed from modelers and data sources to improve prediction accuracy (48 h) Large Spread (806 n mi) Large Error Recognize erroneous guidance group or outlier, and formulate SCON that improves on NCON Small Spread (59 n mi) Large Spread (406 n mi) Small Error No forecaster reasoning required -- use the non-selective consensus (NCON) Recognize situation as having inherently low predictability; must detect error mechanisms in both outliers to avoid making SCON>>NCON

References Cannon, A. J. , P. H. Whitfield, and E. R. Lord, 2002: Automated, supervised synoptic mappattern classification using recursive partitioning trees. AMS Symposium on Observations, Data Assimilation, and Probabilistic Prediction, p. J 103 -J 109. Carr. L. E. III, R. L. Elsberry, and M. A. Boothe, 1997: Condensed and updated version of the systematic approach meteorological knowledge base – Western North Pacific. NPS-MR-98 -002, pp 169. Ebert, E. E. , 2001: Ability of a poor man’s ensemble to predict the probability and distribution of precipitation. Mon. Wea. Rev. , 129, 2461 -2480. Gilmour, I. , L. A. Smith, R. Buizza, 2001: Is 24 hours a long time in synoptic weather forecasting. J. Atmos. Sci. , 58, -. Grumm, R. and R. Hart, 2002: Effective use of regional ensemble data. AMS Symposium on Observations, Data Assimilation, and Probabilistic Prediction, p. J 155 -J 159. Marzban, C. , 1998: Scalar measures of performance in rare-event situations. Wea. and Forecasting, 13, 753 -763.

Current Forecast Paradigm

NOGAPS ETA centroid MM 5 Output MM 5 Eval. MM 5 J 2 EE Control/ Interface Bean evaluation Wt. evaluation J 2 EE Control/ Interface Bean Ensemble Products Wt. J 2 EE Control/ Interface Bean Java Server Pages for each Bean interface Control and Server-side components Server Protocols (HTTP, RMI, CORBA. . ) IMS & Viz. Tools (XIS) Stat. tools Meteorology tools

Forecaster-in-the-Loop Concept

A New Paradigm (Bob’S)