Evaluating uncertainty predictions using an ensemble of CMAQ

Evaluating uncertainty predictions using an ensemble of CMAQ model configurations Robert W. Pinder, Alice B. Gilliland, Robert C. Gilliam, K. Wyat Appel Atmospheric Modeling Division, NOAA Air Resources Laboratory, in partnership with USEPA National Exposure Research Laboratory 2007 CMAS Conference October 2, 2007

Objective • Uncertainty: what is the likelihood that the observed value is within a given range? • Applications: § Exposure studies § Diagnostic evaluation tool • One measure of uncertainty is the error when the model is compared with observations. • Can we use an ensemble approach to make a better estimate of this range? Histogram of CMAQ O 3 Model Error 8 -hour max O 3, 228 AQS Sites from the SE US

Sources of Uncertainty Structural Uncertainty: VOC species lumping Physical processes Approach: vary representation Parameter Uncertainty: Emissions Meteorology Chemical rate constants Approach: Monte Carlo methods Challenge: Monte Carlo methods are not feasible given CMAQ’s computational requirements.

Method • Variety of CMAQ / MM 5 model configurations • Direct sensitivity calculations • Use observations to remove spurious ensemble members

Generate Ensemble Members for Structural Uncertainty using Multiple Model Configurations Planetary Boundary Layer / Land Surface Model § Pleim-Xiu Land Surface Model; ACM: Asymmetric Convective Model (Pleim and Chang, 1992) § Miller-Yamada-Janjic (Janjic, 1994), NOAH Land Surface Model § Medium Range Forecast (Hong and Pan, 1996), NOAH Land Surface Model Chemical Mechanism § Carbon Bond IV § SAPRC-99 Six structural uncertainty cases

Generate Ensemble Members for Parametric Uncertainty using Direct Sensitivity Calculation Use the Direct Decoupled Method (DDM) to calculate sensitivity to: • NOx Emissions • VOC Emissions • Second-order sensitivity • O 3 Boundary conditions At each grid cell, calculate ozone response to emissions and boundary concentrations Compared to brute-force calculation, errors are 5 -10% (Cohan et al. , 2005)

Direct Calculation of Ozone Sensitivity July 16, 2002, 2 PM O 3 (ppb) >30 25 20 15 10 5 0 -5

Use Observations to Constrain Ensemble AQS O 3 Monitoring Sites Used to evaluate boundary conditions Used to evaluate ensemble quality

Structural uncertainty simulations (6) Use DDM to calculate O 3 sensitivity to NOx, VOC, and boundary conditions. Use observations to remove spurious ensemble members Repeat 200 times Randomly sample from range of uncertain NOx emissions, VOC emissions, boundary concentrations, and structural uncertainty cases Generate an ensemble member by calculating the O 3 field across SE US domain

Example: Atlanta, Georgia July 1 -28, 2002

Structural Uncertainty

Structural + Parametric Uncertainty Spread is large – can we use the observations to narrow this range?

Prune ensemble members not consistent with observations Remove ensemble members that do not constrain the range

Pruned ensemble has narrow range while still including observations 200 member ensemble 10 member ensemble

Compare with +/-30% Range is 40% lower ± 30% of base case CMAQ 10 member ensemble

Analysis at all sites AQS O 3 Monitoring Sites Dataset: • 38 locations, 28 days • 1064 observations Evaluation: • Randomly reserve 50% of dataset • Derive ensemble, prune using half of observations • Evaluate using the reserve dataset § Ensemble range includes 85% of observations § Range is 40% smaller than ± 30% of base case

Trade-off between coverage of observations and spread in range Ensemble Size Observations within Range Average Ensemble Range 10 70% ± 12% 20 72% ± 12% 50 75% ± 15% 100 83% ± 18% 200 85% ± 18% ± 30% CMAQ 91% ± 30%

Conclusions • This ensemble generation and pruning technique provides a more robust uncertainty range: § Observed value is within the range 85% of the time § 40% reduction in spread compared to +/- 30% rule • Simultaneously narrowing these bounds and improving the performance depends on reducing structural errors in CMAQ • Locations and times that fall outside of the ensemble range should be targeted for uncovering structural errors in the model

Acknowledgements: Sergey Napelenok, Jenise Swall, Kristen Foley DISCLAIMER: The research presented here was performed under the Memorandum of Understanding between the U. S. Environmental Protection Agency (EPA) and the U. S. Department of Commerce's National Oceanic and Atmospheric Administration (NOAA) and under agreement number DW 13921548. This work constitutes a contribution to the NOAA Air Quality Program. Although it has been reviewed by EPA and NOAA and approved for publication, it does not necessarily reflect their policies or views.