Streamlining Uncertainty Conceptual Model and Scenario Uncertainty FRAMES2

Streamlining Uncertainty Conceptual Model and Scenario Uncertainty FRAMES-2. 0 Workshop U. S. Nuclear Regulatory Commission Bethesda, Maryland November 15 -16, 2007 Pacific Northwest National Laboratory Richland, Washington

Model Applications Regulatory and design applications of hydrologic models of flow and contaminant transport often involve using the models to make predictions of future system behavior l Performance assessment of new facilities (safety evaluation, environmental impact assessment) l Monitoring network design for contaminant detection or performance monitoring l License termination l Design of a subsurface contaminant remediation system 2 11/24/2020

New Reactor Potential Model Applications Assessing effects of accidental releases on ground and surface waters l groundwater flow pathways l transport characteristics Assessing flood design bases l Stream flooding l Local flooding, site drainage Impacts of water use l Watershed analysis – impacts on other users of water source upstream and downstream, particularly during drought conditions 3 11/24/2020

Framework for the Application of Hydrologic Models to Regulatory Decision Making Model Application for Comparison with Regulatory or Design Criteria Model Development & Evaluation History Matching - reproduce observed behavior l Demonstrate understanding of site behavior l Provide confidence in use of models to support decisions Prediction – forecast future behavior l Apply model results to decisions l For risk-informed decision making, provide estimates of risk 4 11/24/2020

Model Predictive Uncertainty Quantifies Element of Risk In general, uncertainty is assessed in the history-matching period and propagated into the predictive period l Reduce these uncertainties by collecting additional data Some uncertainties only apply in the predictive period l Irreducible characteristics of the system being modeled 5 11/24/2020

Prediction Uncertainty Model conceptualization uncertainty l A hypothesis about the behavior of the system being modeled and the relationships between the components of the system l Each site is unique and heterogeneous/variable. Behavior typically involves complex processes. Site characterization data is limited. l Assessed in history-matching period, applied in the predictive period Parameter uncertainty l Model-specific quantities required to obtain a solution l Measurement/sampling errors. Disparity among sampling, simulation, and actual scales of the system. l Assessed in history-matching period, applied in the predictive period Scenario uncertainty l Future state or condition that affects the hydrology l Historical record not representative of future conditions – process variability, limited historical record, land/water use changes, climate change l Applies to predictive period only 6 11/24/2020

Model Uncertainty Common to rely on a single conceptual model of a system. This approach is inadequate when there are: l different interpretations of data l insufficient data to resolve differences between conceptualizations 7 11/24/2020

Failure to Consider Model Uncertainty Has two potential pitfalls: l l rejection by omission of valid alternatives (underestimates uncertainty) reliance on an invalid model (produces biased results) 8 11/24/2020

Hydrogeologic Model Application (Ref. ) Comments Error Phoenix (Konikow 1986) Assumed past groundwater pumping would continue in future Scenario/ Conceptual Cross Bar Ranch Wellfield (Stewart and Langevin 1999) Assumed a 75 -day, no-recharge scenario would represent long-term maximum drawdown Scenario/ Conceptual Arkansas Valley (Konikow and Person 1985) Needed a longer period of calibration Scenario/ Parameter Coachella Valley (Konikow and Swain 1990) Recharge events unanticipated Scenario INEL (Lewis and Goldstein 1982) Dispersivities poorly estimated Parameter Milan Army Plant (Andersen and Lu 2003) Extrapolated localized pump test results to larger area Parameter Blue River (Alley and Emery 1986) Storativity poorly estimated Parameter/ Conceptual Houston (Jorgensen 1981) Including subsidence in model improved predictions Conceptual HYDROCOIN (Konikow et al. 1997) Boundary condition modeled poorly Conceptual Ontario Uranium Tailings (Flavelle et al. 1991) Inadequate chemical reaction model Conceptual Los Alamos (Bredehoeft 2005) Flow through unsaturated zone not understood Conceptual Los Angeles (Bredehoeft 2005) Flow vectors 90 off in model Conceptual Summitville (Bredehoeft 2005) Seeps on mountain unaccounted for Conceptual Santa Barbara (Bredehoeft 2005) Fault zone flow unaccounted for Conceptual WIPP (Bredehoeft 2005) Assumed salt had no mobile interstitial brine Conceptual Fractured Rock Waste Disposal (Bredehoeft 2005) Preferential flow in unsaturated zone unaccounted for Conceptual 9 11/24/2020

How to Proceed? Desirable characteristics of a methodology for uncertainty assessment l Comprehensive: as many types of uncertainty as possible should be included l Quantitative: it should be possible to compare results with regulatory criteria or design requirements l Systematic: able to be applied to a wide range of sites and objectives and to enable the common application of computer codes and methods 10 11/24/2020

Deterministic Approach Assumptions l l l Model parameters are correct Model is correct Scenario is known 11 11/24/2020

Parameter Sensitivity Approach Assumptions l l l Model parameters are unknown Model is correct Scenario is known 12 11/24/2020

Parameter Sensitivity Approach Results l Probability of peak dose represents the degree of plausibility of the model result l ? indicates that the actual values of the probabilities are unknown; statements about the relative values may be possible l Bounding (conservative) analysis: the desired predicted value represents the worst plausible behavior of the system Limitations l Can’t quantitatively estimate risk since probabilities are unknown [risk = p(D > 25 mrem/yr)] l Significance of bounding case must be assessed to avoid overconservatism l Significant sources of uncertainty not included 13 11/24/2020

Parameter Uncertainty Approach Assumptions l l l Model parameters are uncertain Model is correct Scenario is known 14 11/24/2020

Parameter Uncertainty Approach Method l Assign joint probability distribution to model parameters and propagate through the model (e. g. , using Monte Carlo simulation) Results l Peak dose probability density represents the degree of plausibility of the model result l Quantitative estimates of probabilities can be computed l Quantitative estimates of risk can be computed Limitations l Joint probability distribution of parameters must be determined l May be computationally expensive l Significant sources of uncertainty not included 15 11/24/2020

Conceptual Model Sensitivity Approach 16 11/24/2020

Conceptual Model Sensitivity Approach Method l Postulate alternative conceptual models for a site that are each consistent with site characterization data and observed system behavior, Results l Each model is used to simulate the desired predicted quantity l Parameters of each model (which may be different) are represented using a joint probability distribution Limitations l Without a quantitative measure of the degree of plausibility of model alternatives, it is impossible to determine the risk of a decision based on the model predictions l A conservative approach to model uncertainty relies on an implied belief that the most conservative model has a non-negligible degree of plausibility l Requires formulation & simulation of multiple models 17 11/24/2020

Quantitative Model Uncertainty Assign a discrete probability distribution to the conceptual model alternatives l Analogous to the interpretation of parameter probability, the discrete model probability distribution represents the degree of plausibility of the model alternatives What quantity to compare with regulatory/design criteria? 18 11/24/2020

Probability-Based Model Selection Use the model with the highest probability for predictions l Potentially biased result if significant probability with alternative models l If variance due to model uncertainty is desired, must compute predicted value using each model 19 11/24/2020

Conservative Model Selection Use the model with the most significant consequence l How little probability must lie with the highest consequence model before it is judged implausible? l Consequence must be computed with each model to determine the conservative model 20 11/24/2020

Probability-Weighted Model Averaging 21 11/24/2020

Probability-Weighted Model Averaging Method l Model predictions are combined using a weighted average with the weight for each model’s prediction consisting of that model’s probability Results l Model-averaged probability density function represents the degree of plausibility of the predicted value that takes into consideration the joint effect of parameter and model uncertainties l Reduces bias l Less likely to underestimate predictive uncertainty l Consistent treatment of parameter and model uncertainties l Quantitative estimates of risk can be computed from the modelaveraged result 22 11/24/2020

Probability-Weighted Model Averaging Limitations l Model probability is a relative measure with respect to the other model alternatives considered l Requires specifying model probability distribution l Requires formulating & simulating multiple models l Doesn’t consider scenario uncertainty 23 11/24/2020

Model-Averaging Informative Results Mean Dose Prob (Dose > 25) 90%ile Model 1 (prob = 0. 5) 10. 0 8. 2 23. 0 Model 2 (prob = 0. 25) 20. 0 23. 9 32. 7 Model 3 (prob = 0. 25) 45. 0 97. 7 57. 8 Model Average 21. 2 34. 5 48. 5 Results suggest collection of additional data to better discriminate between models (i. e. , to modify model probabilities until one model dominates) Exceedance probability and 90 th percentile suggest that a conservative regulatory action may be preferred l based on a fully-informed consideration of model and parameter uncertainty (i. e. , risk), rather than on adoption of the most conservative model 24 11/24/2020

Scenario Uncertainty: Unknown Future State or Condition of the System Scenario uncertainty can’t be reduced through the application of data (unlike parameter & model uncertainty) 25 11/24/2020

Scenario Uncertainty Sensitivity Approach 26 11/24/2020

Scenario Averaging Approach 27 11/24/2020

Probability-Weighted Scenario Averaging Method l Model-averaged predictions for each scenario are combined using a weighted average with the weight for each scenario’s prediction consisting of that scenario’s probability Results l Scenario- and model-averaged probability density function represents the degree of plausibility of the predicted value that takes into consideration the joint effect of parameter, model, and scenario uncertainties l Quantitative estimates of risk can be computed from the scenario- and model-averaged result Limitations l Requires specifying scenario probabilities l Requires simulations of each model under each scenario 28 11/24/2020

Scenario-Averaging Informative Results Mean Dose Prob (Dose > 25) 90%ile Scenario 1 (prob = 0. 7) (model-average) 21. 2 34. 5 48. 5 Scenario 2 (prob = 0. 3) (model-average) 27. 3 44. 8 58. 5 Scenario Average 23. 0 37. 6 52. 1 Mean dose results straddle regulatory threshold suggesting that a conservative regulatory action may be preferred l based on a fully-informed consideration of model, parameter, and scenario uncertainty (i. e. , risk), rather than on adoption of the most conservative modeling choices 29 11/24/2020

NRC Staff Application of Probability. Weighted Model Averaging MODEL 2 MODEL 4 MODEL 3 MODEL 5 30 11/24/2020

Alternative Model Development Models developed using Groundwater Modeling System (GMS). l Model 2: average values for hydraulic conductivity, recharge, and evapotranspiration l Model 3: average values for hydraulic conductivity and evapotranspiration, zonal values for recharge l Model 4: average value for hydraulic conductivity, zonal values for recharge and evapotranspiration l Model 5: same as model 4 with a general head boundary, recharge, and evapotranspiration 31 11/24/2020

Model & Scenario Averaging Application Simulation Results under Two Scenarios (Well 399 -1 -1) 32 11/24/2020

Scenario Average (Baseline 70%) 33 11/24/2020

Project Objectives Improve access to the uncertainty assessment methodology by integrating methods with FRAMES l Provide guidance on the use of model abstraction techniques to generate plausible and realistic alternative conceptual models for a site l Parameter estimation l Quantitative model comparison l Simulation using multiple models and scenarios Demonstrate using a realistic application relevant to NRC/NRO analyses 34 11/24/2020

Project Schedule Summer 2008 l Implementation of methods completed l NRC workshop Summer 2009 l Completion of application l NRC workshop 35 11/24/2020