Overall procedure of validation Calibration Adjusting of physical

Overall procedure of validation Calibration Adjusting of physical modeling parameters in the model to improve agreement with experimental data Model accuracy assessment by comparison of model outputs with experimental measurements. Validation If calibrated by experiment, prediction at untried conditions and validate again. Prediction Validated Model Blind prediction New Experiment Figure 12. 4 Validation, calibration, and prediction (Oberkampf and Barone, 2004 ). -1 - Optional

Approaches for calibration • Traditional (deterministic) calibration – Parameters are estimated as a single value such that minimize the squared error between the computer model and experimental data. – As a result, the model is given by a single function. • Statistical calibration – Also called Calibration under Uncertainty (CUU). – Parameters are estimated using statistical inference technique to incorporate uncertainty due to the observation error. – As a result, the model is given by the confidence bounds. -2 -

Approaches for statistical calibration – Based on section 13. 5, Oberkampf textbook. • Frequentist approach – Parameter is constant, but unknown because of limited data. – The most popular method is to follow the two steps. 1. point estimate parameters by maximum likelihood estimation (MLE). 2. draw samples of parameters by bootstrap technique. – Advantage is they are simpler and easier to use than Bayesian methods. But less applied to the calibration problem. • Bayesian approach – Parameter is treated as random variable, characterized by the probability distribution conditional on the data. – Also called Bayesian updating. Before updating, the distribution is prior, after updating , it is posterior. -3 -

Calibration in order of complexity • Deterministic calibration – Carry out parameter estimation using optimization technique to obtain a single function as a calibrated model. – The method is not useful because uncertainty is not included. It is like to use the mean value of the uncertainty in the design decision. • Statistical calibration without discrepancy – Carry out parameter estimation using statistical technique to obtain confidence bounds of the calibrated model. – Bayesian approach is common, with the MCMC as the technique for parameter estimation in the probabilistic way. – Due to lack of knowledge, model often differs inherently from the reality. No matter how many data used for calibration, they may fail to agree. – Without accounting for this, assuming the model is correct, we end up with large error, mistaken that they are from experiments, not the model. -4 -

Calibration in order of complexity • Statistical calibration with discrepancy – How to model the discrepancy ? Gaussian process regression (GPR) is employed to express the discrepancy in approximate manner. – Estimation includes not only the calibration parameters but also the associated GPR parameters. – The discrepancy term has two purposes. 1. close the gap between the model and reality, making further improvement of the calibration. 2. validate the model accuracy. If small discrepancy, the model is good. -5 -

Calibration in order of complexity • Statistical calibration with surrogate model – During the MCMC, thousands of model evaluations are needed. If the model is expensive, surrogate model should be introduced. – GPR is employed for this purpose, where design of computer experiments (DACE) is critical in the process. – Estimation includes three parts: calibration parameters, GPR parameters for surrogate model, GPR parameters for discrepancy. Efficiency decreases quickly as the number of parameters increases. – MLE plug-in approach: Surrogate GPR model is deterministic. Parameters are point-estimated. Only the others are estimated probabilistically. – Full Bayesian approach: Includes all the parameters in the estimation. This is the ultimate complexity in the calibration. This is the topic the KOH has addressed. -6 -

Motivating example • Problem addressed in – Loeppky, Jason L. , Derek Bingham, and William J. Welch. "Computer model calibration or tuning in practice. " Technometrics, submitted for publication (2006). – Bayarri, Maria J. , et al. "A framework for validation of computer models. " Technometrics 49. 2 (2007). – Originally Fogler, H. S. , (1999), Elements of Chemical Reaction Engineering, Prentice Hall. • Chemical kinetics model – Describes a chemical reaction process with initial chemical concentration 5 and reaction rate 1. 7. Amount of chemical remaining at time x is investigated. -7 -

Motivating example • Chemical kinetics model – Make virtual experimental (or observation) data with the noise Three replicates are made at 11 points of equal interval in [0, 3]. – Repeated with data for right figure given in note page • Objective – Find computer model to simulate observations as closely as possible. -8 -

Simplest computer model – Somewhat wrong guess due to lack of knowledge. – Calibrate q such that minimizes the SSE between model and data. • Optimum solution from matlab fminsearch • Using nlinfit and nlparci with second data but with n-1, (see notes page) -9 -

Discussion • If error was due only to noise, we would have expected RMSE=0. 3 • Given the true function, we can see that the model is not good. What are the clues without looking at the true function? • What is the hierarchy of calibration methods without discrepancy?

Calibration with improved models • Computer model and optimum solution – Two parameters optimization problem – Solution improved. but still substantial gap. • Computer model and optimum solution – Three parameters optimization problem – Excellent match (true q 1 = 1. 5, q 2 =3. 5, q 3 =1. 7) – Model change made on ad-hoc basis. Besides, the close match is undoubtedly just luck. Is this possible in the real practice ? - 11 -

Calibration under uncertainty • Bayesian approach – Assume that the model is accurate representation of reality. – Field data is given by – Posterior distribution of the unknown parameters q, s 2 • Posterior distribution - 12 -

Calibration under uncertainty • Posterior samples after MCMC (N=5 e 3) Means of q and s 0. 6295 0. 5921 0. 6235 0. 5870 0. 6249 0. 5866 0. 6269 0. 5952 • Posterior prediction - 13 -

Calibration under uncertainty • Frequentist approach – Likelihood of YF • Maximum likelihood estimation - 14 -

Calibration under uncertainty • Bootstrap sampling – Make virtual experimental data by applying the estimated parameter into the model and the noise. – Go through MLE estimation using the data, and repeat this N times to get the samples of parameters. Meeker, William Q. , and Luis A. Escobar. Statistical methods for reliability data. Vol. 314. Wiley. com, 1998. - 15 -

Calibration under uncertainty • Discussion – Confidence bounds of model is now obtained. i. e. , at x=1. 5, the bound is (0. 75, 3. 19). – Due to the incorrect model, end up with large bound. However, this is the best available solution under this condition. – Within this large bounds, not only measurement error but also the model error are included. We need to account for this by introducing discrepancy function. - 16 -

Practice problem • Repeat the calculations shown in this lecture generating your own data with the same equations and same noise amplitudes. - 17 -