1 DSST Ensemble Verification Call 4 Slides Kevin

1 DSST Ensemble Verification Call 4 Slides Kevin Warner Hank Herr Won Kwock/Edwin Welles Julie Demargne June 8, 2005

2 CBRFC MRF Project DSST Verification Call June 8, 2005 Kevin Werner NWS/WRH/SSD

3 Outline • MRF / ESP hindcasting experiment • Pseudo reforecasting code • Verification of hindcast experiment • Future Work

4 CBRFC AHPS PROJECT A cooperative effort between:

5 Schematic of Using Ensembles from MRF(day 1 -14) As Input to ESP Blending/Attachment Days 1 -14 Ensembles From MRF Days 15 -365 Ensembles From Climatology

ESP Reforecast Probabilistic forecast (or model) verification requires a large dataset. This was accomplished through reforecasting. Reforecasts done for every basin for every day between 1979 – 1999. Reforecasts made with both reforecasted MRF and historical MAT/MAPs. 6

ESP Reforecast For EACH reforecast day (i. e. 1/1/79, 1/2/79, … , 7/1/99)… 1) NWSRFS carryover states from historical simulated saved. 2) MAT and MAP ensembles created from MRF reforecast. 3) Two sets of flow ensembles (i. e. *. CS time series) created from ESP: 1) Control case: uses historically observed forcings 2) MRF case: uses 14 day forcings from MRF model 7

8 ESP Reforecast Schematic: State variables derived from historical simulation and used to initialize reforecast conditional simulations.

ESP Reforecast Example: ESP reforecasted flow ensembles (colored lines) produced from MAT/MAP ensembles 9

ESP Reforecast Scripts • Reforecasting done with two TCL scripts that run ESP iteratively: – gen_fg_co. tcl – Runs ESP historical simulation based on existing carryover in fs 5 files. Carryover iteratively saved for each day. – gen_fg_ts. tcl – Runs ESP simulations based on carryover created from first script. Resulting conditional time series (*. CS) are archived for further analysis. 10

11 gen_fg_co. tcl #Created 9/03 Kevin Werner #Generates ESP carryover for each day in specified range. #Sequentially creates input file for fcst and calls fcst. # # #Define Parameters # set env(calb_data_dir) /local/lx/rfc/nwsrfs/calb/data/area_ts set env(calb_area_ts_dir) /local/lx/rfc/nwsrfs/calb/data/area_ts set startsec [clock scan 12/29/2005] set endsec [clock scan 1/03/2006] set fcstindir "/local/lx/rfc/nwsrfs/ofs/input/oper/fcst/" set fcstinfile "co_gen" set ofsoutdir "/local/lx/rfc/nwsrfs/output/kvw/" #set fcstindir "/fs 1/home/oper/reforecast/" # #Loop over each day, basin # for {set csec $startsec} {$csec <= $endsec} {incr csec 86400} { # #Create fcst input file # set of [open "[subst $fcstindir][subst $fcstinfile]" w] puts $of "@SETOPT" puts $of "STARTESP 0606/2005/" puts $of "WINDOWS(1) 1030/2005/ 0601/2006/" puts $of "NUMCOSAV(1) [clock format $csec -format "%m%d/%Y/"]" puts $of "HISTSIM(1)" puts $of "REGULATE(0)" puts $of "TSUNITS(1) 91 92 93 94 95" puts $of "HISTWYRS 1980 2002" puts $of "FGROUP SPSV" puts $of "@COMP ESP" puts $of "@STOP" close $of if {[catch {exec /local/lx/rfc/nwsrfs/ofs/scripts/ofs -p fcst -i co_gen -o co_gen} msg]} {puts $msg} } Define range of days to create carryover Loop over each day required Save Carryover Run ESP

12 gen_fg_ts. tcl # #Created 10/03 Kevin Werner #Generates ESP time series from carryover generated from gen_fg_co. tcl. # # #Define Parameters # set env(calb_data_dir) /local/lx/rfc/nwsrfs/calb/data/area_ts set env(calb_area_ts_dir) /local/lx/rfc/nwsrfs/calb/data/area_ts set startyear 1981 set endyear 2002 set startmonth 1 set startday 1 set endmonth 1 set endday 2 set fcstindir "/local/lx/rfc/nwsrfs/ofs/input/oper/fcst/" set fcstinfile "ts_gen_rls" set basinfile "spsv. basins" set basins [read_file -nonewline $basinfile] set basins [split $basins n] set fgroup "SPSV" Define range of days to perform reforecast over set esptsdir "/local/lx/rfc/nwsrfs/ens/files/kvw/espts/spsv/" set ofsoutdir "/local/lx/rfc/nwsrfs/output/kvw/" #set fcstindir "/fs 1/home/kvw/reforecast/" # #Loop over each year # for {set year $startyear} {$year <= $endyear} {incr year} { set startsec [clock scan [subst $startmonth]/[subst $startday]/$year] set endsec [clock scan [subst $endmonth]/[subst $endday]/$year] #Loop over each day for {set csec $startsec} {$csec <= $endsec} {incr csec 86400} { # #Create fcst input file # set of [open "[subst $fcstindir][subst $fcstinfile]" w] puts $of "@SETOPT" puts $of "STARTESP [clock format $csec -format "%m%d/%Y/"]24 MST" puts $of "WINDOWS(1) 0401/[clock format $endsec -format "%Y/"] 0801/[clock format $endsec -format "%Y/"]" puts $of "ESPOTDIR=spsv" puts $of "ESPINDIR=spsv" puts $of "HISTWYRS $startyear $endyear" puts $of "FGROUP $fgroup" puts $of "@COMP ESP" puts $of "@STOP" close $of if {[catch {exec /local/lx/rfc/nwsrfs/ofs/scripts/ofs -p fcst -i $fcstinfile -o $fcstinfile} msg]} {puts $msg} } Define directory structure Time loops over each day required Write fcst input file Iterate over required days Run ESP

$13 gen_fg_ts. tcl continued # #Loop over each basin # foreach basin $basins {$

13 gen_fg_ts. tcl continued # #Loop over each basin # foreach basin $basins { set tsfname "[subst $basin]. SIM 24. SQME. 24. CS" file rename -force [subst $esptsdir]$tsfname [subst $esptsdir][subst $tsfname] [clock format $csec -format ". %m. %d. %Y"] set tsfname "[subst $basin]. ADJ 1. QINE. 01. CS" file rename -force [subst $esptsdir]$tsfname [subst $esptsdir][subst $tsfname][clock format $csec -format. %m. %d. %Y"] #end basin loop } } #end day loop Loop over each segment (“basin”) Rename and archive ESP conditional simulations (i. e. *. CS) files for analysis #end year loop } End time and basin loops

Reforecasting analysis • Analysis and verification done with combination of TCL and R scripts – TCL scripts for extracting required time series from archived *. CS files (i. e. flows, seasonal volumes, etc. ) – R scripts for verifying, analyzing, and plotting • Examples available upon request. 14

ESP Reforecast Example Following example from Granby, CO (GBYC 2) reforecast for May 1, 1985. This is a snowmelt situation. 15

Temperature Input into ESP 16 MRF derived MATs and MAPs (not shown) are attached to historical years (“ensembles”) and ‘fed’ to ESP. Note MRF is warmer in first week

Input into ESP 17 MRF derived MATs related to the entire year of historical ensembles.

ESP flow time series Hourly instantaneous flow ensembles are created by ESP and saved. MRF shows higher flows than historical when it is warmer (during the first week). These may be converted into probabilistic forecasts… 18

19 ESP peak flow Peak flow forecasts shown as Probability Density Functions (PDFs). MRF shows higher probabilities in higher flows for two weeks.

ESP Forecast Verification Forecast lead time Mean hydrograph and RPSS values… Forecast valid date Good forecast skill improvements during rising limb of hydrograph. 20

ESP Forecast Verification 21

Future Reforecasting Work • Currently exploring use of the Climate Forecasting System (CFS) run by CPC/EMC using similar framework – CFS output typically in monthly averages – Reforecast archive available from 1981 – Shows some promise during ENSO events particularly forecasts made during the autumn months – May be useful for early season water supply forecasts 22

23 Ensemble Hindcast Time Series Generator (ETSGEN)

Overview • Etsgen gui is a graphical user interface (GUI) which enables a user to construct HCL input files for the OFS FCST ESP function and run those HCL input files to generate ESP hindcasts to be used for verification studies. 24

25

Hindcast Procedure • ESP is run in historical simulation mode to acquire past carryover – One set of carryover is saved for each year of historical simulation run • ESP is executed using each of the past saved carryover as the state of system at forecast time 26

ETSGEN • Input – Hindcast parameters • • Carryover group, forecast group(s), segment(s) Carryover save date Hindcast start and end dates Hindcast start and end years • Output – OFS input decks that can be used to generate carryover for hindcasts and the ESP time series files that comprise the hindcasts • Script espvs_generate. sh executes OFS jobs – ETSGEN can be used to run espvs_generate. sh 27

28 JProb. VS Wen Kwock / Edwin Welles May 24, 2005

Purpose of This Presentation • Introduce functions of JProb. VS • General description of computational process 29

30 JProb. VS Diagram JProb. VS Batch mode espadp. Iteration_DMI Discrimination RPS Reliability RPSS ROC

31 JProb. VS Process espadp. Iteration_DMI Data. Card/ ESP TS RPS Stat Result Process espadp Tables RPSS Discrimination Reliability ROC Stat Result Data/File

JProb. VS Input/Output • Input files: Ø Input deck: prob. VSBatch Ø ESP time series files Ø Datacard files Ø Observe file name must in *. OBS format Ø Forecast file name must in *. VS. CCYYMMMDD format v BAYI 4. SQME. 24. VS. 1981 Feb 01 • Output files: Ø. Data files v BAYI 4. SQME. 24. Feb 01. QCVFUZZ. Dis_stat. data Ø. IWK files v BAYI 4. SQME. 24. Feb 01. QCVFUZZ. Dis_stat. iwk Ø. Results files v BAYI 4. SQME. 24. Feb 01. QCVFUZZ. Dis_stat. results Ø. XML files v BAYI 4. SQME. 24. Feb 01. QCVFUZZ. Dis_stat. XML Ø. Table files v BAYI 4. SQME. 24. Feb 01. QCVFUZZ. Dis_stat. table v Readable by matlab 32

RPS 1) RPS Results -----Year | RPS ----+----1950 | 1. 0640051509569648 1951 | 1. 2982671726400459 1952 | 0. 13029338333090265 1953 | 1. 5452837682028684 … 2001 | 1. 78505677402705 ----+----Average | 0. 5358592525789612 33

RPS %Formatted for Matlab and Spreadsheets; <tab> delimited columns. % % ============= % Ranked Probability Score % ============= % Columns are Year, RPS, with the first row the average (and therefore it has a year of Nan) % NEED to include in here the probability thresholds used in the calculations Nan 0. 5359 1950 1. 0640 1951 1. 2983 1952 0. 1303 1953 1. 5453 … 2001 1. 7851 34

RPS 35

RPSS 1) RPSS Results -----Year | RPSc RPSf | RPSS ----+---------+----1950 | 1. 625 1. 064 | 34. 523 1951 | 1. 625 1. 298 | 20. 107 1952 | 0. 125 0. 13 | -4. 235 1953 | 1. 625 1. 545 | 4. 906 … 2001 | 1. 625 1. 785 | -9. 85 -------+----Average | 0. 625 0. 0 | 0. 0 36

RPSS % Formatted for Matlab and Spreadsheets; <tab> delimited columns. % % ============= % Ranked Probability Skill Score. % ============= % Columns are Year, climate RPS, Forecast RPS, and RPSS % with the first row the average (and therefore it has a year of Nan) % the Skill score is reported as a % so divide by 100 in plts % NEED to include in here the probability thresholds used in the calculations. Na. N 0. 62 -999. 0 1950 1. 62 1. 06 34. 52 1951 1. 62 1. 30 20. 11 1952 0. 13 -4. 23 1953 1. 62 1. 55 4. 91 … 2001 1. 62 1. 79 -9. 85 37

RPSS 38

Discrimination 1) Discrimination Results -----------3. 2) Discrimination the forecast was: | For all forecasts where the obs occurred from C 1 to <C 2, the probability of | Observation | 0% to <25%| 25% to <50%| 50% to <75%| 75% to <100% From C 1 to < C 2|------------------|---------|-------% # Obs| F 0 F 1 F 2 ----------|------------------|------------------694. 0 |0. 0 0. 692 1. 0 |0. 154 0. 231 0. 0 |0. 308 0. 077 0. 0 |0. 538 0. 0 694. 0 1500. 0 |0. 577 0. 154 0. 538 |0. 077 0. 577 0. 154 |0. 346 0. 231 0. 192 |0. 038 0. 115 1500. 0 |0. 923 0. 077 0. 154 |0. 077 0. 385 0. 462 |0. 0 0. 462 0. 308 |0. 077 Where the following applies | From to < -----+----------F 0 | 694. 0 F 1 | 694. 0 1500. 0 F 2 | 1500. 0 39

Discrimination % Formatted for Matlab and Spreadsheets; <tab> delimited columns. % % ============= % Discrimination Diagram Points. % ============= % First row is lower category bound. Second row is upper category bound. % Starts third row, columns are lower probability bound, upper probability % bound, probabilities for category one, probabilities for category two, . . . Na. N -inf 694. 00 1500. 00 Nan Na. N 694. 00 1500. 00 inf 0 25 0 69 100 58 15 54 92 8 15 25 50 15 23 0 8 58 15 8 38 46 50 75 31 8 0 35 23 19 0 46 31 75 100 54 0 0 0 4 12 0 8 8 40

Discrimination 41

Reliability 1) Reliability Results ---------Forecast Categories | Probability ranges from #1% to less than #2% ( or to 100% ) From to < | 0 25 25 50 50 75 75 100 ------------+----------------------698. 68 | 0. 00 0. 40 0. 36 1. 00 698. 68 1, 540. 48 | 0. 25 0. 64 0. 46 0. 50 1, 540. 48 | 0. 07 0. 60 0. 45 0. 00 42

Reliability % Formatted for Matlab and Spreadsheets; <tab> delimited columns. % % ============= % Reliability Diagram Points. % ============= % First line provides forecast category max values corresponding to data % in that column. % For every row after, the first column is lower bound of the forecast % probability bin, the second column is the upper bound of the bin, % and then the reliability diagram point is provided for each category. Na. N 698. 68 1540. 48 INF 0. 00 0. 25 0. 07 0. 25 0. 50 0. 40 0. 64 0. 60 0. 50 0. 75 0. 36 0. 45 0. 75 1. 00 0. 50 0. 00 43

Reliability 44

ROC 1) ROC Results ---------| Forecast Categories Probability | 1, 192. 32 | 1, 268. 65 | 684. 52 | Thresholds | POFD POD | ------------+--------+-------+0. 25 | 0. 09 0. 26 | 0. 09 0. 24 | 0. 00 0. 68 | 0. 75 | 0. 45 0. 79 | 0. 40 0. 88 | 0. 58 0. 98 | 45

ROC % Formatted for Matlab and Spreadsheets; <tab> delimited columns. % % ========= % ROC Diagram Points. % ========= % First line provides forecast thresholds corresponding to data % in that column. % For every row after, first column is probability threshold used to % calculate the POFD and POD points. Then, POFD and POD points are % provided for each of the forecast thresholds. Na. N 1192. 32 1268. 65 684. 52 0. 25 0. 09 0. 26 0. 09 0. 24 0. 00 0. 68 0. 75 0. 45 0. 79 0. 40 0. 88 0. 58 0. 98 46

ROC 47

48 For JProb. VS documentation, go to http: //hsp. nws. noaa. gov/oh/hrl/nwsrfs/esp/jp robvs. pdf

49 Ensemble Hindcaster for Precipitation, Temperature, & Streamflow Probabilistic Verification Hydrologic Ensemble Prediction Team Hydrology Laboratory Office of Hydrologic Development NOAA/National Weather Service June 8, 2005 1

Ensemble Hindcaster 50 • Goal: capability for systematic and rigorous hindcasting to validate ensemble science and assess benefit-cost effectiveness of newly proposed solutions • Statement of Need: – Why: validate ensemble science and evaluate probabilistic forecast performance – Originator: AB-, CN-, and MARFC currently generating shortterm hydrologic ensembles – Existing capabilities: hindcasting capability at CBRFC; limited capability for short-term precipitation and temperature ensembles – Benefits: improve predictions and validate improvements relative to forecast reliability and accuracy; serve RFC’s operational need for ensemble system calibration and forecast validation 2

Ensemble Hindcaster: Data & Processes Atmospheric Ensemble Preprocessor For one segment Carryover Data Generation MAP & MAT re-sampled climatology ensembles Historical MAP & MAT time series 1 Current carryover data Retrospective carryover data 2 51 Meteorological Forecast Generation MAP & MAT short-term RFC & resampled climatology ensembles 3 Hydrologic Ensemble Processor From 1, 2, or 3 Retrospective hydrological ensembles MAPE, QME, PTPE… time series Hydrologic Forecast Generation 3

Preprocessor: Re-sampled Climatology 52 • MAP & MAT ensemble forecasts generated by re-sampling climatology probability distribution for all lead times At each time step of the forecast period and for each location, generate ensemble members from the climatology distribution with Schaake Shuffle technique T 1 1 Historical Distribution Climatology Distribution 0 1960 1949 1983 Precipitation Amount Climatology Distribution 0 1960 Climatology Distribution Probability Historical Distribution Probability T 3 T 2 … 0 1983 1949 Precipitation Amount 1960 1983 1949 Precipitation Amount Space-time properties of resulting time series are similar to the historical events properties 4

Preprocessor: Short-Term RFC Ensembles 53 • MAP & MAT ensemble forecasts generated by using Conditional Probability Distribution based on HPC/RFC forecast At each time step of the forecast period and for each location, compute the conditional distribution of future values given the single-value forecast Conditional distribution Normal Space Joint distribution Normal Space 1 Probability FMAP FMAT NQT forecast Observed ZY ZX z. Y 0 Forecast z. X 0 forecast z. X 0 Probabilistic forecast given Inverse the NQT FMAP/FMAT forecast P ( ZY | ZX = z. X ) z. X 0 Observed ZX Future tests done with GFS forecasts to use the longer GFS reforecast archive for calibration 5

Preprocessor: Short-Term RFC Ensembles 54 • MAP & MAT ensemble forecasts generated by sampling conditional probability distribution based on HPC/RFC forecasts for short-term lead times At each time step of the forecast period and for each location, generate ensemble members from the conditional distribution with Schaake Shuffle technique T 1 T 3 T 2 1 1 1 Conditional Distribution 0 1960 1949 1983 Precipitation Amount Conditional Distribution Climatology Distribution 0 1960 Probability Climatology Distribution Probability Conditional Distribution Climatology Distribution … 0 1983 1949 Precipitation Amount 1960 1983 1949 Precipitation Amount Space-time properties of resulting time series are similar to the historical events properties 6

Hindcaster: Preprocessor (1) 55 • Generate MAP & MAT hindcasts without RFC parameters: re-sampled climatology ensembles Precipitation (IN), Huntingdon, MARFC EST Climatology MAP time series in local standard time Re-sampled climatology MAP ensembles in internal time INT 7

Hindcaster: Preprocessor (2) 56 • Generate MAP & MAT hindcasts with RFC parameters: short-term RFC ensembles blended with re-sampled climatology ensembles Precipitation (IN), Huntingdon, MARFC Re-sampled climatology EST Climatology MAP time series in local standard time Short-term RFC & resampled climatology MAP ensembles in internal time INT 8

Hindcaster: Preprocessor (3) 57 • Generate 6 -hr MAT hindcasts with RFC parameters from daily maximum and minimum temperature ensembles User-defined temporal disaggregation process for each time step: T 6 = Tmin + TP x (Tmax – Tmin) Temperature (DEGF), Huntingdon, MARFC Re-sampled climatology EST Climatology MAT time series in local standard time Short-term RFC & resampled climatology MAT ensembles in internal time INT 9

Hindcaster: Preprocessor (4) 58 • Archiving data to run Hindcaster with various input hindcasts: 1) Climatology time series: <ts_path><ts_name>. MAP 06 and <ts_path><ts_name>. MAT in calb_area_ts_dir/climato/ 2) Re-sampled climatology hindcasts: <ts_path><ts_name>. MAP 06 and <ts_path><ts_name>. MAT in calb_area_ts_dir/pre_climato/ 3) Short-term RFC and re-sampled climatology MAP & MAT hindcasts based on start date yyyy/mm/dd: <ts_path><ts_name>. MAP 06 and <ts_path><ts_name>. MAT in calb_area_ts_dir/pre_short/yyyy/mm/dd/ Hindcast starting on 02/04/1998 for HUNP 1 JUN segment: calb_area_ts_dir/pre_short/1998/02/04/huntingdon. MAP 06 calb_area_ts_dir/pre_short/1998/02/04/huntingdon. MAT 10

Hindcaster: Carryover Data (1) 59 • Generate retrospective carryover information from existing carryover data (fs 5 files) using FCST program with ESP technique FCST input control file for 02/03 Current carryover date Carryover date to be saved on 24 hr local time zone Historical simulation Historical water years to use FCST output file: saving carryover data, 02/03/1950 -1998 […] Output: Carryover data file in carryover directory: HUNP 1 JUN. 02. 03. 24. EST […] 11

Hindcaster: Carryover Data (2) 60 • Generate retrospective carryover information from existing carryover data (fs 5 files) for a set of dates using a ksh script FCST input control files created for every week from 09/30 to 09/29 … Output: Carryover data files for every week from 09/30 to 09/29 from 1950 to 1998 12

Hindcaster: Streamflow Ensembles (1) 61 • Generate streamflow hindcasts from various MAP & MAT hindcasts using FCST program with ESP technique • Time issue: if using MAP & MAT hindcasts in internal time, run FCST with the PQPFTIME technique FCST input control file for 02/04/1988 Using hindcasts in INT Using hindcasts in LST Carryover date Forecast period Conditional sim. Internal time Historical water years to use Output file: streamflow hindcast file HUNP 1 JUN. QINE. 06. CS saved as HUNP 1 JUN. QINE. 06. VS. 19880204 13

Hindcaster: Streamflow Ensembles (2) 62 • Using MAP & MAT hindcasts from a specific calb_area_ts_dir directory, generate streamflow hindcasts in the corresponding espts_dir directory From climatology EST Streamflow ensembles (cfs), Huntingdon, MARFC From re-sampled climatology INT From short-term RFC and re-sampled climatology INT 14

Hindcaster: Streamflow Ensembles (3) 63 • Generate streamflow hindcasts from MAP & MAT hindcasts for a set of dates using a ksh script FCST input control files created for every week from 10/01/1997 to 09/23/1998 … Output: streamflow hindcast files for every week from 09/30/1997 to 09/29/1998 15

Hindcaster: Streamflow ensembles (4) 64 • Archiving streamflow hindcasts to run Jprob. VS based on start date yyyy/mm/dd: <segment_ID>. <ts_ID>. QINE. VS. yyyymmmdd • Correspondence between input and output directories: Methodology Input directory Output directory 1) Climatology calb_area_ts_dir/climato/ 2) Re-sampled climatology calb_area_ts_dir/pre_climato/ espts_dir/pre_climato/ 3) Short-term RFC calb_area_ts_dir/pre_short/yyyy/mm/dd/ espts_dir/pre_short/ 16

Hindcaster: Ensemble Verification (1) 65 • Running Jprob. VS for a specific espts_dir directory Methodology Output directory 1) Climatology espts_dir/climato/ 2) Re-sampled climatology espts_dir/pre_climato/ ROC from climatology 3) Short-term RFC espts_dir/pre_short/ Comparison of performance between climatology-based streamflow hindcasts and re-sampled climatology-based streamflow hindcasts ROC from resampled climatology 17

Hindcaster: Ensemble Verification (2) 66 • Verification of short-term RFC hydro-meteorological hindcasts Verification statistics on 10 MARFC basins for precipitation Perfect scores: Score = 0 Scores computed on 3 categories Verification statistics on 1 CNRFC basin for temperature 18

Hindcaster: Issues 67 1. Required data for ensemble generation and verification: • Precipitation for short-term RFC ensemble: – MAP observations, up to present, datacard format – HPC/RFC forecasts, up to present • Temperature for short-term RFC ensemble: – MAT observations, up to present, datacard format – HPC/RFC forecasts for TMax and TMin, up to present • Streamflow: – Observations, up to present, datacard format 2. Integrating other ensemble generation methodologies into Hindcaster: GFS-based preprocessor, MRF-based preprocessor, etc. 19

Future Ensemble Hindcaster Atmospheric Ensemble Preprocessor For one segment Carryover Data Generation Historical MAP & MAT time series Current carryover data Retrospective carryover data MAP & MAT re-sampled climatology ensembles 1 2 Meteorological Forecast Generation MAP & MAT short-term RFC & resampled climatology ensembles 3 Hydrologic Ensemble Processor From 1, 2, 3, 4, or … Retrospective hydrological ensembles 68 4 Other MAP & MAT ensembles … … MAPE, QME, PTPE… time series Hydrologic Forecast Generation 20

Examples of Results 69 Comparison of performance between climatology-based streamflow hindcasts and re-sampled climatology-based streamflow hindcasts using Jprob. VS statistics: - Reliability diagram - ROC - RPSS - Discrimination diagram Streamflow hindcasts for Huntingdon, MARFC, generated every week from 10/01 to 02/25 (22 dates) for 1986, 1987, and 1988 water years 21