Validating the Global Ocean Forecast System Version 3

Validating the Global Ocean Forecast System Version 3. 1 Alan Wallcraft and Joe Metzger Naval Research Laboratory NCEP EMC Seminar College Park, MD May 10, 2017

Contributors • NRL: • Joe Metzger • Bob Helber • Pat Hogan • Pam Posey • Prasad Thoppil • Tammy Townsend • Alan Wallcraft • Vencore: • Ole Martin Smedstad • Debbie Franklin • Florida State: • Luis Zamudio • Jacobs Technology: • Michael Phelps All NRL 7320 publications available at: https: //www 7320. nrlssc. navy. mil The GOFS 3. 1 Validation Test Report (Metzger et al, 2017) will be there soon

GOFS 3. 1 vs. GOFS 3. 0 • GOFS 3. 1 • • 41 layer 1/12° global HYCOM NCODA-3 DVar Los Alamos Community Ice Cod. E (CICE) Improved Synthetic Ocean Profiles (ISOP) • GOFS 3. 0 • • 32 layer 1/12° global HYCOM NCODA-3 DVar Energy loan ice Modular Ocean Data Assimilation System (MODAS)

GOFS 3. 1 vs. GOFS 3. 0 • HYCOM changes: • Newer source code: v 2. 2. 99 DHi vs. v 2. 2. 19 • Improved base bathymetry: 30” GEBCO vs ETOP 05 • 0. 1 m coastline vs. coastline at 10 m isobath • Improved Equation of State: 17 -term vs. 7 -term • Improved vertical structure: additional 9 layers near the surface – mixed layers are typically better resolved • Changes to surface momentum forcing: calculated in-line in GOFS 3. 1 using HYCOM SST and allowing for HYCOM surface currents • Improved ocean turbidity scheme: chlorophyll-based vs. photosynthetically available radiation-based. • Improved SSS relaxation

GOFS 3. 1 vs. GOFS 3. 0 • Sea ice model changes: • Full two-way coupling between HYCOM-CICE via ESMF vs. thermodynamic “energy-loan” ice model • Ocean-ice coupling frequency is every hour • Ocean-ice models share the same grid • Full rheology in CICE vs. ice growth/melt in response to temperature and heat fluxes and no ice advection by wind or ocean currents • GOFS 3. 1 ice validation is compared against operational Arctic Cap Nowcast/Forecast System (ACNFS), not GOFS 3. 0 • Same version of CICE (v 4. 0) within GOFS 3. 1 and ACNFS

GOFS 3. 1 vs. GOFS 3. 0 • NCODA-3 DVAR changes: • NCODA analysis at 12 Z using 24 -hour HYCOM forecast daily mean vs. NCODA analysis at 18 Z using 24 -hour HYCOM forecast instantaneous field • Daily mean will filter out the tidal signal when tides are implemented in GOFS 3. 5 • Data selection for assimilation at receipt time vs. observation time • GOFS 3. 1 uses First Guess at Appropriate Time (FGAT) to account for late arriving data • Cap on the maximum Rossby radius of deformation

Assimilating Subsurface Observations • There are vastly too few daily subsurface observations for data assimilation to constrain the global ocean • About 4, 000 Argo floats surface every 10 days (400 per day) • Fixed equatorial arrays provide continuity in time but are too far apart in longitude to fully characterize equatorial dynamics

Assimilating Satellite Observations • Only satellites provide dense near global coverage of the ocean • Satellite altimeters observe sea surface height anomaly, w. r. t. the mean geoid o SSHA includes information about the total water column o Without altimeters, global ocean nowcasting would be impractical

SSHA Assimilation Approach Use SSHA, SST, etc. Estimate subsurface (MODAS/ISOP) SSHA, SST observations all bu indirect SSH assimilation ts T, S profiles (synthetics) sh a mixed layer depth, background T, S 3 D VAR (NCODA) NCODA Navy Coupled Ocean Data Assimilation HYCOM Hybrid Coordinate Ocean Model MODAS Modular Ocean Data Assimilation System ISOP Improved Synthetic Ocean Profiles T inc , S, rem u, v en for ts ec err ast or Ocean Model (HYCOM)

GOFS 3. 1 vs. GOFS 3. 0 • NCODA-3 DVAR: downward projection of surface information • Synthetic profiles are generated along altimeter tracks • Use observed SSH and SST to create synthetic T & S profiles based on historical relationships between surface and subsurface observations • GOFS 3. 0 uses MODAS • Linear regression of bi-monthly EOF ocean structure • Mike Carnes, Bill Teague, and Jim Mitchell, 1994: Inference of Subsurface Thermohaline Structure from Fields Measurable by Satellite; JTECH, 11, 551 -566 • GOFS 3. 1 uses ISOP • Monthly 3 -layer (mixed, dynamic, deep) 1 D variational technique • T. L. Townsend, C. N. Barron and R. W. Helber, 2015: Ocean Prediction with Improved Synthetic Ocean Profiles (ISOP); NRL Review 105 -112 • https: //www 7320. nrlssc. navy. mil/pubs/2015/townsend-2015. pdf

GOFS 3. 1 Timeline • GOFS 3. 1 received Validation Test Panel (VTP) approval in September 2014 o Report included a comparison against surface drifters using “separation distance” as a function of forecast length. It concluded that GOFS 3. 1 outperformed GOFS 3. 0 in most regions. • GOFS 3. 1 OPTEST didn’t really get started until June 2015 • In August 2015, NRL noticed deficiencies in upper ocean EKE and velocity and notified NAVOCEANO o “Separation distance” was a poor metric, and was not calculated correctly • GOFS 3. 1 was pulled from OPTEST in September 2015 • By March 2016, ISOP issues were largely resolved • New GOFS 3. 1 hindcast was delayed until July 2016, so it could use NAVGEM 1. 4 forcing • GOFS 3. 1 VTP meeting in November 2016 o Better drifter metrics • GOFS 3. 1 VTP approval in January 2017 • GOFS 3. 1 entered OPTEST for ocean (NAVOCEANO) and sea ice (NIC) in February 2017

ISOP 1. 0 vs 1. 1 • Changes to ISOP between the original GOFS 3. 1 and current GOFS 3. 1 • Reducing the input SSH anomaly error estimate for computing the synthetics • Revising and improving the databases used for constraining the SSH anomalies and errors • Increasing the allowable range of input SSH anomalies • Effect is to make ISOP SSHA closer to altimeter SSHA

ISOP SSH is better SSH Anomaly from Synthetic Profiles MODAS • Twin experiments run 1 day from the same 60. 4 background state • HYCOM results are after ISOP → NCODA → incremental insertion • Altimeter SSH anomaly wrt altimeter mean • HYCOM steric SSH anomaly wrt MODAS 4. 2 annual mean climatology full depth T&S profiles ISOP 1. 0 ISOP 1. 1

Improvements to ISOP Steric SSHA – 02 June 2014 GOFS 3. 1 – original ISOP GOFS 3. 0 – MODAS 45 N 40 N 35 N 75 W 65 W 55 W 45 W 35 W Altimeter 75 W 65 W 55 W 45 W 35 W GOFS 3. 1 – new ISOP 45 N 40 N 35 N 75 W 65 W 55 W 21 November 2016 45 W 35 W GOFS 3. 1 VTP Meeting 75 W 65 W 55 W 14

Improvements to ISOP Eddy Kinetic Energy (EKE) at 15 m GOFS plots are averages over Jul-Sep 2014 Drifter data is a 1979 -2012 average m 2/s 2. 2 GOFS 3. 0 – MODAS GOFS 3. 1 – original ISOP . 1 0 Drifting buoys (1979 -2012) from Lumpkin and Johnson (2013) GOFS 3. 1 – new ISOP

Improvements to ISOP Eddy Kinetic Energy (EKE) at 15 m Percent RMS EKE Variability Cumulative % RMS EKE Variability 100 12 10 80 8 60 6 40 4 20 2 0 0. 02 GOFS 3. 1 VTP Meeting 0. 04 0. 06 0. 08 0 0. 05 0. 1 m 2/s 2 Drifting buoys (1979 -2012) GOFS 3. 0 (MODAS, Jul-Sep 2014) GOFS 3. 1 (original ISOP, Jul-Sep 2014) GOFS 3. 1 (new ISOP, Jul-Sep 2014) 0. 15 m 2/s 2 0. 2 21 November 2016 | 16

Improvements to ISOP Comparison against independent drifters Statistics over Jul-Sep 2014 at 15 m depth vs. Forecast Length Mean Error (cm/s) GOFS - Drifting Buoys 4 Mean observed drifter speed = 24. 2 cm/s 21 RMSE (cm/s) GOFS vs. Drifting Buoys . 65 Vector Correlation GOFS vs. Drifting Buoys 20 2 19 0 -2 . 55 18 -4 00 06 12 18 24 GOFS 3. 0 (MODAS, No Vocn) GOFS 3. 1 (original ISOP, Vocn) GOFS 3. 1 (new ISOP, No Vocn) 17 00 06 12 18 Forecast Hour 24 . 45 00 06 12 18 Forecast Hour 24 Only difference is inclusion of Vocn GOFS 3. 1 with new ISOP beats GOFS 3. 0 for all metrics and including V ocn in the surface stress formulation produces the lowest RMSE and highest vector correlation, but GOFS 3. 1 with new ISOP and not including Vocn has the lowest mean error. GOFS 3. 1 VTP Meeting 21 November 2016 | 17

GOFS 3. 1 vs. ACNFS • NCODA-3 DVAR changes: • Ice assimilation changes: • In both systems NCODA produces a daily ice concentration field based on the 24 -hr CICE forecast as the background and derived ice concentration products from 1) the Special Sensor Microwave Imager/Sounder (SSMIS), 2) the Advanced Microwave Scanning Radiometer 2 (AMSR 2) and the 4 km gridded Interactive Multisensor Snow and Ice Mapping System (IMS) sea mask. • These data sources are used for the entirety of the GOFS 3. 1 hindcast but were implemented in operational ACNFS on 3 Feb 2015 • ACNFS only trusts the data sources near the ice edge and largely leaves the field untouched away from the edge. GOFS also trusts data sources near the ice edge, but away from the edge it uses NCODA ± 10% GOFS 3. 1 VTP Meeting 21 November 2016 | 18

GOFS 3. 1 vs. GOFS 3. 0 • Runstream configuration changes: • One NCODA analysis at 12 Z using FGAT and receipt time for observations received since the previous analysis vs. five NCODA analyses at 18 Z using observation time • Significant reduction in daily run-time of the system • ACNFS runstream is like GOFS 3. 0 except that it only goes back three days GOFS 3. 1 VTP Meeting 21 November 2016 | 19

GOFS 3. 1 vs. GOFS 3. 0 • Atmospheric forcing differences: • GOFS 3. 1 used NAVGEM 1. 4 atmospheric forcing during the hindcast period • GOFS 3. 0 and ACNFS both used NAVGEM 1. 2 forcing • Because of the rapid NAVGEM upgrade schedule, it is difficult to ensure consistent forcing between the operational systems and the system being transitioned • However, similar wind speed, surface radiation and net heat flux calibrations are applied to both NWP products and so it is assumed that the different atmospheric forcing in GOFS 3. 1 and GOFS 3. 0/ACNFS will not bias the validation metrics within. GOFS 3. 1 VTP Meeting 21 November 2016 | 20

GOFS 3. 1 Hindcast/Forecast • A GOFS 3. 1 hindcast was integrated emulating the real-time operational system: GLBb 0. 08 -56. 3 • It ran forward one day at a time but did not create multi-day forecasts each hindcast day • Hindcast period: 1 July 2014 – 30 June 2015 • This hindcast will be run up to the time point when NAVGEM 1. 4 became operational (12 October 2016) • Beyond this time it will switch to operational NAVGEM 1. 4 and become GLBb 0. 08 -92. 8 • A series of 5 -day GOFS 3. 1 forecasts were integrated to examine forecast skill: GLBb 0. 08 -56. 4 • Initialized from GLBb 0. 08 -56. 3 restarts and forced with forecast quality NAVGEM 1. 4 forcing • Ten forecasts each month, approximately three days apart GOFS 3. 1 VTP Meeting 21 November 2016 | 21

Ocean Validation: Ocean Analysis Regions GOFS 3. 1 VTP Meeting 21 November 2016 | 22

Ocean Validation: Profile Data • Only unassimilated observations are used in the error analyses within • For a given observation, both systems are sampled at the nearest model grid point • Both the observations and hindcast output are remapped in the vertical to the NCODA depth levels: 0, 2, 4, 6, 8, 10, 12, 15, 20, 25, 30, 35, 40, 45, 50, 60, 70, 80, 90, 100, 125, 150, 200, 250, 300, 350, 400, and 500 m • Model-data differences that exceed 3 standard deviations are thrown out, i. e. 99% confidence interval • This is why the observation count may differ between the two systems GOFS 3. 1 VTP Meeting 21 November 2016 | 23

Ocean Validation: T vs. Depth - Nowcast Mean error GOFS 3. 0 For the global domain, ME is similar but GOFS 3. 1 has lower RMSE than 3. 0. Regionally the results vary. RMSE GOFS 3. 1 Mean over this depth range Number of observations GOFS 3. 1 VTP Meeting 21 November 2016 | 24

Ocean Validation: T vs. Depth - Nowcast GOFS 3. 0 RMSE Mean error GOFS 3. 1 VTP Meeting 21 November 2016 | 25

Mean error GOFS 3. 1 VTP Meeting RMSE Ocean Validation: T vs. Depth - Fore 1 3 5

Mean error GOFS 3. 1 VTP Meeting RMSE Ocean Validation: T vs. Depth - Fore 1 3 5

Large Scale Ocean Prediction Key Accomplishments 2. Temperature (°C) vs. Depth Error Analysis vs. Forecast Length Western Pacific Kuroshio GOFS 3. 1 1 -day forecast 3 -day forecast 5 -day forecast GOFS 3. 0 nowcast For some subregions, the GOFS 3. 1 5 -day forecast error is less than the GOFS 3. 0 nowcast error

Ocean Validation: MLD - Nowcast MLD is defined by a positive density difference of 0. 15 kg/m 3 between the surface and a given depth if both T and S profiles are available, or a negative temperature difference of 0. 5°C between the surface and a given depth if only T profiles are available Lower RMSE in all regions in GOFS 3. 1 GOFS 3. 0 GOFS 3. 1 VTP Meeting 21 November 2016 | 29

Ocean Validation: Md. BE MLD - Nowcast Median Bias Error GOFS 3. 1 Md. BE = -0. 1 m RMSE = 30. 0 m 2° bins Box size indicates observation count GOFS 3. 1 VTP Meeting GOFS 3. 0 Md. BE = -0. 8 m RMSE = 34. 5 m 21 November 2016 | 30

Ocean Validation: Upper Ocean Velocity • GOFS 3. 1 / 3. 0 are compared against independent drifting buoys obtained from the NOAA Global Drifter Program • SST from these drifters is assimilated into GOFS but velocity is not • Drifters have a 15 m drogue and report every 6 hours • Metrics: • Speed: mean error and RMSE • Direction: vector correlation (Crosby et al. 1993) ranges between 0 -2 GOFS 3. 1 VTP Meeting 21 November 2016 | 31

Ocean Validation: Upper Ocean Velocity Speed: Mean error (cm/s) as a function of forecast length GOFS 3. 1 VTP Meeting GOFS 3. 0 21 November 2016 | 32

Ocean Validation: Upper Ocean Velocity Speed: RMSE (cm/s) as a function of forecast length GOFS 3. 1 VTP Meeting GOFS 3. 0 21 November 2016 | 33

Ocean Validation: Upper Ocean Velocity Direction: Vector correlation as a function of forecast length GOFS 3. 1 GOFS 3. 0 0 = no correlation; 2 = perfect correlation GOFS 3. 1 VTP Meeting 21 November 2016 | 34

Ocean Validation: Scorecard • To keep track of the performance of each of the seven error metrics, a scorecard is generated. Let: • Error_model(i) for i=1, … 7, and model = GOFS 3. 1 or GOFS 3. 0 be the errors for the individual metrics • Maximum_error(i) = largest of (Error_model(i)) • The maximum error of the hindcast period for each individual region • Score_model(i) = 1 – (Error_model(i) / Maximum_error(i)) • The scores of the models per each individual metric • Total_score_model = mean (Score_model(i)) • The total score per each individual model GOFS 3. 1 VTP Meeting 21 November 2016 | 35

Ocean Validation: Scorecard Metrics that go into the scorecard analysis 1. 2. 3. 4. 5. 6. 7. T vs. depth: Absolute mean error and RMSE S vs. depth: Absolute mean error and RMSE Cut-off frequency: False / true ratio MLD: Absolute mean error and RMSE SLD: Absolute mean error and RMSE BLG: Absolute mean error and RMSE Drifting buoys: Absolute mean error, RMSE, inverse vector correlation The scorecards presented here are for the nowcast time. All metrics carry equal weight in the determination of skill score. GOFS 3. 1 VTP Meeting 21 November 2016 | 36

Ocean Validation: Scorecard at the nowcast time Global (50°S-50°N) scorecard: Composite and 7 metrics Perfect No skill GOFS 3. 1 GOFS 3. 0 Number of observations GOFS 3. 1 VTP Meeting 21 November 2016 | 37

Ocean Validation: Scorecard at the nowcast time Composite scorecard: 8 regions Perfect No skill GOFS 3. 1 GOFS 3. 0 GOFS 3. 1 VTP Meeting 21 November 2016 | 38

Ice in GOFS 3. 1 and ACNFS Ice Thickness (m) – 20 June 2016 ACNFS GOFS 3. 1 VTP Meeting GOFS 3. 1 ACNFS - GOFS 3. 1 21 November 2016 | 39

Ice Validation: Ice Analysis Regions GOFS 3. 1 VTP Meeting 21 November 2016 | 40

Ice Validation: NH Ice Edge Location Error • Recall the hindcast period spans: 1 July 2014 − 30 June 2015 • On 3 February 2015, operational ACNFS was updated to incorporate ~12. 5 km resolution AMSR 2 satellite ice concentration data and the 4 km gridded Interactive Multisensor Snow and Ice Mapping System (IMS) sea ice mask • GOFS 3. 1 used these data sources for the entirety of the hindcast period • Posey et al. (2015, The Cryosphere) showed a significant reduction in ice edge error using these blended data sources • Thus, ice edge location statistics will be limited to the common time frame when the same data sources were in use in GOFS 3. 1 and ACNFS, i. e. February-June 2015 GOFS 3. 1 VTP Meeting 21 November 2016 | 41

Ice Validation: NH Ice Edge Location Error 24 Hour Forecast Ice Edge Error (km) Pan Arctic 22. 2 km 25. 2 km Barents/Kara 24. 2 km 26. 3 km Bering/Chukchi 21. 1 km 25. 5 km GIN Seas 20. 6 km 26. 7 km GOFS 3. 1 ACNFS Statistics for Feb-Jun 2015 GOFS 3. 1 VTP Meeting 21 November 2016 | 42

Ice Validation: SH Ice Edge Location Error 24 Hour Forecast Ice Edge Error (km) Pan Antarctic 56. 6 km Shackleton Sea 53. 1 km Ross Sea 58. 4 km Weddell Sea 62. 1 km Statistics span the entire year No 4 km IMS assimilation GOFS 3. 1 VTP Meeting 21 November 2016 | 43

Ice Validation: NH Ice Edge Location Error 120 Hour Forecast Ice Edge Error (km) – Pan Arctic Region Tau 000 22. 2 26. 7 22. 2 Tau 024 21. 4 25. 2 23. 1 Tau 048 23. 3 26. 4 24. 7 Tau 072 29. 2 28. 2 25. 4 At long forecast Tau 096 33. 4 times, GOFS 3. 1 is better 31. 2 26. 9 than persistence during freeze season Tau 120 35. 3 33. 4 28. 0 GOFS 3. 1, ACNFS, dashed black line (gray number) is GOFS 3. 1 persistence GOFS 3. 1 VTP Meeting 21 November 2016 | 44 Statistics for February – June 2015

Ice Validation: NH Ice Edge Location Error 120 Hour Forecast Ice Edge Error (km) – Pan Arctic Region Tau 000 22. 2 26. 7 22. 2 Tau 024 21. 4 25. 2 23. 1 Tau 048 23. 3 26. 4 24. 7 Tau 072 29. 2 28. 2 25. 4 At long forecast Tau 096 33. 4 times, GOFS 3. 1 is worse 31. 2 26. 9 than persistence during melt season Tau 120 35. 3 33. 4 28. 0 GOFS 3. 1, ACNFS, dashed black line (gray number) is GOFS 3. 1 persistence GOFS 3. 1 VTP Meeting 21 November 2016 | 45 Statistics for February – June 2015

Ice Validation: NH Ice Edge Location Error GOFS 3. 1 Ice Concentration Difference 5 -day forecast minus analysis: 3 June 2015 During the melt season, GOFS 3. 1 has a tendency to melt ice too quickly at the longer forecast times Red = melt (less forecast ice) GOFS 3. 1 VTP Meeting Blue = freeze (more forecast ice) 21 November 2016 | 46

Large Scale Ocean Prediction Key Accomplishments 2. Forecast Sea Ice Edge Location Error “Wins” NH Ice Edge Location Error GOFS 3. 1 vs. ACNFS: 0 to 5 day forecast NH Ice Edge Location Error GOFS 3. 1 vs. GOFS Persistence: 1 to 5 day forecast Total “Wins” Tau GOFS 3. 1 ACNFS 000 6 0 000 − − 024 5 1 024 7 0 048 4 2 048 7 0 072 2 4 072 4 3 096 1 5 096 2 5 120 1 5 120 2 5 GOFS 3. 1 has more skill at shorter forecasts, but ACNFS has more skill at longer forecasts GOFS 3. 1 VTP Meeting GOFS 3. 1 has more skill at shorter forecasts, but ACNFS has more skill at longer forecasts 21 November 2016 | 47

Large Scale Ocean Prediction Key Accomplishments 2. Sea Ice Thickness (m) Validation 2015 Ice. Bridge Flight Paths Ice Thickness (m) Ice. Bridge GOFS 3. 1 ACNFS 10 Ice Thickness (m) 0 10 GOFS 3. 1 VTP Meeting Central Arctic Line MAE: 0. 77, 0. 97 RMSE: 1. 15, 1. 16 North of Alaska Line MAE: 0. 51, 1. 10 RMSE: 0. 60, 1. 26 0 21 November 2016 | 48

Status of GOFS 3. 1 • GOFS 3. 0 is a good ocean prediction system, and it is hard for an upgrade to provide a large improvement over an established operational system, but GOFS 3. 1 provides new capabilities with the addition of dynamic sea ice and is a clear and substantial improvement over GOFS 3. 0 in the ocean • GOFS 3. 1 received Validation Test Panel (VTP) approval in January 2017 • The VTP concluded: • GOFS 3. 1 is performing better than GOFS 3. 0 w. r. t. the ocean metrics examined • GOFS 3. 1 is performing better than ACNFS w. r. t. the sea ice metrics examined • GOFS 3. 1 entered OPTEST for ocean (NAVOCEANO) and sea ice (NIC) in February 2017