Computing requirements for WISER A Weather climate change

Computing requirements for WISER A Weather climate change Impact Study at Extreme Resolution (WISER) Alan Gadian, Ralph Burton & James Groves NCAS, Leeds JASMIN– forum– 2015 -02 -02 – 1 of 12

Background JASMIN supports for the WISER (Weather climate change Impact Study at Extreme Resolution) project. This project will embed a nested simulation using the NCAR WRF (Weather Research and Forecasting) model within climate reanalyses and predictions from ECMWF and NCAR (US National Center for Atmospheric Research). The nested WRF simulations will cover +/- 680 of latitude, 20 km resolution at 300 latitude, 8 km at 680 latitide, and nested down to 4 km resolution over western Europe (d 02), capturing almost all important weather scales. JASMIN– forum – 25/9/12 – 2 of

NCAR parallel simulation to study hurricane formation Collaborations with NCAR (National Center for Atmospheric Research) (NESL. . Greg Holland et al. ) and EFAS (European Flooding Awareness System, Thielen et al. ) 45 S - 45 N 36 km resolution 51 levels 10 mb TOA 2000 - 2005 Reynolds SST data Periodic EW boundary JASMIN– forum – 25/9/12 – 3 of

What are we trying to do with WISER? Results from NCAR simulation, for domains of 12 km and 36 km A typical weather system pdf The Problem: Typical regional climate models do not capture the most extreme events observed. The pdfs are truncated at high end due to: • resolution – cannot resolve the relevant processes; • capacity – do not include the relevant processes; • rarity – cannot be run long enough to sample JASMIN– forum– 2015 -02 -02 – 4 of 12

Computing requirements for WISER Validation runs will include 1970 -present day and predictive runs will include present day to 2080. Runs are planned in 10 year blocks, each with several completed simultaneously on 3000 -5000 cores. Each 1 year block will use ~300 hours on 4000 cores on Archer. The work is part of an ongoing collaboration between NCAR's Nested Regional Climate Modelling group and NCAS. The project aims to provide new insight into the technology of regional climate modelling (especially from the validation “hindcast” runs) and provide statistical predictions of the European regional impacts of climate change. Typical job • pre-processing in three stages ( Jasmin North and at RAL ) • running the job ( Cray ) • post-processing (Jasmin North and at RAL ) JASMIN– forum– 2015 -02 -02 – 5 of 12

Computing requirements for WISER Preprocessing: This involves running “geogrid”, “ungrib” “metgrid” and “real”, using initial ECMWF or CCSM data which needs to be transferred in. This is carried out at RAL and Jasmin North. Jasmin is ideally suited with fast disc access and as the preprocessing is on the large domains (i. e. suitable for < a few tens of cores. Main runs: The data is transferred to Archer. Each month takes about 30 hours on ~ 4, 000 cores ~171 nodes. Data is dumped every 6 hours ( domains 1 & 2 , each ~ 6 Gb ), in total about 1. 5 Tb. (This transfer to RAL is “slow”. . Investigate bbcp /multiple rsyncs to optimise speed of transfer. ) Postprocessing: Carried out at Leeds. Data transfer to Jasmin-North is fast. JASMIN– forum– 2015 -02 -02 – 6 of 12

Computing requirements for WISER Questions please? JASMIN– forum– 2015 -02 -02 – 7 of 12

JASMIN structure and links JASMIN– forum– 2015 -02 -02 – 9 of 12

WISER Jasmin North JASMIN– forum – 25/9/12 – 10 of

Jasmin – North setup JASMIN– forum– 2015 -02 -02 – 11 of 12

Machine room structure, leading to outside connections 1 Gb lightpath to Jasmin - South Gnodal 10 Gb University Network R 610 Panasas shelf 1 R 610 Panasas shelf 2 R 815 Panasas shelf 3 ~100 Tb Plus several hundred Tb of user discs available for permanent storage JASMIN– forum – 25/9/12 – 12 of

Bottlenecks Issues: - 1. Transfer of CCSM data files from the US to Jasmin 2. Transfer of data files to and from Archer and RAL – Leeds 1 Gb lightpath is good. Hartree – RAL was slow ( 6 months ago, not sure now) 3. Poor client performance on the R 815 (having said that better than university facilities) 4. Network speed issues in and around campus. Care is needed to minimise this (e. g. using quiet times) JASMIN– forum – 25/9/12 – 16 of