NCI Updates a brief update Ben Evans nci

NCI Updates – a brief update Ben Evans nci. org. au

NCI Supercomputer Upgrade Status • • National Tender - robust process under modern conditions. Various approvals on offers through NCI partnership/governance Not finalized … but close. Expect a general announcement soon. User details with follow later. • Operations by 1 Jan 2020, and will notice some transitional arrangements taking place in 2019 q 3 • System requirement was mix of CPU and GPU - changing to approx 70/30. Transitions • Policy Changes – complying with funding, improving access for “Merit”, and management adjustments • System interaction and Code • A “relatively” easy transition – basically the same NCI environment … but always detailed work/code sensitivities • End of Moore’s law – means more focus into future of software and how we invest (“ACCESS-NRI scoping study”) • HPC scaling and optimisation team will be looking at ACCESS components again for this transition • Physical and Service transitions • Expect some unavoidable planned outages: power, cooling, filesystem upgrades nci. org. au

Caveats • NCRIS funding to NCI does not cover all critical parts of our infrastructure, services, staff and future needs • Feels like users have now caught up with the infrastructure and starting to use new software • Increase in other resources – particularly interactive compute and data collections • NCRIS aware and recognize (some) gaps, but don’t have easy answers • And Multiple department responsibilities/dependencies • Funding will(/is? ) come from a variety of sources: • There are “business model” changes to cover/discuss each gap • Funding always comes with more discussion on processes • Bear with us while we manage all these transitions/funding discussions • That said, three important things: 1. Still progressing with new software and service releases - ongoing improvements and robustness 2. Changes to policies and software prepare for next scale as infrastructure is made available 3. Working consistently and to priorities nci. org. au

NCI Cloud and Layered Services Cloud underpinning infrastructure - underpins accessdev, VDI, all NCI data services, … • Current Tenjin cloud hardware is EOL • Contingencies are being put in place, particularly repurposing Raijin components Cloud 3. 0 (Tenjin++) • Refreshed design for the cloud • Three biggest issues: • I/O performance and scalability (network and filesystem) • Better resource management (memory contention, new hardware) • Increased scale-out NCI managed service needs (K 8 s) • Progressively transition from Tenjin - from 2019 q 4 and into next year VDI • Demand across all users have increased – not just climate/weather. • Great the service has been effective for so long and clearly filled a gap, but order of magnitude increase always means changes/improvements. • User behaviour changes, new analysis environments Jupyterhub – new service • Better interaction for this specific scale-out service • Testing in progress. Minimal release in 2019 q 4. Looking for more challenging test examples (python and R) nci. org. au

Reference Datasets Management Improvements and Maturing • Stronger requirement for FAIR and Trusted Repository • All Improvements required to improve future funding (and its sources), better management and future growth Improvements and growth • Ongoing cleaning of spaces and improvements to quality and fidelity of datasets, access mgt and discoverability • How to find major datasets - geonetwork. nci. org. au • Despite storage funding discussions • Planning for new key datasets • But first step is on improved organisation before random growth • Differentiated – collection by collection. Better collection spaces are much easier to plan for growth • Refreshed data training and examples • To note: • Data storage output spaces are project based, and are more within yours/shareholder management issues • Datasets and data sharing needs– “major ones” should be under our data collections management • We are organising data this way to make manageable • Let me know if you have needs nci. org. au

Other A priority activity for the last year - ESGF and CMIP • Ongoing improvements with the ACCESS and climate and weather models – ACCESS-OM 2, CM 2, ESM 1. 5 • Major activity under a program called Climate DEVL (NCI, BOM, CSIRO and CLEX with ARDC funding) • Focus was to prepare for CMIP 6 operations • ESGF - http: //esgf. nci. org. au and automated (software) data replication processes • Expecting to store the prioritised replicated reference variables plus Australian published reference data • User materials centralised - https: //opus. nci. org. au/display/CMIP+Community+Home Australian Leadership Computing Symposium (ALCS) • http: //nci. org. au/event/alcs 2019/ • Early Nov • Registration now open. • Further announcements on NCI system • Training • Plenary talks and streams with keynotes: Climate/Weather, Geophysics, Material Science, Genomics, Astronomy • NCMAS User Forum nci. org. au