Experiences with MCGPFS in DEISA Andreas Schott schottrzg

Experiences with MC-GPFS in DEISA Andreas Schott (schott@rzg. mpg. de) Monterey 2007 -06 -25 HPDC Workshop

Overview Introduction to DEISA Partners in DEISA Aims of DEISA Introduction to Multiple Cluster GPFS Concepts of GPFS Local GPFS Multi-Cluster GPFS Evolution of MC-GPFS in DEISA MC-GPFS until now MC-GPFS in the future Discussion Monterey 2007 -06 -25 HPDC Workshop 2

DEISA Partners Monterey 2007 -06 -25 HPDC Workshop 3

Aims of DEISA Providing HPC resources to the Scientific Community Offering an add-on value to local facilities optimal hardware selection easy usability transparent data access Achievement of these Aims common network structure Monterey 2007 -06 -25 HPDC Workshop 4

DEISA – Network (estimated Q 3 / 2007) UKERNA 1 Gb/s Red. Iris FUNET GARR 10 Gb/s RENATER 10 Gb/s GÉANT LSP 10 Gb/s SURFnet GÉANT 2 10 Gb/s DFN/GÉANT Frankfurt Dedicated 10 Gb/s wavelength 10 Gb/s Monterey 2007 -06 -25 Dedicated 10 Gb/s wavelength (potential) 1 Gb/s LSP DFN HPDC Workshop ralph. niederberger@fz-juelich. de 5

Aims of DEISA Providing HPC resources to the Scientific Community Offering an add-on value to local facilities optimal hardware selection easy usability transparent data access Achievement of these Aims common network structure using internal features of job schedulers additional middleware for easy access (e. g. UNICORE) Monterey 2007 -06 -25 HPDC Workshop 6

user job IDRIS IBM P 4 FZJ IBM P 4 HLRS NEC SX 8 AIX LL-MC ECMWF IBM P 5 Super-UX NQS II AIX LL-MC AIX LL HPCX IBM P 5 AIX LL CSC IBM P 4 LRZ SGI ALTIX AIX LL-MC CINECA IBM P 5 LINUX PBS Pro AIX LL-MC LINUX LL LINUX LSF BSC IBM PPC Monterey 2007 -06 -25 RZG IBM P 4 SARA SGI ALTIX HPDC Workshop 7 johannes. reetz@rzg. mpg. de

CINECA user job Gateway CINECA AIX LL-MC Super-UX NQS II AIX LL-MC AIX LL-MC NJS CINECA IBM P 5 IDB LINUX PBS Pro AIX LL-MC UUDB LINUX LL Monterey 2007 -06 -25 LINUX LSF HPDC Workshop 8 johannes. reetz@rzg. mpg. de

CINECA user Gateway ECMWF Gateway CSC job Gateway CINECA Gateway IDRIS Gateway FZJ IDB Gateway HPCX Gateway LRZ NJS IDRIS IBM P 4 NJS FZJ IBM P 4 Gateway BSC Gateway HLRS Gateway RZG NJS HLRS NEC SX 8 UUDB IDB UUDB Gateway SARA UUDB NJS ECMWF IBM P 5 IDB NJS HPCX IBM P 5 AIX LL-MC UUDB Super-UX NQS II AIX LL-MC AIX LL NJS LRZ SGI ALTIX AIX LL-MC UUDB NJS CINECA IBM P 5 IDB UUDB AIX LL NJS CSC IBM P 4 IDB LINUX PBS Pro AIX LL-MC UUDB LINUX LL Monterey 2007 -06 -25 LINUX LSF NJS BSC IBM PPC NJS SARA SGI ALTIX IDB UUDB NJS RZG IBM P 4 IDB UUDB HPDC Workshop IDB UUDB 9 IDB UUDB johannes. reetz@rzg. mpg. de

CINECA user Gateway ECMWF Gateway CSC job Gateway CINECA Gateway IDRIS Gateway FZJ IDB Gateway HPCX Gateway LRZ NJS IDRIS IBM P 4 NJS FZJ IBM P 4 Gateway BSC Gateway HLRS Gateway RZG NJS HLRS NEC SX 8 UUDB IDB UUDB Gateway SARA UUDB NJS ECMWF IBM P 5 IDB NJS HPCX IBM P 5 AIX LL-MC UUDB Super-UX NQS II AIX LL-MC AIX LL NJS LRZ SGI ALTIX AIX LL-MC UUDB NJS CINECA IBM P 5 IDB UUDB AIX LL NJS CSC IBM P 4 IDB LINUX PBS Pro AIX LL-MC UUDB LINUX LL Monterey 2007 -06 -25 LINUX LSF NJS BSC IBM PPC NJS SARA SGI ALTIX IDB UUDB NJS RZG IBM P 4 IDB UUDB HPDC Workshop IDB UUDB 10 IDB UUDB johannes. reetz@rzg. mpg. de

Aims of DEISA Providing HPC resources to the Scientific Community Offering an add-on value to local facilities optimal hardware selection easy usability transparent data access Achievement of these Aims common network structure using internal features of job schedulers additional middleware for easy access (e. g. UNICORE) global file system in a network of trust Monterey 2007 -06 -25 HPDC Workshop 11

General Concepts of MC-GPFS = Multiple Cluster General Parallel File System available for all HPC architectures in DEISA servers available for AIX and Linux Principle Structure distributed – shared – striped kernel add-on for file system block oriented data transfer Features achieved shared and high performance access safe and secure data high administrative flexibility Monterey 2007 -06 -25 HPDC Workshop 12

General Concepts of MC-GPFS Technical Aspects each site with its own servers possible local disk space locally administered scalability and high performance access by inherent parallelism easy extensible file consistency by sophisticated token management high recoverability and increased data availability simplified storage management storage pools, file sets simplified administration globally acting commands Monterey 2007 -06 -25 HPDC Workshop 13

General Concepts of MC-GPFS Security Aspects separate network communication for administration possible remote security authenticated remote access for servers mount and/or data with SSL-keys easy root-mapping easy no-suid functionality userid mapping for remote access via interfaces Monterey 2007 -06 -25 HPDC Workshop 14

General Concepts of MC-GPFS Access and Availability transparent access no special data transfer commands required global visibility inside DEISA extended access rights no single point of failure communication delegated locking and other communication Monterey 2007 -06 -25 HPDC Workshop 15

Summary of MC-GPFS Local and Remote High Performance Access high parallelism in data and file access very large file and file system support High Availability each site with its own servers redundant access path simply extensible and scalable striped data parallel access path Monterey 2007 -06 -25 HPDC Workshop 16

Local GPFS File Servers Network File Server 1 File Server 2 . . . File Server N . . . Disk System M FC-Switch Disk System 1 Monterey 2007 -06 -25 Disk System 2 HPDC Workshop 17

Local GPFS Access Network File Server 1. . . N Compute Server 1 Compute Server N FC-Switch Disk System 1. . . M Monterey 2007 -06 -25 HPDC Workshop 18

Remote GPFS Access WAN Network Site A File Server 1. . . N Compute Server 1 FC-Switch Disk System 1. . . M Monterey 2007 -06 -25 Network Site B File Server 1. . . N Compute Server 1 FC-Switch Compute Server N HPDC Workshop Disk System 1. . . M Compute Server N 19

Advantages of GPFS (admin) Easy Management Easy Extensibility High Performance Good Security Features Add-On Features like HSM Functionality Monterey 2007 -06 -25 HPDC Workshop 20

Advantages of GPFS (user) Standard Access Methods Transparent Access Data globally visible No special actions for data transfer required Simplicity Extended Access Right Features Add-On Features like HSM Functionality Monterey 2007 -06 -25 HPDC Workshop 21

GPFS Configuration in DEISA Each AIX-site provides its own server Some non-AIX-sites will provide servers based on Linux RZG hosts disk space for non-AIX-sites without servers RZG provides HSM-functionality on GPFS locally disk space performs like local disk space total of more than 30 TB wide area network connection with 10 GBit/s (mostly) remotely disk space no longer limited by network Monterey 2007 -06 -25 HPDC Workshop 22

DEISA „proof of concept“ phase 1 Gb/s RENATER GÈANT DFN RENATER GARR Premium IP: IP Priority: LSPs: Monterey 2007 -06 -25 HPDC Workshop 23

Evolution of GPFS in DEISA FZJ (DE) Power 4 AIX October Initial 2004 IDRIS (FR) Power 4 AIX RZG (DE) Power 4 AIX CINECA (IT) Power 5 AIX Monterey 2007 -06 -25 HPDC Workshop 24

DEISA – Tera. Grid Connection Super Computing 2005 Chicago Teragrid Amsterdam New York SDSC FZJ Jülich Internet 2/Abilene Paris 1 Gb/s Premium IP IDRIS Orsay RENATER NREN France 1 Gb/s LSP 10 Gb/s Frankfurt DFN NREN Germany GEANT Milano RZG Munich Cineca GARR Bologna. NREN Italy 30 -40 Gb/s 10 Gb/s Monterey 2007 -06 -25 HPDC Workshop 25 R. Niederberger@fz-juelich. de

DEISA 1 Gb/s network infrastructure FUNET DFN SURFnet UKERNA GÉANT LSPs GARR RENATER Red. Iris Monterey 2007 -06 -25 HPDC Workshop 26

Evolution of GPFS in DEISA SARA (NL) SGI-Altix Linux CSC (FI) Power 4 AIX FZJ (DE) Power 4 AIX October July May 2006 2004 2005 IDRIS (FR) Power 4 AIX BSC (ES) Power. PC Linux Monterey 2007 -06 -25 RZG (DE) Power 4 AIX CINECA (IT) Power 5 AIX HPDC Workshop 27

Upgrade of Multiple Cluster GPFS Problems with GPFS 2. 3 Initial MC-functionality not inherently integrated Each-to-Any communication required Limitation of participating nodes Advantages of GPFS 3. 1 Better Multi-Cluster Support Better Encapsulation by possible use of private addresses Higher Independence between sites Higher Stability Better Performance Monterey 2007 -06 -25 HPDC Workshop 28

Evolution of GPFS in DEISA SARA (NL) SGI-Altix Linux ECMWF (GB) Power 5+ AIX CSC (FI) Power 4 AIX September February July 2006 2007 2006 IDRIS (FR) Power 4 AIX BSC (ES) Power. PC Linux Monterey 2007 -06 -25 FZJ (DE) Power 4 AIX CINECA (IT) Power 5 AIX HPDC Workshop LRZ (DE) SGI-Altix Linux RZG (DE) Power 4 AIX 29

Status of Multiple Cluster GPFS Site File. Storage Compute-CPUs server TFlops Memory CINECA 2 2 TB 480 Power 5 (1. 9 GHz) 2. 6 1152 GB CSC 2 2 TB 512 Power 4 (1. 1 GHz) 2. 2 672 GB ECMWF 2 1 TB 2640 Power 5+ (1. 9 GHz) 20. 1 2250 GB FZJ 2 4 TB 1288 Power 4 (1. 7 GHz) 8. 9 5152 GB IDRIS 2 2 TB 1024 Power 4 (1. 3 GHz) 6. 7 3136 GB LRZ (RZG) 0 TB 9728 Montecito (1. 6 GHz) 62. 3 39064 GB RZG 2 10 TB 928 Power 4 (1. 3 GHz) 4. 6 2368 GB Monterey 2007 -06 -25 HPDC Workshop 30

Evolution of GPFS in DEISA EPCC (GB) Power 4 AIX ECMWF (GB) Power 5+ AIX SARA (NL) SGI-Altix Power 5 Linux CSC (FI) Cray Power 4 XT 4 Linux AIX /deisa/<site>/home/<group>/<user> September February October May. Final July Initial 2006 2007 2006 2004 2005 2007 2006 2007 /deisa/<site>/data /<group>/<user> IDRIS (FR) Power 4 AIX BSC (ES) Power. PC Linux Monterey 2007 -06 -25 CINECA (IT) Power 5 AIX HPDC Workshop LRZ (DE) SGI-Altix Linux FZJ (DE) Power 4 AIX HLRS (DE) NEC-SX 8 Super-UX RZG (DE) Power 4 AIX 31

Discussion Questions? Thanks. Monterey 2007 -06 -25 HPDC Workshop 32