Center for Computational Sciences Cray X 1 and

Center for Computational Sciences Cray X 1 and Black Widow at ORNL Center for Computational Sciences Buddy Bland CCS Director of Operations SOS 7 Workshop March 5, 2003 OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Center for Computational Sciences Cray X 1 Overview Cray X 1 is the commercial name for the “SV 2” project that Cray has been building for the NSA for more than 4 years. q Combines multi-streaming vector processors with a memory interconnect system similar to the T 3 E. q ORNL is evaluating the X 1 for the Office of Advanced Scientific Computing Research q Specific details of the configuration are on Rolf’s web page. q OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Center for Computational Sciences Q 1: What is unique in structure and function of your machine? System Processor Performance (GF) Memory Bandwidth (B/F) Interconnect Bandwidth (B/F) Balance! OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY HP Long's Peak & Quadrics Cray X 1 Power 4 Red Storm 12. 80 5. 20 4. 00 6. 40 1. 06 1. 33* 1. 00 0. 19 0. 69 1. 6* 0. 19 3. 00 1. 00 * Corrections by Jim Tomkins

Center for Computational Sciences Cray X 1 System Picture of the Cray X 1 at the factory awaiting shipment to ORNL q Delivery scheduled for March 18 th q 32 processor, liquidcooled cabinet q 128 GB memory q 8 TB disk q OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Center for Computational Sciences Phase 1 b – Summer 2003 256 Vector Processors q 1 TB shared memory q 32 TB of disk space q 3. 2 Tera. FLOP/s OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Center for Computational Sciences Cray X 1/Black Widow 4 phase evaluation and deployment Phase 4 120 TF, 40 TB, 400 TB BW architecture per DOE apps Phase 3 40. 96 TF, 13. 107 TB, 102 TB Phase 2 8. 192 TF, 2. 621 TB, 24 TB Phase 1 3. 2 TF, 1 TB, 20 TB Currently Funded Simulator 1 Q 2003 Phase 1 3 TF (256 CPU) 2 Q 2003 3 Q 2003 Phase 2 8 TF (640 CPU) 4 Q 2003 OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 1 Q 2004 2 Q 2004 3 Q 2004 Phase 3 40 TF (3200 CPU) Phase 4 120 TF 4 Q 2004 1 Q 2006 4 Q 2005

Center for Computational Sciences Q 2: What characterizes your applications? “Every day the same thing, variety” - Yosemite Sam OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Center for Computational Sciences CCS is focused on capability computing for a few applications Sci. DAC Astrophysics Nanophase Materials Sci. DAC Fusion OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY Genomes to Life Sci. DAC Climate Sci. DAC Chemistry

Center for Computational Sciences Climate (CCSM) simulation resource projections At current scientific complexity, one century simulation requires 12. 5 days Single researcher transfers 80 Gb/day and generates 30 TB storage each year Science drivers: regional detail / comprehensive model CCSM Coupled Model Resolution Configurations: Atmosphere Land Ocean 2008/2009 230 km. L 26 50 km Sea Ice Model years/day 10 km. L 80 100 km National Resource (dedicated TF) Storage (TB/century) 8 2002/2003 30 km. L 96 5 km 100 km. L 40 10 km 8 3 750 1 250 OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY • Blue line represents total national resource dedicated to CCSM simulations and expected future growth to meet demands of increased model complexity • Red line shows data volume generated for each century simulated

Center for Computational Sciences GTL resource projections Protein machine Interactions 1000 TF Molecule-based cell simulation Molecular machine classical simulation 100 TF 10 TF Cell, pathway, and network simulation Community metabolic regulatory, signaling simulations Constrained rigid docking Constraint-based flexible docking Current U. S. computing 1 TF* Genome-scale protein threading Comparative Genomics *Teraflops Biological Complexity OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Center for Computational Sciences Q 3: What prior experience guided you to this choice q Workshops q IBM, with vendors and users Cray, HP, SGI q Long experience in evaluation of new systems OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Center for Computational Sciences Sustained performance of 0. 02 TF for climate calculations on IBM Power 4 The Community Climate System Model (CCSM) is fullycoupled, global climate model that provides state-of-theart computer simulations of Earth’s past, present, and future climate states Scaling to 352 processors is expected to be possible with Federation interconnect and recent improvements in software engineering q q q q T 42 with 1 degree ocean Memory requirements: 11. 6 GB 96 IBM Power 4 processors Processor speed: 5. 2 GF Total peak: 500 GF Sustained performance: 16. 2 GF 3. 2% of peak 7. 9 years simulated per day OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY

Center for Computational Sciences Questions ? OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY